I was looking for a backup solution for some personal servers; I had a few requirements in mind:
- Not agent based
- Stores backups in a Dropbox application folder (so each server can have its own folder not accessible to the others)
- Encrypted storage of backups
- Only backup the data that I specifically want
- No requirement to store all previous backups on the server
- Differential backups
After searching around for a while I came across duplicity which met all of my requirements. How ever, I came across one major problem – relatively recently Dropbox made a change which removes support for “long lived access tokens”. This is a problem for duplicity as it requires an access token to store the backup and with the token expiring after 4 hours it is next to useless.
I had a look at the other backends available for duplicity and found that a Rclone backend is available. Rclone does not encounter the same problem with the access tokens (it will use a refresh token in order to get a short lived access token) and it ended up working perfectly.
Only two packages are required,
duplicity. For Debian and Ubuntu this was easy enough:
apt -y install rclone duplicity
For RHEL 7 based OS’s the
duplicity release is too old and it will result in an error:
InvalidBackendURL: Syntax error (port) in: rclone://dropbox:/Duplicity AFalse BNone Cdropbox:
The fix for this was to install the current release of
duplicity release from source. The following packages were required:
yum install rclone python3 python36-future python3-devel gcc librsync-devel
You can then get a copy of the
duplicity tar.gz file, extract it and run the setup:
cd /usr/local/src wget "https://launchpad.net/duplicity/0.8-series/0.8.23/+download/duplicity-0.8.23.tar.gz" tar zxf duplicity-0.8.23.tar.gz cd duplicity-0.8.23 python3 setup.py install
GPG is used to encrypt the backups. Even if there is an existing private key for the server for other purposes I prefer to keep it completely separate from the backup one. To generate a new key for duplicity:
gpg --expert --full-gen-key
--full-gen-key argument must be
--gen-key for RHEL 7 based hosts otherwise you will get an invalid option error.
If available, I use the ECC/ECC key type as the keys are much shorter and easier to copy/paste. If the option is not available the default RSA/RSA type is fine.
Once the key is generated, take note of the key ID (best practice is to use the long key ID, not the short 8 or 16 character ID). It is imperative that a backup of the private key is taken and stored in a safe place at this point – if you lose the private key your backups are useless and you cannot recover them. To take a backup of the private key:
## Export key in ASCII format gpg --output privkey.txt --armor --export-secret-key BD841A370D3003ACC61EDB2722DAAA52A9076D45 ## Export key in GPG keyring format gpg --output privkey.gpg --export-secret-key BD841A370D3003ACC61EDB2722DAAA52A9076D45
Once the backup has been taken and stored securely be sure to destroy the exported key files.
In the Dropbox Developer Tools, create a new application with “App folder” access:
Once the application is created, select the permissions tab and add set the following permissions:
Go back to the settings page and set the following OAuth 2 redirect URI:
You will then need to take note of the app key and app secret; these will be used to configure
rclone in the next step:
rclone needs to have the Dropbox backend configured. Run the
rclone configuration command:
Select the option to add a new backend (
n). The name of the backend can be set to anything; I prefer to set it to
dropbox to make it obvious what it is. You will then be prompted for the type of storage – enter the relevant ID of Dropbox.
You will then be prompted for the following settings:
- OAuth Client Id: Leave blank
- client_id: The Dropbox App key
- client_secret: The Dropbox App secret
- Edit advanced config? (y/n): No
The next prompt is for remote configuration to authorize the application. Since this is being ran remotely, select
n for the remote/headless machine option. You will then be provided with a command to run on a machine that has a browser and
rclone available; this will open a web browser and prompt you to authorize the application with your Dropbox account. After it has been authorized enter the token back into the
rclone configuration wizard.
rclone remote has now been configured and is ready to use by duplicity.
duplicity needs to be aware of the GPG key ID and passphrase to unlock the key in order to encrypt the backups. These are sourced from the following environment variables:
Create a file (I prefer to put it in
/root/.duplicity_env) to source the variables in the backup script:
## GPG Config export GPG_KEY="B0D763E0A6D5069DC60F0D99937AF361A9647ACA" export PASSPHRASE="MY LONG GPG PASSPHRASE"
To be safe, make sure the file is only accessible by the root user:
chmod 0600 /root/.duplicity_env
duplicity is now ready to use, the backup script can be created.
I only need to backup some directories/files, I don’t need an entire copy of the server. For the purposes of this example will be backing up the following directories
/opt/backup.sh(the backup script itself)
/var/wwwwith an exclusion of
The backups will be retained for 360 days. A full backup will be taken every 30 days; other backups will be differential only.
Here is the contents of the backup script (
#!/bin/bash ## Terminate if any command fails set -e ## Source variables source /root/.duplicity_env ## Remove old backups duplicity \ remove-older-than 360D \ --force \ rclone://dropbox:/Duplicity ## Run backup duplicity \ --verbosity info \ --encrypt-sign-key="$GPG_KEY" \ --full-if-older-than 30D \ --asynchronous-upload \ --log-file "/var/log/duplicity.log" \ --exclude '/var/www/cache/*' \ --include '/etc' \ --include '/root/.bash_history' \ --include '/root/.zsh_history' \ --include '/opt/backup.sh' \ --include '/var/www' \ --exclude '**' \ / \ rclone://dropbox:/Duplicity ## Cleanup duplicity \ cleanup \ --force \ rclone://dropbox:/Duplicity ## Unset variables for safety unset GPG_KEY unset PASSPHRASE
Make sure the script is executable (
chmod +x /opt/backup.sh) and try running it to ensure that there are no errors and the first backup runs. If everything works as expected, the backup can then be scheduled using your preferred method.
I prefer to use systemd to schedule the backups rather than cron. To create a timer that runs the backup daily:
cat << EOF > /etc/systemd/system/duplicity.timer [Unit] Description=Call the Duplicity backup script each day RefuseManualStart=no RefuseManualStop=no [Timer] OnCalendar=daily Persistent=true RandomizedDelaySec=3600 Unit=duplicity.service [Install] WantedBy=timers.target EOF cat << EOF > /etc/systemd/system/duplicity.service [Unit] Description=Execute the Duplicity backup script [Service] Type=simple ExecStart=/opt/backup.sh TimeoutSec=3600 TimeoutStartSec=3600 TimeoutStopSec=3600 StandardOutput=null StandardError=null [Install] WantedBy=default.target EOF systemctl enable duplicity.timer systemctl start duplicity.timer