Linux Backup – duplicity + Dropbox

By gbe0Posted Desktop, Linux, Scripting, Server

I was looking for a backup solution for some personal servers; I had a few requirements in mind:

  • Not agent based
  • Lightweight
  • Stores backups in a Dropbox application folder (so each server can have its own folder not accessible to the others)
  • Encrypted storage of backups
  • Only backup the data that I specifically want
  • No requirement to store all previous backups on the server
  • Differential backups

After searching around for a while I came across duplicity which met all of my requirements. How ever, I came across one major problem – relatively recently Dropbox made a change which removes support for “long lived access tokens”. This is a problem for duplicity as it requires an access token to store the backup and with the token expiring after 4 hours it is next to useless.

I had a look at the other backends available for duplicity and found that a Rclone backend is available. Rclone does not encounter the same problem with the access tokens (it will use a refresh token in order to get a short lived access token) and it ended up working perfectly.

Software Requirements

Only two packages are required, rclone and duplicity. For Debian and Ubuntu this was easy enough:

apt -y install rclone duplicity

RHEL 7

For RHEL 7 based OS’s the duplicity release is too old and it will result in an error:

InvalidBackendURL: Syntax error (port) in: rclone://dropbox:/Duplicity AFalse BNone Cdropbox:

The fix for this was to install the current release of duplicity release from source. The following packages were required:

yum install rclone python3 python36-future python3-devel gcc librsync-devel

You can then get a copy of the duplicity tar.gz file, extract it and run the setup:

cd /usr/local/src
wget "https://launchpad.net/duplicity/0.8-series/0.8.23/+download/duplicity-0.8.23.tar.gz"
tar zxf duplicity-0.8.23.tar.gz
cd duplicity-0.8.23
python3 setup.py install

Encryption/GPG Key

GPG is used to encrypt the backups. Even if there is an existing private key for the server for other purposes I prefer to keep it completely separate from the backup one. To generate a new key for duplicity:

gpg --expert --full-gen-key

Note: The --full-gen-key argument must be --gen-key for RHEL 7 based hosts otherwise you will get an invalid option error.

If available, I use the ECC/ECC key type as the keys are much shorter and easier to copy/paste. If the option is not available the default RSA/RSA type is fine.

Once the key is generated, take note of the key ID (best practice is to use the long key ID, not the short 8 or 16 character ID). It is imperative that a backup of the private key is taken and stored in a safe place at this point – if you lose the private key your backups are useless and you cannot recover them. To take a backup of the private key:

## Export key in ASCII format
gpg --output privkey.txt --armor --export-secret-key BD841A370D3003ACC61EDB2722DAAA52A9076D45

## Export key in GPG keyring format
gpg --output privkey.gpg --export-secret-key BD841A370D3003ACC61EDB2722DAAA52A9076D45

Once the backup has been taken and stored securely be sure to destroy the exported key files.

Dropbox Configuration

In the Dropbox Developer Tools, create a new application with “App folder” access:

Once the application is created, select the permissions tab and add set the following permissions:

  • account_info.read
  • files.metadata.write
  • files.metadata.read
  • files.content.write
  • files.content.read
  • sharing.write
  • sharing.read

Go back to the settings page and set the following OAuth 2 redirect URI:

http://localhost:53682/

You will then need to take note of the app key and app secret; these will be used to configure rclone in the next step:

Rclone Configuration

rclone needs to have the Dropbox backend configured. Run the rclone configuration command:

rclone config

Select the option to add a new backend (n). The name of the backend can be set to anything; I prefer to set it to dropbox to make it obvious what it is. You will then be prompted for the type of storage – enter the relevant ID of Dropbox.

You will then be prompted for the following settings:

  • OAuth Client Id: Leave blank
  • client_id: The Dropbox App key
  • client_secret: The Dropbox App secret
  • Edit advanced config? (y/n): No

The next prompt is for remote configuration to authorize the application. Since this is being ran remotely, select n for the remote/headless machine option. You will then be provided with a command to run on a machine that has a browser and rclone available; this will open a web browser and prompt you to authorize the application with your Dropbox account. After it has been authorized enter the token back into the rclone configuration wizard.

The rclone remote has now been configured and is ready to use by duplicity.

duplicity Configuration

duplicity needs to be aware of the GPG key ID and passphrase to unlock the key in order to encrypt the backups. These are sourced from the following environment variables:

  • GPG_KEY
  • GPG_PASSPHRASE

Create a file (I prefer to put it in /root/.duplicity_env) to source the variables in the backup script:

## GPG Config
export GPG_KEY="B0D763E0A6D5069DC60F0D99937AF361A9647ACA"
export PASSPHRASE="MY LONG GPG PASSPHRASE"

To be safe, make sure the file is only accessible by the root user:

chmod 0600 /root/.duplicity_env

duplicity is now ready to use, the backup script can be created.

Backup Script

I only need to backup some directories/files, I don’t need an entire copy of the server. For the purposes of this example will be backing up the following directories

  • /etc
  • /root/.bash_history
  • /root/.zsh_history
  • /opt/backup.sh (the backup script itself)
  • /var/www with an exclusion of /var/www/cache

The backups will be retained for 360 days. A full backup will be taken every 30 days; other backups will be differential only.

Here is the contents of the backup script (/opt/backup.sh):

#!/bin/bash

## Terminate if any command fails
set -e

## Source variables
source /root/.duplicity_env

## Remove old backups
duplicity \
  remove-older-than 360D \
  --force \
  rclone://dropbox:/Duplicity

## Run backup
duplicity \
  --verbosity info \
  --encrypt-sign-key="$GPG_KEY" \
  --full-if-older-than 30D \
  --asynchronous-upload \
  --log-file "/var/log/duplicity.log" \
  --exclude '/var/www/cache/*' \
  --include '/etc' \
  --include '/root/.bash_history' \
  --include '/root/.zsh_history' \
  --include '/opt/backup.sh' \
  --include '/var/www' \
  --exclude '**' \
  / \
  rclone://dropbox:/Duplicity

## Cleanup
duplicity \
  cleanup \
  --force \
  rclone://dropbox:/Duplicity

## Unset variables for safety
unset GPG_KEY
unset PASSPHRASE

Make sure the script is executable (chmod +x /opt/backup.sh) and try running it to ensure that there are no errors and the first backup runs. If everything works as expected, the backup can then be scheduled using your preferred method.

systemd Scheduling

I prefer to use systemd to schedule the backups rather than cron. To create a timer that runs the backup daily:

cat << EOF > /etc/systemd/system/duplicity.timer
[Unit]
Description=Call the Duplicity backup script each day
RefuseManualStart=no
RefuseManualStop=no

[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=3600
Unit=duplicity.service

[Install]
WantedBy=timers.target
EOF
cat << EOF > /etc/systemd/system/duplicity.service
[Unit]
Description=Execute the Duplicity backup script

[Service]
Type=simple
ExecStart=/opt/backup.sh
TimeoutSec=3600
TimeoutStartSec=3600
TimeoutStopSec=3600
StandardOutput=null
StandardError=null

[Install]
WantedBy=default.target
EOF
systemctl enable duplicity.timer
systemctl start duplicity.timer

Leave a Reply

Your email address will not be published. Required fields are marked *