Skip to content

Backing Up Your Database

Overview

  • Installing and configuring Litestream
  • Using rclone for off-site backups

Running in production is much more than just deploying your application. It's about keeping your application running smoothly and your data safe. If your server goes down, a new one can be started in minutes. If your data is lost, it is lost forever.

If you look at your config/database.yml, you will see that it contains definitions for four databases: primary, cache, queue, and cable. For sqlite3, these are stored in a storage directory, which is mapped to a directory on your host machine. This is where your data is stored. It is possible to back up all four, but for now we are going to focus on the primary database. The primary database is the one that contains your users, products, and orders. It's the one you can't afford to lose. Because you set up a volume, your data is safe as long as your host machine is running. But what if your host machine goes down?

# config/database.yml
# ...your database.yml content here...

As the comments state, these databases are stored in a persistent Docker volume. Docker stores its data in a directory called /var/lib/docker. Inside that directory is a directory called volumes. You created a volume called depot_storage. Inside the volumes directory is a file named production.sqlite3. This is your database.

Litestream is a tool that can be used to back up your database. It is free and open source, easy to set up, and can back up to the S3 Object Storage you have already set up. It can also be used to restore your database to a point in time.

We could run it in your Kamal container, or as a Kamal accessory, but that requires building an image and pushing it to a repository. Instead we are going to run it on your host machine.

Following Litestream's installation instructions, first SSH into your host and run the following commands:

apt-get update
apt-get install -y wget
export LITESTREAM=https://github.com/benbjohnson/litestream/releases/download
wget $LITESTREAM/v0.3.13/litestream-v0.3.13-linux-amd64.deb
dpkg -i litestream-v0.3.13-linux-amd64.deb
rm litestream-v0.3.13-linux-amd64.deb

Note: if you have chosen an arm64 host, substitute arm64 for amd64 in the instructions above.

Now find the ACCESS_KEY_ID, SECRET_ACCESS_KEY, ENDPOINT_URL, REGION, and BUCKET_NAME in your password manager. Then run nano /etc/litestream.yml. Replace the file with the following, substituting in the values you found in your password manager:

# /etc/litestream.yml
# This is the configuration file for Litestream.
#
# For more details, see: https://litestream.io/reference/config/
#
dbs:
  - path: /var/lib/docker/volumes/depot_storage/production.sqlite3
    replicas:
      - type: s3
        access-key-id: ACCESS_KEY_ID
        secret-access-key: SECRET_ACCESS_KEY
        endpoint: ENDPOINT_URL
        region: REGION
        bucket: BUCKET_NAME
        path: storage/production.sqlite3

Exit nano by pressing Ctrl-X, then press Y to save the file. Note: Do not store this data in git.

Now start the Litestream service:

systemctl enable litestream
systemctl start litestream

Once configured, Litestream is pretty much set-and-forget. It will back up your database after every write, and can be used to restore your database to a point in time. The Litestream reference is a good place to start if you want to learn more.

While our data is backed up in S3, it is not truly off-site. On a separate machine (this can be your laptop), install rclone and use rclone sync to extract a copy of your data. Set up a cron job to run this command daily.