Backup strategy for your server

BLACKBOX does not make automatic backups of your server. The “Backups” button on the server card, on the “My Servers” page of the app.blackbox.md panel (app.blackbox.md/my-servers; from blackbox.md: “Contul meu” (My account) → “Panou de administrare” (Admin panel) → “My Servers”), is not active. This guide shows what to back up, where to keep the copies, how to automate them and how to check that you can restore them. Run all the commands on the server, over SSH, as root.

Available to: the owner of a BLACKBOX VPS or dedicated server, with root access over SSH: Connecting to your server over SSH (Windows, macOS, Linux). The commands are for Ubuntu, Debian, AlmaLinux and CentOS Stream.

The blackbox.md website is available in Romanian and Russian; this article quotes the Romanian labels with their English meaning.

Who is responsible for backups

You have root access to the server and control everything that runs on it, so you are also responsible for backing up its data: Responsibility for server data and backups. Chapter 9, “Limitarea răspunderii” (Limitation of liability), of the Terms and Conditions states that “BLACKBOX nu răspunde pentru daune indirecte, pierdere de profit, pierderi de date sau întreruperi de activitate suferite de Client sau terți” (BLACKBOX is not liable for indirect damages, loss of profit, data loss or business interruptions suffered by the Customer or third parties).

The panel has no automatic backups and no snapshots. If you click “Backups” on the server card, “Loading…” appears, followed by the message backup_extra_disabled: the automatic backup service is not available. The server and the data on it are not affected. See The “My Servers” page: the server card.

Important A server deleted with “Terminate” or by canceling its order cannot be recovered. The only copies of the data are the ones you made yourself, outside the server.

The 3-2-1 rule

A copy in another directory on the same server is not a backup: it disappears along with the server if the server is deleted, compromised or no longer boots.

What to back up

Don't copy the database files (/var/lib/mysql, /var/lib/postgresql, /var/lib/pgsql) while the database is running: the copy may be inconsistent, and the database won't start from it. The same goes for Docker volumes that hold databases: make a dump with the database's own command, run inside the container. An example for PostgreSQL, as a line in the script from step 1:

docker exec CONTAINER_NAME pg_dump -U USERNAME -Fc DB_NAME > "$DIR/DB_NAME-$DATE.dump"

For MariaDB or MySQL in a container, use mysqldump the same way (in recent MariaDB images, mariadb-dump), with the user and password from .env. See also Installing Docker and Docker Compose.

You don't need to back up /proc, /sys, /dev, /run, /tmp, caches, Docker images (they can be downloaded or built again) or system packages (they can be reinstalled). Code kept in an external Git repository doesn't need a separate copy, as long as you check that your changes actually get there.

Where to keep the copies

For the offsite copy in the 3-2-1 rule, choose at least one destination at another provider or in your own office. For a backup server at BLACKBOX, the plans in the Basic family have large disks, up to 400 GB: VPS plans: how to choose.

BLACKBOX Cloud Storage can't be ordered online. On the blackbox.md/hosting/cloud-storage page (“Găzduire” (Hosting) → “Cloud Storage”), the “Comandați stocare” (Order storage) button takes you to the “Contacte” (Contacts) page. You get the size, the price and the access methods in the team's offer; ask which access methods the storage space supports, so you can choose between restic and rsync. See Products and configurations available on request.

Step 1: a daily database dump, from cron

The script below exports the database to /var/backups/db every night and deletes dumps older than 7 days from there. Step 2 then copies them off the server. Installing databases: Databases: MariaDB/MySQL and PostgreSQL.

You edit the files with nano. If it's missing (AlmaLinux, CentOS Stream), install it:

dnf install -y nano

1. Create the script:

   nano /usr/local/sbin/db-dump.sh

2. Paste the content below, replace DB_NAME with the name of your database, save with Ctrl+O and Enter, then exit with Ctrl+X:

   #!/bin/bash
   set -euo pipefail
   umask 077
   cd /
   DIR=/var/backups/db
   DATE=$(date +%F)
   mkdir -p "$DIR"
   mysqldump --single-transaction --routines DB_NAME | gzip > "$DIR/DB_NAME-$DATE.sql.gz"
   find "$DIR" -type f -mtime +7 -delete

   For PostgreSQL, replace the mysqldump line with this one:

   sudo -u postgres pg_dump -Fc DB_NAME > "$DIR/DB_NAME-$DATE.dump"

   For several databases, repeat the line for each one. On MariaDB, the command is also called mariadb-dump.

3. Make the script executable, run it once and check the file it created:

   chmod 700 /usr/local/sbin/db-dump.sh
   /usr/local/sbin/db-dump.sh
   ls -lh /var/backups/db

   If you see Access denied for user 'root'@'localhost', the database's root user requires a password. Put it in the /root/.my.cnf file, with the content below, then restrict access to the file:

   [client]
   user=root
   password=DB_PASSWORD

   chmod 600 /root/.my.cnf

4. Schedule the script for every night at 02:00 (server time). Create the file:

   nano /etc/cron.d/db-dump

   and paste the line:

   0 2 * * * root /usr/local/sbin/db-dump.sh >> /var/log/db-dump.log 2>&1

Files in /etc/cron.d have a user column (root), their names must not contain dots, and the last line must end with Enter. The % character can't be written directly in a cron line; that's why the date is calculated in the script. If the job doesn't run, check that the cron service is installed and running.

Ubuntu and Debian:

apt update
apt install -y cron
systemctl enable --now cron

AlmaLinux and CentOS Stream:

dnf install -y cronie
systemctl enable --now crond

Dumps take up disk space. If the disk is small, keep fewer days (for example -mtime +3): The server's disk is full.

Step 2: an off-server copy with restic

restic encrypts the data on the server, before sending it, keeps a history of copies (snapshots), and on each run transfers only what has changed. The example uses S3-compatible object storage; the option with another server, over SFTP, is in the rsync section. Official documentation: restic.readthedocs.io.

Installation

Ubuntu and Debian:

apt update
apt install -y restic

AlmaLinux and CentOS Stream (restic comes from the EPEL repository):

dnf install -y epel-release
dnf install -y restic

The repository and the password

1. At your storage provider, create a bucket and a pair of access keys. Note down the service address (endpoint), the bucket name, the key ID and the secret key.

2. Generate the repository password and display it:

   mkdir -p /etc/restic
   chmod 700 /etc/restic
   openssl rand -base64 32 > /etc/restic/password
   chmod 600 /etc/restic/password
   cat /etc/restic/password

3. Copy the displayed password into your password manager.

4. Create the settings file:

   nano /etc/restic/env

   RESTIC_REPOSITORY=s3:https://S3_ENDPOINT/BUCKET_NAME/SERVER_NAME
   RESTIC_PASSWORD_FILE=/etc/restic/password
   AWS_ACCESS_KEY_ID=KEY_ID
   AWS_SECRET_ACCESS_KEY=SECRET_KEY

   SERVER_NAME is a folder in the bucket; give each server its own folder. If the provider requires a region, add the line AWS_DEFAULT_REGION=REGION.

5. Restrict access to the file, load the settings into the current session and initialize the repository (only once):

   chmod 600 /etc/restic/env
   set -a
   . /etc/restic/env
   set +a
   restic init

   The response starts with created restic repository.

Risk Without the repository password, nobody can decrypt the copies, not even BLACKBOX. Keep the password outside the server too: if the server is lost, you can't restore anything without it.

The backup script

nano /usr/local/sbin/restic-backup.sh

#!/bin/bash
set -euo pipefail
set -a
. /etc/restic/env
set +a
restic backup /etc /root /var/www /var/spool/cron /var/backups/db --exclude-caches
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-tag manual --prune

Make the script executable, run it once and look at the list of copies:

chmod 700 /usr/local/sbin/restic-backup.sh
/usr/local/sbin/restic-backup.sh
restic snapshots

Running it daily with a systemd timer

1. Create the service:

   nano /etc/systemd/system/restic-backup.service

   [Unit]
   Description=Restic backup
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   Type=oneshot
   Environment=HOME=/root
   ExecStart=/usr/local/sbin/restic-backup.sh

2. Create the timer:

   nano /etc/systemd/system/restic-backup.timer

   [Unit]
   Description=Daily restic backup
   
   [Timer]
   OnCalendar=*-*-* 03:00:00
   RandomizedDelaySec=30min
   Persistent=true
   
   [Install]
   WantedBy=timers.target

3. Enable the timer and check the next run, in the NEXT column:

   systemctl daemon-reload
   systemctl enable --now restic-backup.timer
   systemctl list-timers restic-backup.timer

4. Run the service once and read the log:

   systemctl start restic-backup.service
   journalctl -u restic-backup.service -n 50

The backup starts every day between 03:00 and 03:30 (server time), after the 02:00 dumps. With Persistent=true, a run missed while the server was stopped takes place at the next startup.

The simple option: rsync to another server

If you have a second server, rsync copies directories to it over SSH. rsync must be installed on both servers. How to install it and its options: Transferring files to the server (scp, rsync, SFTP).

1. On the main server, create an SSH key used only for backups, without a passphrase, so that cron can use it:

   ssh-keygen -t ed25519 -f /root/.ssh/backup_ed25519 -N ""
   cat /root/.ssh/backup_ed25519.pub

2. Add the displayed line to /root/.ssh/authorized_keys on the backup server. If that server is also at BLACKBOX, use “Add SSH key” on its card: Adding and removing SSH keys on the server.

3. Create the destination directory and run the first copy by hand. On the first connection, confirm the backup server's key with yes:

   ssh -i /root/.ssh/backup_ed25519 root@BACKUP_SERVER_IP "mkdir -p /backups/SERVER_NAME"
   rsync -aR --delete -e "ssh -i /root/.ssh/backup_ed25519" /etc /var/www /var/backups/db root@BACKUP_SERVER_IP:/backups/SERVER_NAME/

   With -R, the full paths are kept: on the backup server you get /backups/SERVER_NAME/etc, /backups/SERVER_NAME/var/www and so on.

4. Schedule the copy daily at 03:30, in the /etc/cron.d/backup-rsync file:

   30 3 * * * root rsync -aR --delete -e "ssh -i /root/.ssh/backup_ed25519" /etc /var/www /var/backups/db root@BACKUP_SERVER_IP:/backups/SERVER_NAME/ >> /var/log/backup-rsync.log 2>&1

rsync keeps only the latest state: with --delete, a file deleted or damaged on the server also disappears from the copy at the next run. In addition, the files on the backup server are not encrypted. For history and encryption, use restic over SFTP to the same server. Add to /root/.ssh/config:

Host backup
    HostName BACKUP_SERVER_IP
    User root
    IdentityFile /root/.ssh/backup_ed25519

The /etc/restic/env file then contains only these two lines:

RESTIC_REPOSITORY=sftp:backup:/backups/SERVER_NAME-restic
RESTIC_PASSWORD_FILE=/etc/restic/password

The other steps are the ones in the restic section: installation, password, initialization, script and timer.

Encrypting off-server copies

• restic encrypts everything with the repository password before sending it. The storage provider sees only encrypted data.
• SSH, and therefore rsync and scp, encrypts data only in transit. If the destination doesn't belong to you, encrypt the files before copying them, for example with gpg and a password kept in a file (the gnupg package on Ubuntu and Debian, gnupg2 on AlmaLinux and CentOS Stream):

openssl rand -base64 32 > /root/.backup-pass
chmod 600 /root/.backup-pass
gpg --batch --yes --pinentry-mode loopback --passphrase-file /root/.backup-pass --symmetric --cipher-algo AES256 -o FILE.gpg FILE

To decrypt:

gpg --batch --pinentry-mode loopback --passphrase-file /root/.backup-pass -o FILE -d FILE.gpg

Keep the contents of /root/.backup-pass in your password manager too.

Security The /etc/restic/env and /etc/restic/password files and the /root/.ssh/backup_ed25519 key give access to the copies; the key even gives root access to the backup server. If the server is compromised, an attacker can use them to delete the copies. Also keep a copy that the server can't delete, for example one you regularly download to your computer.

Test a restore at least once a month

A copy you have never restored from is not a guarantee. Once a month, and after any change to the scripts, restore to a temporary directory, not over the data in use.

1. Load the settings, then check the list of copies and the integrity of the repository:

   set -a
   . /etc/restic/env
   set +a
   restic snapshots
   restic check

   The latest copy must be from the last 24 hours, and restic check must finish with no errors were found.

2. Restore the configuration and the dumps to a temporary directory:

   restic restore latest --target /root/restore-test --include /etc --include /var/backups/db
   ls -lh /root/restore-test/var/backups/db

3. Load the dump into a test database and check the tables and a few recent records (for example the latest order or the latest published article). Run the last command in the block only after you've checked: it deletes the test database. Replace YYYY-MM-DD with the date in the file name.

   MariaDB and MySQL:

   mysql -e "CREATE DATABASE restore_test"
   gunzip < /root/restore-test/var/backups/db/DB_NAME-YYYY-MM-DD.sql.gz | mysql restore_test
   mysql -e "SHOW TABLES" restore_test
   mysql -e "DROP DATABASE restore_test"

   PostgreSQL:

   sudo -u postgres createdb restore_test
   sudo -u postgres pg_restore -d restore_test < /root/restore-test/var/backups/db/DB_NAME-YYYY-MM-DD.dump
   sudo -u postgres psql -d restore_test -c "\dt"
   sudo -u postgres dropdb restore_test

4. Delete the temporary directory:

   rm -rf /root/restore-test

A restore takes up disk space, so restore only what you check (--include). The safest test is a full restore on another server, with the application started: it is also a rehearsal for a real incident. With the rsync option, copy the dump back from the backup server and load it the same way.

An extra copy before risky operations

Make a new dump and a restic copy with the manual tag, which forget doesn't delete:

/usr/local/sbin/db-dump.sh
set -a
. /etc/restic/env
set +a
restic backup /etc /root /var/www /var/spool/cron /var/backups/db --exclude-caches --tag manual
restic snapshots --tag manual

When you no longer need it, delete it using the ID from the ID column:

restic forget SNAPSHOT_ID --prune

Before you delete a server, save the repository password and the settings from /etc/restic/env outside it: without them, you can no longer reach the copies. Also download a copy to your computer, with rsync or scp.

Frequently asked questions

Can BLACKBOX restore my server if I deleted something by mistake? No. The panel doesn't make automatic copies and has no snapshots, and a deleted server can't be recovered. Restore from your own copies.

What does the backup_extra_disabled message under the “Backups” button mean? It means that the automatic backup service is not available. It is not a fault of the server, and the data is not affected.

How do I restore a single deleted file? Load the restic settings, as in step 1 of the restore test. Restore the file to a temporary directory, then copy it back into place. Don't restore directly with --target /: you would overwrite files in use. To pick an older copy, use its ID from restic snapshots instead of latest.

restic restore latest --target /root/restore-test --include /var/www/site/config.php
cp -a /root/restore-test/var/www/site/config.php /var/www/site/config.php

I have a “VPS hosting” plan with a control panel. What should I do? The installed control panel may have its own backup feature; ask support what it offers: “VPS hosting” plans with a control panel. Either way, the copy must end up outside the server.

I lost the restic repository password. What should I do? If the /etc/restic/password file still exists on the server, the password is there: save it to your password manager again. If the password no longer exists anywhere, the copies in that repository can no longer be read. In that case, create a new repository, with a different password and a different folder in RESTIC_REPOSITORY, and make a full copy right away.

Can I ask the team to set up the backups? If you'd rather not set them up yourself, ask support about server administration on request, with a separate offer: Professional services on request: administration, DevOps, migration.