CPI

Automatically create (incremental) backups of zfs snapshots on a file server.

Holger Frey 37534861d9 working on comparing checksums		5 years ago
.gitignore	ignoring temp and key files	5 years ago
README.md	fixed typos in readme	5 years ago
compare_checksums.py	working on comparing checksums	5 years ago
copy_snapshots_to_remote.py	fixed weird bug on scp	5 years ago
create_snapshot_backup.py	working on comparing checksums	5 years ago
old_backup_script	first test	5 years ago
run_snapshot_backups.sh	fixed bug in removing copied archives	5 years ago

README.md

zfs-snapshot-backup

Automatically create (incremental) backups of zfs snapshots on a file server.

This is currently tailored only to our setup of a FreeNAS in combination with our virtual server and storage provided by the Rechenzentrum.

The script will create backups of ZFS snapshots for elab journals. The name of the ZFS dataset (from wich the snapshots are created) must start with elabfs- followed by the elab member name, e.g. LukasMetzler

The snapshots should be made with the periodic snapshot task provided by the FreeNAS system. Snapshots should be kept on the NAS for a couple of days (currently one week) to allow transfer errors not to interfere with the backup process.

The backup process should be run after a snapshot is taken. Proposed is to take a daily snapshot at night (currently between 2am and 3am) and start the backup process later via cron (currently set to 4am).

The documentation in our wiki can be found at: https://wiki.cpi.imtek.uni-freiburg.de/CPIvServerDocumentation/ElabOnZfs

restoring backups

copy a snapshot backup to the NAS

cd /mnt/Datenspeicher/snap-backup-dataset/temporary-backups/
scp -i ../backup_key zfs_snap_backup@etha.cpi.imtek.uni-freiburg.de:~/zfs-backups/<member-name>/<backup-file> tmp-backup-file.gz

unzip the backup file and restore it into a dataset

gunzip -c tmp-backup-file.gz | zfs receive -F Datenspeicher/test-backup

remove the (local) backup file
```
rm tmp-backup-file.gz
```

notes on implementation

At first I tried to implement the complete process in one python file, but this lead to an unforseen (and very strange) error:

The created gzip file was not completely written before copying it to the remote server. Even closing the file descriptor manually or waiting for a very long time would not solve the problem. The gzip was only completed if the python process was stopped.

Therefore creating the backups and copying them to the remote server is now a two step process, combined in the run_snapshot_backups.sh shell script.

I have the idea, that it has todo with the copy on write feature of zfs, but I'm not sure about it.