snackup (was: Re: [gentoo-user] backup to a cold-swap drive)

[Alex Schuster]

Iain Buchanan writes:

> On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote:

> > using this script, adapted to their needs, I started to rewrite it in
> > a way that it reads a config file, and no modification of the script
> > itself is necessary. If anyone is interested, send me an email.
>
> interested! So is it on sourceforge yet ;)

Sorry, it took a little longer. I was ill, and then these scripts tend to 
grow and grow until I am satisfied with them. It's still not perfect, but 
I think at the moment it does what it should do.

Still not on sourceforge, but here: http://www.wonkology.org/utils/snackup
The name is silly, but I kinda like it now. A backup utilizing LVM 
snapshots... snapshot backup... snackup!

It needs a config file that is looked up in some default locations, or can 
be specified with the -c option. Use option -T to generate a template that 
has information on the syntax and some examples. In short:

The config file defines targets you give as arguments to snackup. So 
'snackup home' backs up your /home partition, 'snackup etc' creates a .tgz 
file of your /etc. The config file is sourced, so you can put any bash 
stuff you want into there, like some shell functions you want to have 
executed. For example, I want my /var/log/portage/*.log files to be 
compressed before I backup my /var partition.

A target is started by a colon, followed by name, type, source and 
destination (and optionally more), separated by one or more tab 
characters. Type can be 'cp', 'dd', 'tar', 'rsync' or 'rdiff'. I only 
tested dd, tar and rdiff so far, though. The source directory may be 
prefixed by the volume group and LVM of that partition in order to create 
an lvm snapshot first. You may add a LUKS key if the partition is 
encrypted. The source may contain wildcards. In that case, all matching 
files are backed up to the destination. If the type has a * appended, all 
matching files are backed up individually. Some examples:

:	tar 		kde		.kde*	/backup/kde.%s

Target 'kde' will backup all your .kde* directories to /backup/kde.tar 
(the %s is replaced by a suffix, usually tar). You need to be in your home 
directory, otherwise the path will not be found. This is not necessary if
you use ~//.kde* instead.

:	kernels	tar*	/usr/src//linux-*		/backup/src/%f.%s		-j	-	lsf

Target 'kernels' will backup your /usr/src/linux-* directories 
individually - one tar file for each (note the * appendended to the tar 
target type). The -z option will be passed to the tar command, and the 
suffix is changed to tgz accordingly.
The // denotes that the backup should be done locally from the directory 
left to the //. Otherwise, the full path would be included in the tar 
file.

:	tar		etc		/etc		/backup/etc_%d.%s	-j

Target 'etc' will tar your /etc directory, compressed with bzip2. The %d 
will be replaced by the current date. You can change the date format by 
re-defining a date() shell function.
You will get a message about tar removing the leading /, you can avoid it 
by replacing /etc by ///etc, meaning that your local directroy will be /.

:	boot		dd		/dev/sda5		/dev/sdb5		bs=32M

Target 'boot' will backup a boot partition with dd. Options like the 
bs=32M are added directly to the dd command.

:	home		rsync	system/home::.	/backup/home/

Target 'home' will create a LVM snapshot of the /dev/system/home logical 
volume, mount it and back it up with rdiff-backup. The '.' could also be a 
/ or left off.

:	var		rsync	system/var::/	/backup/var/	-	ziplog

Similar, but the command ziplog will be executed before the backup, 
compressing some log files. It must be defined in the config file.

The script has some options:

  -c file location of config file
  -C      clear destination first
  -d      dummy mode, just show what would be done
  -f      force backup (the initial rdiff-backup may need this)
  -h      show this help
  -l      output to log file, too
  -L      log file (default:/home/wonko/log/snackup.log); may be a
          directory (add a trailing slash) to create target-specific logs
  -n l    use nice level l (default:10, 0 to turn off)
  -N l    use ionice level l (default:3, 0 to turn off)
  -o      extra options that will added to the actual backup command
  -s size size of LVM snapshot (default:2G)
  -S char replace tab as delimiter for targets in config file
  -T      output config file template
  -v      verbose output; may be given multiple times
          1: some extra output; 2: add -v to commands; 3: set -xv
  -V      output version information and exit

Most important are:
  -c to specify the location of the config file
  -d to output what would be done, you will see the final tar / rdiff-
backup / whatever command with all its options.
  -T to see a template. Save it with snackup -T > ~/.snackup and then 
modify your ~/.snackup file.
  -o to add whichever options you like to the final command. These (and -j 
and -z) override options already defined in the config file.

Too complicated? Overkill? Maybe. But I like it, and now I do not have to 
write another script for every system I want to backup. And I like the LVm 
snapshotting to be done automatically. It's not that hard, but it's 
annoying to do it by hand, even more as all my volumes are encrypted.

These are my own targets:

:	boot		dd		/dev/boot					/dev/sdb5						-	-		-
:	backup	rdiff	weird/backup::/etc/key::.	/backup/weird/backup			-	-		-
:	local	rdiff	weird/local::/etc/key::.	/backup/weird/local			-	-		-
:	home		rdiff	weird/home::/etc/key::.	/backup/weird/home				-	-		-
:	med		rdiff	weird/med::/etc/key::.		/backup/weird/med				-	-		-
:	mp3		rdiff	weird/mp3::/etc/key::.		/backup/weird/mp3				-	-		-
:	mpeg		rdiff	weird/mpeg::/etc/key::.	/backup/weird/mpeg				-	-		-
:	opt		rdiff	weird/opt::/etc/key::.		/backup/weird/opt				-	-		-
:	root		rdiff	weird/root::/etc/key::.	/backup/weird/root				-	-		-
:	usr		rdiff	weird/usr::/etc/key::.		/backup/weird/usr				-	-		-
:	var		rdiff	weird/var::/etc/key::.		/backup/weird/var				-	zipLog	-
:	etc		tar		///etc					/backup/conf/etc/etc_%d.%s		-z	-		lsf
:	kernels	tar*	/usr/src//linux-*				/backup/weird/src/%f.%s		-j	-		lsf
:	portage	rsync	/var/portage/packages		/backup/weird/portage			-	-		-
:	kde		tar	~//.kde*						/backup/wonko/home/kde-%d.%s	-z	-		lsf
:	dot		tar	~//.??*						/backup/extern/dot-%d.%s		-z	-		lsf


At last, the disclaimer: Use it at your own risk! This is a script of some
800 lines of bash code some guy wrote, it messes around with LVM stuff,
creates and removes temporary directories and even has an option to wipe
the destination completely. I'm quite confident that no harm will be done,
and I do not suggest you to take a backup before daring to try this script,
but anyway.

Use the -d option first to test what actually would be going on, before you
do a real backup!

Hope this is of good use to someone. For me it is, I will be using it on 
several systems, and now I only have to make little changes in the config
file for each system, but not at the script.

Suggestions and bug reports are welcome of course.

	Wonko