Re: snackup (was: Re: [gentoo-user] backup to a cold-swap drive)

Alex Schuster <wonko@×××××××××.org> wrote:

> Iain Buchanan writes:
>
> > On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote:
>
> > > using this script, adapted to their needs, I started to rewrite it in
> > > a way that it reads a config file, and no modification of the script
> > > itself is necessary. If anyone is interested, send me an email.
> >
> > interested! So is it on sourceforge yet ;)
>
> Sorry, it took a little longer. I was ill, and then these scripts tend to
> grow and grow until I am satisfied with them. It's still not perfect, but
> I think at the moment it does what it should do.
>
> Still not on sourceforge, but here: http://www.wonkology.org/utils/snackup
> The name is silly, but I kinda like it now. A backup utilizing LVM
> snapshots... snapshot backup... snackup!
>
> It needs a config file that is looked up in some default locations, or can
> be specified with the -c option. Use option -T to generate a template that
> has information on the syntax and some examples. In short:
>
> The config file defines targets you give as arguments to snackup. So
> 'snackup home' backs up your /home partition, 'snackup etc' creates a .tgz
> file of your /etc. The config file is sourced, so you can put any bash
> stuff you want into there, like some shell functions you want to have
> executed. For example, I want my /var/log/portage/*.log files to be
> compressed before I backup my /var partition.
>
> A target is started by a colon, followed by name, type, source and
> destination (and optionally more), separated by one or more tab
> characters. Type can be 'cp', 'dd', 'tar', 'rsync' or 'rdiff'. I only
> tested dd, tar and rdiff so far, though. The source directory may be
> prefixed by the volume group and LVM of that partition in order to create
> an lvm snapshot first. You may add a LUKS key if the partition is
> encrypted. The source may contain wildcards. In that case, all matching
> files are backed up to the destination. If the type has a * appended, all
> matching files are backed up individually. Some examples:
>
> :	tar 		kde		.kde*	/backup/kde.%s
>
> Target 'kde' will backup all your .kde* directories to /backup/kde.tar
> (the %s is replaced by a suffix, usually tar). You need to be in your home
> directory, otherwise the path will not be found. This is not necessary if
> you use ~//.kde* instead.
>
> :	kernels	tar*	/usr/src//linux-*		/backup/src/%f.%s		-j	-	lsf
>
> Target 'kernels' will backup your /usr/src/linux-* directories
> individually - one tar file for each (note the * appendended to the tar
> target type). The -z option will be passed to the tar command, and the
> suffix is changed to tgz accordingly.
> The // denotes that the backup should be done locally from the directory
> left to the //. Otherwise, the full path would be included in the tar
> file.
>
> :	tar		etc		/etc		/backup/etc_%d.%s	-j
>
> Target 'etc' will tar your /etc directory, compressed with bzip2. The %d
> will be replaced by the current date. You can change the date format by
> re-defining a date() shell function.
> You will get a message about tar removing the leading /, you can avoid it
> by replacing /etc by ///etc, meaning that your local directroy will be /.
>
> :	boot		dd		/dev/sda5		/dev/sdb5		bs=32M
>
> Target 'boot' will backup a boot partition with dd. Options like the
> bs=32M are added directly to the dd command.
>
> :	home		rsync	system/home::.	/backup/home/
>
> Target 'home' will create a LVM snapshot of the /dev/system/home logical
> volume, mount it and back it up with rdiff-backup. The '.' could also be a
> / or left off.
>
> :	var		rsync	system/var::/	/backup/var/	-	ziplog
>
> Similar, but the command ziplog will be executed before the backup,
> compressing some log files. It must be defined in the config file.
>
> The script has some options:
>
>   -c file location of config file
>   -C      clear destination first
>   -d      dummy mode, just show what would be done
>   -f      force backup (the initial rdiff-backup may need this)
>   -h      show this help
>   -l      output to log file, too
>   -L      log file (default:/home/wonko/log/snackup.log); may be a
>           directory (add a trailing slash) to create target-specific logs
>   -n l    use nice level l (default:10, 0 to turn off)
>   -N l    use ionice level l (default:3, 0 to turn off)
>   -o      extra options that will added to the actual backup command
>   -s size size of LVM snapshot (default:2G)
>   -S char replace tab as delimiter for targets in config file
>   -T      output config file template
>   -v      verbose output; may be given multiple times
>           1: some extra output; 2: add -v to commands; 3: set -xv
>   -V      output version information and exit
>
> Most important are:
>   -c to specify the location of the config file
>   -d to output what would be done, you will see the final tar / rdiff-
> backup / whatever command with all its options.
>   -T to see a template. Save it with snackup -T > ~/.snackup and then
> modify your ~/.snackup file.
>   -o to add whichever options you like to the final command. These (and -j
> and -z) override options already defined in the config file.
>
> Too complicated? Overkill? Maybe. But I like it, and now I do not have to
> write another script for every system I want to backup. And I like the LVm
> snapshotting to be done automatically. It's not that hard, but it's
> annoying to do it by hand, even more as all my volumes are encrypted.
>
> These are my own targets:
>
> :	boot		dd		/dev/boot					/dev/sdb5						-	-		-
> :	backup	rdiff	weird/backup::/etc/key::.	/backup/weird/backup			-	-		-
> :	local	rdiff	weird/local::/etc/key::.	/backup/weird/local			-	-		-
> :	home		rdiff	weird/home::/etc/key::.	/backup/weird/home				-	-		-
> :	med		rdiff	weird/med::/etc/key::.		/backup/weird/med				-	-		-
> :	mp3		rdiff	weird/mp3::/etc/key::.		/backup/weird/mp3				-	-		-
> :	mpeg		rdiff	weird/mpeg::/etc/key::.	/backup/weird/mpeg				-	-		-
> :	opt		rdiff	weird/opt::/etc/key::.		/backup/weird/opt				-	-		-
> :	root		rdiff	weird/root::/etc/key::.	/backup/weird/root				-	-		-
> :	usr		rdiff	weird/usr::/etc/key::.		/backup/weird/usr				-	-		-
> :	var		rdiff	weird/var::/etc/key::.		/backup/weird/var				-	zipLog	-
> :	etc		tar		///etc					/backup/conf/etc/etc_%d.%s		-z	-		lsf
> :	kernels	tar*	/usr/src//linux-*				/backup/weird/src/%f.%s		-j	-		lsf
> :	portage	rsync	/var/portage/packages		/backup/weird/portage			-	-		-
> :	kde		tar	~//.kde*						/backup/wonko/home/kde-%d.%s	-z	-		lsf
> :	dot		tar	~//.??*						/backup/extern/dot-%d.%s		-z	-		lsf
>
>
> At last, the disclaimer: Use it at your own risk! This is a script of some
> 800 lines of bash code some guy wrote, it messes around with LVM stuff,
> creates and removes temporary directories and even has an option to wipe
> the destination completely. I'm quite confident that no harm will be done,
> and I do not suggest you to take a backup before daring to try this script,
> but anyway.
>
> Use the -d option first to test what actually would be going on, before you
> do a real backup!
>
> Hope this is of good use to someone. For me it is, I will be using it on
> several systems, and now I only have to make little changes in the config
> file for each system, but not at the script.
>
> Suggestions and bug reports are welcome of course.
 
I have a question -- where would lvm put a snapshot and how could I pass
some list of excludes to rdiff-backup.  I have an lvm which is taking
all the PEs and a snapshot would take up lots of disk space -- or would
it.  Would I need some free pes to put the snapshot?

Thanks.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@××××××××××.com