[gentoo-amd64] Backup techniques Was: audio/vidio apps and glibc-2.5?

[Duncan]

"Jamie" <gentoo@×××××××.nz> posted
4049.210.55.22.193.1161647305.squirrel@××××××××××××××××××.com, excerpted
below, on  Tue, 24 Oct 2006 12:48:25 +1300:

> A good piece of advice and one that I really should follow. What is the
> best way to take an image of the Gentoo install? In my case my Gentoo
> install resides on /dev/hda2 (boot) ; /dev/hda3 (swap) and /dev/hda5
> (root) - is it possible to use something like dd to image these partitions
> to a disk somewhere else on my network? Is this a good/bad/non-workable
> idea?
>
> I would welcome any constructive advise on this as the many hours of
> building a Gentoo box is not something I want to do too often so being
> able to image prior to any major upgrades would be a great safety net...

There are of course several different alternatives.  I've found what works
for me, and it has been decently tested in recovery situations.

#1 rule for me was that it needed to be relatively convenient, or I'd not
use it regularly.

#2, as you, hard disks seem to be the most cost effective here for nearly
whole-system backup, and are reasonably reliable if you treat them right. 
I do backup choice bits multiple times, often multiple disk backup
entries, but also burned to CD/DVD, particularly where the data isn't
likely to be changing over long periods.

#3 works as a result of 1 and 2:  I'm able to boot directly into my backup
system and resume from there using it as my main system.  Something like
tape backup simply doesn't work that way.  Neither do emergency boot
LiveCDs and the like, tho they have their place as a "worst case situation"
solution, if my main backups fail.

#4, my system scales.  I used it as multiple partitions on a single hard
drive and/or a second hard drive for years, including recovering from two
partial hard drive failures.  The last one was caused by overheating after
my A/C went out. (Phoenix temps can reach 50C/122F outdoor, so after the
A/C went out I figure inside temps like reached 60C/140F, possibly higher.
The CPUs' thermal protection kicked in, but the hard drive understandably
just wasn't the same after that.)  I decided to switch to a RAID system
for additional reliability, and my backup system scaled right along with
it.  I'm now running 4 x 300 gig Seagate SATA drives, slightly slower than
some but a full five year warrantee, using kernel RAID, additional details
below.

Here's how it works.  I have my system divided up into several protection
levels, depending on the content.  

0: Temporary or easily redownloadable content need not be backed up at all.

This includes /tmp, /var/tmp, the portage tree, /usr/src, and my ccache.
All these can be kept in separate directories on the same "partition",
using either config settings (for ccache and the portage tree) or symlinks
(/tmp, /var/tmp).

On a single drive, this will be a single partition, not duplicated.  On a
RAID system, this works very well as RAID-0/striped.  Striped RAID is very
fast, which happens to fit the use of much of this data quite well, but if
one disk goes down, it takes the entire RAID-0 with it. That's just fine,
as this is temporary data anyway, either throw-away or easily
redownloadable/recoverable.  IOW, this is the /perfect/ case for
RAID-0/striped.

1: boot area (MBR and /boot, with GRUB and kernels).

On a single drive, you should keep a removable (ISO, thumb
drive, zip drive, whatever) around with at least GRUB/LILO and a kernel or
two, in case the hard drive's boot area and/or /boot fail.  If you have
two drives, keep a tested/working GRUB and /boot with fairly current
kernels on both drives.  In a RAID installation, RAID-1/mirrored /boot,
and install and test-boot from GRUB on all drives in the mirror set.  Note
that RAID-1/mirroring will provide reasonable protection against physical
drive failure, but it will **NOT** protect against "sysadmin fat-finger
syndrome", since it's still only a single copy of /boot.  Thus, you'll
still want to keep a removable media backup boot solution available.  For
ease of update, a USB thumb-drive is probably the most convenient (128MB
or even 64MB should work), altho a CDRW/DVDRW would work, as would a
bootable zip drive and disk, altho they are somewhat passe by now.

Again, this will likely be a single "partition" or unpartitioned RAID-1
device.

Everything else: data that likely needs some level of redundancy
protection.

On a RAID system, it will be RAID protected, but likely at
RAID-5, 6 or 10.  (Here I use RAID-6.  Again, more detail below.)  For
higher value data, however, just the redundancy of the RAID itself isn't
enough, due to the previously mentioned "sysadmin fat-finger syndrome",
or tho we don't have to worry about it so much on *ix, malware or the
like.  Single and double disk systems will want to backup some but not all
of this data.

This "everything else" can be further divided into three levels, system,
high-value data, mid-value data, with multiple partitions accordingly.

mid-value data:  This consists of stuff like /var/log.  It may also
include content known retrievable from the net, but at some cost in time
and trouble, and other system/purpose specific data that's not extremely
difficult to replace but would require enough time or expense to do so
that you can't rightly call it "temporary".

On a single or double disk system, mid-value data will likely be treated
for purposes of backup much like temporary data -- ignore it.  The hassle
of backing it up is likely to be more work than the value of the recovered
data.  However, on RAID systems, there's a distinction between the two in
that mid-value data will be considered valuable enough to protect it with
the redundancy of RAID, while temporary data is throw-away even at that
level.  Even so, the redundancy of the RAID will normally be considered
safety enough -- no additional backup efforts worth it.  That remains true
despite the risk of "sysadmin fat-finger syndrome", since the risk is
relatively low compared to the value of the data and the hassle of
additional backup arrangements.

A single operational copy of mid-value data will be all that's tracked,
but as the amount and filesystem layout of this data will vary greatly
across individual purposes and installations, it may be one partition,
many, or one or more logical volumes managed with LVM2 or similar.  (More
on that below.)

System data:  This is the critical operational system.

For least trouble managing it, the goal in Gentoo terms should be to
include everything installed by portage, together with the portage
database (in /var/db), and all system level configuration, all on the same
(single) partition, BUT CRITICALLY, TO MAKE THREE IMAGES, three separate
partitions, imaging essentially similar data.

>From experience, I know keeping everything installed by portage together
with the portage database and all system level config data, is the
simplest to manage.  While it's /possible/ to split off for example /var
and /usr, that's needless additional complexity.  The idea here is that
everything should stay in sync.  If you lose /usr, the portage database of
what's installed that's located in /var/db won't be very useful. 
Similarly if / goes bye bye and you boot from "rootbak1", which has older
binaries in /bin and an older /etc configuration, but still matched
against the new /usr.  That's no good and only causes headaches -- I've
had it happen!  Thus, keep the portage installation database in /var/db in
sync with /usr, /, and /etc, by keeping everything on the same
"partition", and you save yourself VAST amounts of trouble!  <<  That last
sentence is worth rereading.  I KNOW from experience!  FWIW, I'm running
10 gig "system image" partitions, giving me plenty of room as I'm only
using 1.5 gig according to df, so 5 gig system partitions should be fine
on most systems, I'd think.

As I mentioned above, ideally you have a minimum of THREE system images,
identically sized.  One will be your "working" system.  The other two (and
additionals if desired) are alternating backup snapshots.  Here, I have no
set schedule for system backups, though every three-ish months might be a
reasonable goal.  When I think everything's working particularly well,
I'll just erase the oldest backup and copy the working system image over,
creating a new snapshot.  Two backups minimum is safest, as that means
even if disaster strikes when I'm in the middle of one, and I lose the
main working image before the backup is complete, I still have a known
working second backup I can boot to.  No sweat!  Of course more system
images lets you have more backups, effectively letting you take snapshots
more frequently as anything over say 9 months old will likely be nearly as
much work to update as it would be to install from scratch, so there's a
practical limit to how old you want to go, and more images simply means
you can take snapshots more frequently within that limit.

It's worth point out again what I mentioned as #3 requirement/advantage
above.  With my system, any of the system snapshots is pretty much self
contained and ready to boot, just as it was when the snapshot was copied
over.  No fiddling around restoring stuff if you are up against a deadline
and have to have a working system NOW.  If the normal/working system image
fails, you simply boot one of your snapshots and continue where you left
off.  You have exactly the same set of applications, with exactly the same
installation specific configuration, as you did when you took the
snapshot, so you are immediately up and running, only possibly having to
think back on how it was configured back then if you changed something
since then.

On a single disk system, you may be able to get away with two system
images, but you lose the additional safety of the second backup.  On a
dual disk system, if space permits, you almost certainly want at least one
backup image on each disk, the main/working image and one backup on
presumably the bigger disk, the other backup on the other disk, thus
maintaining a fully working system if either disk goes out.  On a RAID
system, you'll probably want your images in RAID-5, -6, or -10, /possibly/
RAID-1/mirrored if you have only a two-disk RAID or need the extreme
redundancy.

Taking a paragraph out to discuss RAID for a moment.  RAID-0 as mentioned
is striped for speed, with speed scaling as more devices are added, up to
the speed of the buses they are attached to, but offers no redundancy, so
any disk out and you lose it all. RAID-1 as mentioned is mirrored for
redundancy -- the data is simply copied to each device in the RAID, and
the RAID can continue to function as long as even a single device
survives.  RAID-1 (kernel implemented, anyway) is slow writing as the data
must be written once to each device, but slightly faster than single-disk
for reading under single tasking scenarios, faster still under
multi-tasking i/o scenarios as the kernel will split the tasks between
devices. RAID 4,5,6 are variations on parity RAID. RAID-4 is devices minus
1 for data, with that one device dedicated to parity protection.  If any
data device goes down, the parity device can reconstruct the data, altho
it's slower. Read-speed is comparable to the number of data devices in a
striped array, but write-speed is bottlenecked by the parity device which
is always written, which is the reason RAID-4 isn't commonly used any
more.  RAID-5 is RAID-4 but with the parity stripe alternating between
devices, thus eliminating the write bottleneck of RAID-4.  Both read and
write speeds compare to a striped array of one less device.  RAID-6 is
double-parity, again alternating stripes.  Thus, a four-device RAID-6 array
is comparable in speed to a two-way striped array.  Any TWO devices can
die in a RAID-6, and it can still recover or even continue to operate
altho at reduced speed.  RAID-10 is a combination of RAID-0 and RAID-1,
mirrored and striped. There's also RAID-0+1 which is the reverse
combination but less efficient.  I can never keep straight which is which,
however, but that's fine as I'm not using it and remember the concept and
that if I DO find a use for it, that I need to look it up again.  Speeds
are typically half the speed of a striped array of the same size, but with
symmetric mirroring redundancy.  RAID-10 is typically used in large
arrays, where its efficiency is far better than RAID-5 or -6, while RAID-5
or -6 are typically used in smaller arrays or where total byte capacity is
more important than speed.  RAID-10 also maintains degraded (one or more
bad devices but still operational) speed better than RAID-5 and -6 do, so
is the better choice where continued operation at full speed even as
individual devices fail is a must.

As I mentioned, for my little four-disk RAID, RAID-6 seemed the best
compromise of redundancy and speed for my needs, so that's what I chose to
implement my "everything else" data protection on.  The practical effect
is a two-way striped array, in which I could lose any two of the four
drives and still have the data on the RAID intact.

Below I cover partitioned RAID vs. LVM2 managed volumes.  Here, I'll
simply mention that I have my system images implemented as partitioned
RAID, (mdd partitioned, vs. md and lvm2), as it *significantly*
decomplicates booting and recovery issues.

That leaves high-value data:  Installation specific data (such as /home
and /usr/local) for which the redundancy of RAID is NOT sufficient
protection.

As with system images, here you'll want backup images of the data even if
it's on RAID, to protect against "sysadmin fat-finger syndrome" among
other things.  Actual configuration and number of images will vary by
purpose and site, but the idea is always keep at least two copies of
every "partition". Single disk systems will want at least second partition
copies, and will likely want removable backup copies of the most critical
stuff.  Dual disk systems will likely want to keep one image of at least
the most critical stuff on each disk, just in case, thus lessening the
need for removable media backup (altho not eliminating it entirely for the
super-critical stuff).  Likewise for RAID systems -- a second image
protected by RAID is a must, but that and the RAID protection may well be
considered enough given the lowered risk and the hassle factor of
removable media backups, except for the most critical stuff, of course. 
In practice, it's likely that what would be DVDs worth of data needing
some sort of backup can be reduced to CDs or thumb-drives worth of data at
the critical value level worth backup beyond the second or third on-disk
image.

Here, I didn't worry much about number of partitions as I chose to manage
everything using LVM2 (Logical Volume Management), which maintains most of
the advantages of partitions (with an exception of direct bootability,
thus the lower suitability for bootable system images), but is VASTLY more
flexible.  It's out of scope to detail LVM2 further here, save to note that
even for single disk systems it could be worth the trouble to research it.

Now, a few more notes on implementation and practicality issues.

For general system management, I find mc aka midnight commander, an
extremely valuable tool in my sysadmin toolbox.  If you aren't familiar
with it yet but are interested in an ncurses based
file-manager/viewer/editor complete with scriptable macros and syntax
highlighting, it's certainly worth trying.  app-misc/mc .

As with most of my sysadmin tasks, I use mc when I'm working on my
partition images.  Its copy and move mode maintain attributes by default
and just do "The Right Thing" (TM) when it comes to symlinks, socket
and device files, and other assorted filesystem irregularities of the *ix
world.  What could be simpler than simply copying existing files from the
working location to the backup/snapshot location?  Using mc's two window
layout, it's that easy -- at least after the issue in the next paragraph
is dealt with, anyway.  Sure, one can use the archive switch and etc on a
command line copy, or do impressive stuff using tar, but for me, simply
copying it with mc accomplishes my objective with the least fuss and muss.

So what's that issue I mentioned?  Well, simply this.  Particularly for
the system image (/ fs with /etc and much of /usr and /var) upon which
everything else is mounted, simply copying won't /normally/ stop at the
system image (aka partition) boundary as we'd like in this case.  There's
also the issue of trying to copy the raw device files in /dev with udev
mounted there, and other similar things to worry about.

As it turns out, there's a relatively easy way around that.  Simply use
the mount --bind option to mount a view of the root filesystem (only,
without everything mounted on it recursively) elsewhere, and copy /that/
view of the filesystem, instead of trying to copy the normal one.  This is
easiest accomplished with an entry in fstab similar to the following:

# Dev/Part    MntPnt           Type    MntOpt        D F
# for mount --bind, --rbind, and --move              ###
# /old/dir    /new/dir         none    bind          0 0
/.            /mnt/rootbind    none    bind,noauto   0 0

Now, when I go to create a system image snapshot, I can simply
"mount /mnt/rootbind", and then simply copy (using mc, naturally)
everything from there to my new image snapshot.  Note that the dev/part
is listed as /.  Since there's already an entry for the root filesystem
mounted at /, I can't use that or mount gets confused.  Of course, /.
is self referencing, and this little trick keeps mount happy.  I had fun
trying to figure it out. =8^)

One more caveat:  A very few files may be read-locked under normal
operation and thus not able to be copied directly.  I've never run into
that here, but from what I understand, a typical example might be a
database file.  MySQL users and others running databases might want to pay
attention here, but as I said, I've never run into issues, and those
shouldn't be on the system image, but rather belong in the
high-value data section, likely in their own dedicated partition, which
should make special-casing that particular imaging task not too difficult.

OK, now a word about fstab management.  What I've found works best is to
keep three separate fstabs (or one for each system image), call them
fstab.working, fstab.bak1, and fstab.bak2, if you wish.  They will be
essentially the same except that the root filesystem will be switched
around with the (noauto optioned, so they don't automount) two backup
system image snapshots. Then on the working image, make fstab a symlink
pointing to fstab.working.  When you take a snapshot image, before you
unmount after the copying, simply point the fstab symlink at the
appropriate fstab.bak? file.  Each system image then has a copy of all
three files at the time the snapshot was taken, with the symlink adjusted
to point at the right fstab.* file so everything just works when that
image is booted.

Finally, a bit more detail on the lvm vs partitioned RAID choices I made
and why.  First, it's worthwhile to note that a lot of the documentation
still says (incorrectly) that Linux kernel RAID doesn't handle partitions.
Linux kernel RAID (and the corresponding mdadm, don't know about the older
raidtools) has worked with partitioned RAID devices since at least early
in the 2.6 cycle, and I think sometime in the 2.5 development kernel cycle.

Second, there's an important difference between the way Linux handles LVM2
and the way it handles RAID, including partitioned RAID.  The RAID device
drivers are entirely contained within the kernel (provided they are
configured as built-in, not modules), and can be configured on the kernel
command line (thus from grub), so one can boot RAID, including partitioned
RAID, directly.  Not so with LVM(both 1 and 2), which require userspace
configuration to work, thus complicating the root on LVM2 boot scenario
dramatically.  While it's still possible to use an initrd/initramfs
containing the LVM2 config for at least the root device, that's VASTLY
more complex than simply tweaking the kernel command line in grub.  Of
course, should things go wrong with LVM2, fixing the problem is likely to
be VASTLY more complex as well, while with RAID including partitioned
RAID, if it's fixable, it's fixable from GRUB by altering the kernel
command line or in the event of a wrongly configured kernel, simply
booting back to the old kernel that handled it correctly.

Thus, for ease of boot maintenance and recovery, I chose NOT to use LVM2
on my bootable system rootfs images.  Once that's up and mounted, I have
the full system with all the usual tools available to me to recover
or reconfigure additional RAID devices, or troubleshoot LVM2, if either
one is necessary.  Thus, no big deal having my mid-value and high-value
data on LVM2, where it's easy to manage the volumes than it would be
partitions.  I just don't want the additional complexity of having to
worry about LVM2 on my main root filesystem, whether working image or
backup images, since the purpose is to have all of them bootable, and
managing that is VASTLY less complex with direct kernel boot than it would
be if I had to worry about an initramfs to load LVM2.

So, after all the above, here as an example is how I have my system
configured (roughly):

For each drive, /dev/sd[a-d], same sized partitions on each of the four
300 gig SATA drives:

GRUB on the MBR

/dev/sdX1: partition forming part of a RAID-1 /dev/md0 for /boot

/dev/sdX2: partition forming part of a partitioned RAID-6 /dev/md_d1

/dev/sdX3: partition for swap, all four partitions pri=1, so the
           kernel stripes them automatically when it activates them

/dev/sdX4: extended partition, reserved/ignore

/dev/sdX5: partition forming part of a partitioned RAID=0 /dev/md_d2
           however I didn't actually need this one partitioned

/dev/sdX6 and beyond: ~100 gig remains free and unpartitioned on each 300
gig drive, to be allocated later as needed.

The partitions on the RAID-6 /dev/md_d1 are as follows:

/dev/md_d1p1: working system image
/dev/md_d1p2: /mnt/bak1 (first backup system image)
/dev/md_d1p3: /mnt/bak2 (second backup system image)
/dev/md_d1p4: reserved as an LVM2 physical volume group, vgraid6

vgraid6 has the following logical volumes:

/dev/vgraid6/home
/dev/vgraid6/homebk   (the backup /home image)
/dev/vgraid6/local    (this is /usr/local)
/dev/vgraid6/localbk
/dev/vgraid6/log      (/var/log, raided but not worth a backup image)
/dev/vgraid6/mail
/dev/vgraid6/mailbk
/dev/vgraid6/mm       (multimedia)
/dev/vgraid6/mmbk
/dev/vgraid6/news     (raided but no backup, again, mid-value data)
/dev/vgraid6/pkg      (portages package dir for FEATURES=buildpkg)
/dev/vgraid6/pkgbk

The RAID-0/striped /dev/md_d2p1 (the only partition I'm using)
mounts at /mnt/str (for striped).  Subdirs:

/mnt/str/ccache
/mnt/str/portage      (the portage tree, including distdir but not pkgdir)
/mnt/str/usrsrc       (for /usr/src/linux)

It would have /tmp as well except that with 8 gigs of memory, I mount /tmp
as tmpfs, maximum size 5 gig.  /var/tmp symlinks /tmp, and all portage's
compiles use the tmpfs for scratch space, dramatically speeding up merges!
=8^)

Obviously, I've spent quite some time devising a solution that fits my
needs.  There's only one big thing I'd change about my layout, and I
already accounted for that as described above.  I actually have only two
system images, working and a single backup.  Next time I reorganize
(remember, I've got a third of the space remaining on the drives, a
hundred gigs each, to do so), I'll create a third system image.  I'll
probably do something different with swap as well, since I only made those
partitions four gigs each and with 8 gigs of memory and the fact that
in-kernel suspend to disk writes to a single device/partition only, that's
not enough to properly suspend, even if the kernel's implementation /does/
work right (last I checked, back when I had only a gig of RAM, it didn't,
but there's some dramatic changes going on in that department so it might
now -- except I error out due to lack of room in the partition!).

While the above is my current RAID layout, I was using something very
similar back when I had only two disks, as described.  The layout now is a
bit more sophisticated, however.  In particular, I had /usr and /var on
their own partitions back then, but as I said, that lead to serious
problems when one died and they got out of sync with each other, thus the
warnings about that and the current layout folding those back into the
single system image.  The idea is solid and has demonstrated itself over
two partial drive failures, as I mentioned, before I decided I wanted the
reliability of RAID.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@g.o mailing list