Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Joost Roeleveld]

On Tuesday 27 May 2014 11:28:17 Rich Freeman wrote:
> On Tue, May 27, 2014 at 11:12 AM, J. Roeleveld <joost@××××××××.org> wrote:
> > On Tuesday, May 27, 2014 10:31:26 AM Rich Freeman wrote:
> >> btrfs wouldn't have any issues with this at all.  You'd have an
> >> advantage in that you wouldn't have to unmount the filesystem to
> >> cleanly create the snapshot (which you have to do with lvm).
> >
> > That, or a "sync" prior to creating the snapshot. :)
>
> If the filesystem is still mounted, I'm not sure that a sync is
> guaranteed to give you a clean remount.  It only flushes the
> caches/etc.  You need to remount read-only or unmount before doing the
> sync (and the sync probably isn't actually necessary as I'd think LVM
> would snapshot the contents of the cache as well).

I do this for the OS-partitions of my VMs.
In the VM, run "sync", then on the host, take an LVM snapshot and then mount 
the snapshot on the host.
I have not had any errors from this.

> > I have a yearly (full), monthly, weekly and daily. Each incremental is
> > against the most recent one of itself or longer period.
> > That means having to keep multiple snapshots active, which I prefer to
> > avoid.
> You only need to store snapshots for use with incremental backups.
> So, if all your backups are full, then you don't need to retain any
> snapshots (and you wouldn't use btrfs send anyway).  If your yearly is
> full and your monthlies are incremental against the yearly then you
> need to keep your yearly snapshot for a year.  If your yearly is full
> and your monthlies are incremental against the last month, then you
> only need to keep the yearly until the next monthly.  If your
> monthlies are full then you only need to keep the current monthly
> assuming your dailies are incremental against those, but if they're
> incremental from the last daily then you never need to keep anything
> for more than a day.

That makes for an interesting option. Not sure if I would implement it that 
way.

> > But, it is a good idea for backing up desktops and laptops.
>
> It is really intended more for something like datacenter replication.
> Snapshot every 5 min, send the data to the backup datacenter, delete
> the snapshots upon confirmation of successful receipt.  In such a
> scenario you wouldn't retain the sent files but just keep playing them
> against the replicate filesystem.
>
> They'd be fine for backups as well, as long as you can store the
> snapshots online until no longer needed for incrementals.

"app-backup/dar" uses catalogues for the incrementals. I think I will stick to 
that for the foreseeable future.

> >> But, you can always just create a snapshot, write it to backup with
> >> your favorite tool (it is just a directory tree), and then remove it
> >> as soon as you're done with it.  Creating a snapshot is atomic at the
> >> filesystem level, though again if you want application level
> >> consistency you need to deal with that until somebody comes up with a
> >> transactional way to store files on Linux that is more elegant that
> >> fsyncing on every write.
> >
> > That would require a method to keep database and filesystem perfectly in
> > sync when they are not necessarily on the same machine.
>
> Well, right now we can't even guarantee consistency when everything is
> written by a single process on the same machine.  The best we have is
> a clunky fsync operation which kills the write cache and destroys
> performance, and even that doesn't do anything if you have more than
> one file that must be consistent.

Yep, and that's why those filesystems are actually umounted prior to creating 
the LVM snapshot. Umounting forces the filesystem to be put into a consistent 
state.

> The result is journals on top of journals as nobody can trust the next
> layer down to do its job correctly.
>
> Going across machines does complicate things further as there are more
> failure modes that take out one part of the overall system but not
> another.  However, I'd like to think that an OS that natively supports
> transactions could at least standardize things so that every layer
> along the path isn't storing its own journal.
>
> In fact, many of the optimizations possible with zfs and btrfs are due
> to the fact that you eliminate all those layers.

Either of those 2, probably btrfs as I prefer native support in the kernel, 
will be implemented when I get the opportunity to get the NAS on bare metal 
and remove the virtualization for that component. I need to find a different 
host for the other services first.
That might take a while.

--
Joost