Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Rich Freeman]

On Tue, May 27, 2014 at 11:12 AM, J. Roeleveld <joost@××××××××.org> wrote:
> On Tuesday, May 27, 2014 10:31:26 AM Rich Freeman wrote:
>> btrfs wouldn't have any issues with this at all.  You'd have an
>> advantage in that you wouldn't have to unmount the filesystem to
>> cleanly create the snapshot (which you have to do with lvm).
>
> That, or a "sync" prior to creating the snapshot. :)

If the filesystem is still mounted, I'm not sure that a sync is
guaranteed to give you a clean remount.  It only flushes the
caches/etc.  You need to remount read-only or unmount before doing the
sync (and the sync probably isn't actually necessary as I'd think LVM
would snapshot the contents of the cache as well).

> I have a yearly (full), monthly, weekly and daily. Each incremental is against
> the most recent one of itself or longer period.
> That means having to keep multiple snapshots active, which I prefer to avoid.

You only need to store snapshots for use with incremental backups.
So, if all your backups are full, then you don't need to retain any
snapshots (and you wouldn't use btrfs send anyway).  If your yearly is
full and your monthlies are incremental against the yearly then you
need to keep your yearly snapshot for a year.  If your yearly is full
and your monthlies are incremental against the last month, then you
only need to keep the yearly until the next monthly.  If your
monthlies are full then you only need to keep the current monthly
assuming your dailies are incremental against those, but if they're
incremental from the last daily then you never need to keep anything
for more than a day.

>
> But, it is a good idea for backing up desktops and laptops.

It is really intended more for something like datacenter replication.
Snapshot every 5 min, send the data to the backup datacenter, delete
the snapshots upon confirmation of successful receipt.  In such a
scenario you wouldn't retain the sent files but just keep playing them
against the replicate filesystem.

They'd be fine for backups as well, as long as you can store the
snapshots online until no longer needed for incrementals.

>
>> But, you can always just create a snapshot, write it to backup with
>> your favorite tool (it is just a directory tree), and then remove it
>> as soon as you're done with it.  Creating a snapshot is atomic at the
>> filesystem level, though again if you want application level
>> consistency you need to deal with that until somebody comes up with a
>> transactional way to store files on Linux that is more elegant that
>> fsyncing on every write.
>
> That would require a method to keep database and filesystem perfectly in sync
> when they are not necessarily on the same machine.
>

Well, right now we can't even guarantee consistency when everything is
written by a single process on the same machine.  The best we have is
a clunky fsync operation which kills the write cache and destroys
performance, and even that doesn't do anything if you have more than
one file that must be consistent.

The result is journals on top of journals as nobody can trust the next
layer down to do its job correctly.

Going across machines does complicate things further as there are more
failure modes that take out one part of the overall system but not
another.  However, I'd like to think that an OS that natively supports
transactions could at least standardize things so that every layer
along the path isn't storing its own journal.

In fact, many of the optimizations possible with zfs and btrfs are due
to the fact that you eliminate all those layers.

Rich