Re: [gentoo-user] Backup program that compresses data but only changes new files.

[Rich Freeman]

On Mon, Aug 15, 2022 at 3:05 AM Dale <rdalek1967@×××××.com> wrote:
>
> Rich Freeman wrote:
> >
> > Duplicity uses librsync, so it backs up exactly the same data as rsync
> > would, except instead of replicating entire files, it creates streams
> > of data more like something like tar.  So if you back up a million
> > small files you might get out 1-3 big files.  It can compress and
> > encrypt the data as you wish.
>
> Duplicity sounds interesting except that I already have the drive
> encrypted.

Then don't encrypt it?  Both compression and encryption are optional.

> The reason I mentioned being like rsync, I don't want to rebuild a backup
> from scratch each time as that would be time consuming.

Ah, you just want something that does incremental backups.  Duplicity
does, along with most decent solutions.

I see lots of talk of NAS and zfs/btrfs and snapshots.  IMO these are
NOT really great solutions for backup.  NAS can work of course but it
is overkill for backup storage.

NAS, zfs, btrfs, and snapshots are all great things to use with your
live data.  I use several of these myself.  Your live data should be
protected against bitrot with snapshots/etc.  That has nothing to do
with why you want backups.

We're talking about the storage of backups.  While you can store
backups on any of these they don't really add much value.

Also, you mentioned SMR, and I would definitely not combine SMR with
most of those.  SMR is perfect for backup.  It just isn't perfect for
backup using something like rsync that modifies files in place.  You
want something that only appends to backup files or creates new ones,
which is basically how most backup software works except for stuff
that works like rsync.

The main issue I think you're going to have is having support for
multi-volume backups if you need to be able to split a backup across
drives.  The only thing I've found on Linux that does this is bacula,
and it is a royal pain that I'm embarrassed to even mention.  If
somebody knows of another backup solution that can write the output to
disk (a filesystem, not /dev/rmt) and then pause to mount a new disk
when one fills up, I'm all ears.  For everything else I've tended to
see people suggest using lvm/mdadm/whatever combine disks into a
single block device so that the backup software doesn't see multiple
disks.

If you do want to go the route of combining your disks then since
you're using SMR I'd probably pick something like lvm that doesn't do
any striping/etc and just fills up one disk then moves to the next.
Then use a simple filesystem (not btrfs/zfs) that just starts at one
end and keeps adding.  A log-based filesystem would probably be ideal
but I'm not sure if any are decent.  You do have the issue of what you
do when you start to run out of space, unless you can create multiple
sets of disks so that you can complete a new backup before destroying
the old one.

-- 
Rich