Re: [gentoo-user] backup horror story (happy ending)

[Rich Freeman <rich0@gentoo.org>]

On Sat, Nov 9, 2024 at 5:30 AM ralfconn <mentadent47@yahoo.com> wrote:
>
> But I have a backup, no problem... till I realize the cron job had
> already run so it had overwritten the old files with the new, corrupt
> versions.
>

I highly recommend having multiple backups to avoid this sort of problem.

If you aren't wedded to rsync, then restic seems to be the platform of
choice these days.  Not sure offhand if it handles ntfs very well.  I
use duplicati for backing up windows hosts to an S3 backend and that
works great, but that is more of a windows solution (VSS and so on).
I imagine that restic doesn't care much about the filesystem if it is
running on linux and everything is mounted.

If you are wedded to the rsync approach where your backups are just a
big directory tree that you can easily access, then I suggest using
rsnapshot.  It is basically a wrapper around rsync that maintains a
backup history, in a very clean way.  Basically it does a hard-link
copy of your entire backup set to a new directory tree named by the
timestamp, and then it runs rsync to sync that new tree.  The result
is that you get file-level deduplication effectively, and otherwise
get what looks like a nice big full copy of the backup source in each
directory.  It probably won't be as efficient as something like restic
since I'm guessing that can do deduplication below the file level (I
think it can also deduplicate across multiple hosts/etc if you're
using it that way).  Most of these modern tools still use librsync
under the hood so the actual data transfer is just as efficient.

rsync by itself is nice for its simplicity, but it just isn't a very
elegant backup solution.  You can tell it to preserve old file
versions, but those end up stored next to the original files with
different filenames and that can be a real mess to restore if you
don't want to end up with all those old versions.  With
restic/rsnapshot you can go back in time but still get a clean
restore, and you can still extract individual files from various
snapshots.

If you really are running rsync at any kind of scale also consider
rclone, which is often faster since it can transfer multiple files in
parallel, which is useful if you're more bound by latency than disk
IOPS (often the case on solid state drives over networks).

-- 
Rich