Re: [gentoo-user] Re: WARNING: Crucial MX300 drives SUUUUUCK!!!!

[Poison BL.]

On Mon, Mar 6, 2017 at 2:23 AM, Kai Krakow <hurikhan77@×××××.com> wrote:

> Am Tue, 14 Feb 2017 16:14:23 -0500
> schrieb "Poison BL." <poisonbl@×××××.com>:
> > I actually see both sides of it... as nice as it is to have a chance
> > to recover the information from between the last backup and the death
> > of the drive, the reduced chance of corrupt data from a silently
> > failing (spinning) disk making it into backups is a bit of a good
> > balancing point for me.
>
> I've seen bordbackup giving me good protection to this. First, it
> doesn't backup files which are already in the backup. So if data
> silently changed, it won't make it into the backup. Second, it does
> incremental backups. Even if something broke and made it into the
> backup, you can eventually go back weeks or months to get back the
> file. The algorithm is very efficient. And every incremental backup is
> a full backup at the same time - so you thin out backup history by
> deleting any backup at any time (so it's not like traditional
> incremental backup which always needs the parent backup).
>
> OTOH, this means that every data block is only stored once. If silent
> data corruption is hitting here, you loose the complete history of this
> file (and maybe others using the same deduplicated block).
>
> For the numbers, I'm storing my 1.7 TB system into a 3 TB disk which is
> 2.2 TB full now. But the backup history is almost 1 year now (daily
> backups).
>
> As a sort of protection against silent data corruption, you could rsync
> borgbackup to a remote location. The differences are usually small, so
> that should be a fast operation. Maybe to some cloud storage or RAID
> protected NAS which can detect and correct silent data corruption (like
> ZFS or btrfs based systems).
>
>
> --
> Regards,
> Kai
>
> Replies to list-only preferred.
>

That's some impressive backup density... and I haven't looked into
borgbackup, but it sounds like it runs on the same principles as the
rsync+hardlink based scripts I've seen, though those will back up files
that've silently changed, since the checksums won't match any more, but
that won't blow away previous copies of the file either. I'll have to give
it a try!

As for protecting against the backup set itself getting silent corruption,
an rsync to a remote location would help, but you would have to ensure it
doesn't overwrite anything already there that may've changed, only create
new. Also, making the initial clone would take ages, I suspect, since it
would have to rebuild the hardlink set for everything (again, assuming
that's the trick borgbackup's using). One of the best options is to house
the base backup set itself on something like zfs or btrfs on a system with
ecc ram, and maintain checksums of everything on the side (crc32 would
likely suffice, but sha1's fast enough these days there's almost no excuse
not to use it). It might be possible to task tripwire to keep tabs on that
side of it, now that I consider it. While the filesystem itself in that
case is trying its best to prevent issues, there's always that slim risk
that there's a bug in the filesystem code itself that eats something, hence
the added layer of paranoia. Also, with ZFS for the base data set, you gain
in-place compression, dedup if you're feeling adventurous (not really worth
it unless you have multiple very similar backup sets for different
systems), block level checksums, redundancy across physical disks, in place
snapshots, and the ability to use zfs send/receive to do snapshot backups
of the backup set itself.

I managed to corrupt some data with zfs (w/ dedup, on gentoo) shared out
over nfs a while back on a box with way too little ram a while back
(nothing important, throwaway VM images), hence the paranoia of secondary
checksum auditing and still replicating the backup set for things that
might be important.

-- 
Poison [BLX]
Joshua M. Murphy