Re: [gentoo-user] Backup program that compresses data but only changes new files.

[William Kenworthy]

On 15/8/22 06:44, Dale wrote:
> Howdy,
>
> With my new fiber internet, my poor disks are getting a work out, and
> also filling up.  First casualty, my backup disk.  I have one directory
> that is . . . well . . . huge.  It's about 7TBs or so.  This is where it
> is right now and it's still trying to pack in files.
>
>
> /dev/mapper/8tb            7.3T  7.1T  201G  98% /mnt/8tb
>
>
> Right now, I'm using rsync which doesn't compress files but does just
> update things that have changed.  I'd like to find some way, software
> but maybe there is already a tool I'm unaware of, to compress data and
> work a lot like rsync otherwise.  I looked in app-backup and there is a
> lot of options but not sure which fits best for what I want to do.
> Again, backup a directory, compress and only update with changed or new
> files.  Generally, it only adds files but sometimes a file gets replaced
> as well.  Same name but different size.
>
> I was trying to go through the list in app-backup one by one but to be
> honest, most links included only go to github or something and usually
> doesn't tell anything about how it works or anything.  Basically, as far
> as seeing if it does what I want, it's useless. It sort of reminds me of
> quite a few USE flag descriptions.
>
> I plan to buy another hard drive pretty soon.  Next month is possible.
> If there is nothing available that does what I want, is there a way to
> use rsync and have it set to backup files starting with "a" through "k"
> to one spot and then backup "l" through "z" to another?  I could then
> split the files into two parts.  I use a script to do this now, if one
> could call my little things scripts, so even a complicated command could
> work, just may need help figuring out the command.
>
> Thoughts?  Ideas?
>
> Dale
>
> :-)  :-)
>
The questions you need to ask is how compressible is the data and how 
much duplication is in there.  Rsync's biggest disadvantage is it 
doesn't keep history, so if you need to restore something from last week 
you are SOL.  Honestly, rsync is not a backup program and should only be 
used the way you do for data that don't value as an rsync archive is a 
disaster waiting to happen from a backup point of view.

Look into dirvish - uses hard links to keep files current but safe, is 
easy to restore (looks like a exact copy so you cp the files back if 
needed.  Downside is it hammers the hard disk and has no compression so 
its only deduplication via history (my backups stabilised about 2x 
original size for ~2yrs of history - though you can use something like 
btrfs which has filesystem level compression.

My current program is borgbackup which is very sophisticated in how it 
stores data - its probably your best bet in fact.  I am storing 
literally tens of Tb of raw data on a 4Tb usb3 disk (going back years 
and yes, I do restore regularly, and not just for disasters but for 
space efficient long term storage I access only rarely.

e.g.:

A single host:

------------------------------------------------------------------------------
                        Original size      Compressed size Deduplicated size
All archives:                3.07 TB              1.96 TB            
151.80 GB

                        Unique chunks         Total chunks
Chunk index:                 1026085             22285913


Then there is my offline storage - it backs up ~15 hosts (in repos like 
the above) + data storage like 22 years of email etc. Each host backs up 
to its own repo then the offline storage backs that up.  The 
deduplicated size is the actual on disk size ... compression varies as 
its whatever I used at the time the backup was taken ... currently I 
have it set to "auto,zstd,11" but it can be mixed in the same repo (a 
repo is a single backup set - you can nest repos which is what I do - so 
~45Tb stored on a 4Tb offline disk).  One advantage of a system like 
this is chunked data rarely changes, so its only the differences that 
are backed up (read the borgbackup docs - interesting)

------------------------------------------------------------------------------
                        Original size      Compressed size Deduplicated size
All archives:               28.69 TB             28.69 TB              
3.81 TB

                        Unique chunks         Total chunks
Chunk index: