Gentoo Archives: gentoo-user

From: Rich Freeman <rich0@g.o>
To: gentoo-user@l.g.o
Subject: Re: [gentoo-user] How to compress lots of tarballs
Date: Wed, 29 Sep 2021 15:37:45
Message-Id: CAGfcS_=+igNZDBuR9DyT-+dkSu9DYEt2sw_70YbbyycXFrXR4g@mail.gmail.com
In Reply to: Re: [gentoo-user] How to compress lots of tarballs by Peter Humphrey
1 On Wed, Sep 29, 2021 at 4:27 AM Peter Humphrey <peter@××××××××××××.uk> wrote:
2 >
3 > Thanks Laurence. I've looked at borg before, wondering whether I needed a
4 > more sophisticated tool than just tar, but it looked like too much work for
5 > little gain. I didn't know about duplicity, but I'm used to my weekly routine
6 > and it seems reliable, so I'll stick with it pro tem. I've been keeping a
7 > daily KMail archive since the bad old days, and five weekly backups of the
8 > whole system, together with 12 monthly backups and, recently an annual
9 > backup. That last may be overkill, I dare say.
10
11 I think Restic might be gaining some ground on duplicity. I use
12 duplicity and it is fine, so I haven't had much need to look at
13 anything else. Big advantages of duplicity over tar are:
14
15 1. It will do all the compression/encryption/etc stuff for you - all
16 controlled via options.
17 2. It uses librsync, which means if one byte in the middle of a 10GB
18 file changes, you end up with a few bytes in your archive and not 10GB
19 (pre-compression).
20 3. It has a ton of cloud/remote backends, so it is real easy to store
21 the data on AWS/Google/whatever. When operating this way it can keep
22 local copies of the metadata, and if for some reason those are lost it
23 can just pull that only down from the cloud to resync without a huge
24 bill.
25 4. It can do all the backup rotation logic (fulls, incrementals,
26 retention, etc).
27 5. It can prefix files so that on something like AWS you can have the
28 big data archive files go to glacier (cheap to store, expensive to
29 restore), and the small metadata stays in a data class that is cheap
30 to access.
31 6. By default local metadata is kept unencrypted, and anything on the
32 cloud is encrypted. This means that you can just keep a public key in
33 your keyring for completely unattended backups, without fear of access
34 to the private key. Obviously if you need to restore your metadata
35 from the cloud you'll need the private key for that.
36
37 If you like the more tar-like process another tool you might want to
38 look at is dar. It basically is a near-drop-in replacement for tar
39 but it stores indexes at the end of every file, which means that you
40 can view archive contents/etc or restore individual files without
41 scanning the whole archive. tar was really designed for tape where
42 random access is not possible.
43
44 --
45 Rich

Replies

Subject Author
Re: [gentoo-user] How to compress lots of tarballs Dale <rdalek1967@×××××.com>