Gentoo Archives: gentoo-user

From: Dale <rdalek1967@×××××.com>
To: gentoo-user@l.g.o
Subject: Re: [gentoo-user] How to compress lots of tarballs
Date: Wed, 29 Sep 2021 20:04:51
Message-Id: a3d946b7-7e3d-4183-d82f-43f57561938f@gmail.com
In Reply to: Re: [gentoo-user] How to compress lots of tarballs by Rich Freeman
1 Rich Freeman wrote:
2 > On Wed, Sep 29, 2021 at 4:27 AM Peter Humphrey <peter@××××××××××××.uk> wrote:
3 >> Thanks Laurence. I've looked at borg before, wondering whether I needed a
4 >> more sophisticated tool than just tar, but it looked like too much work for
5 >> little gain. I didn't know about duplicity, but I'm used to my weekly routine
6 >> and it seems reliable, so I'll stick with it pro tem. I've been keeping a
7 >> daily KMail archive since the bad old days, and five weekly backups of the
8 >> whole system, together with 12 monthly backups and, recently an annual
9 >> backup. That last may be overkill, I dare say.
10 > I think Restic might be gaining some ground on duplicity. I use
11 > duplicity and it is fine, so I haven't had much need to look at
12 > anything else. Big advantages of duplicity over tar are:
13 >
14 > 1. It will do all the compression/encryption/etc stuff for you - all
15 > controlled via options.
16 > 2. It uses librsync, which means if one byte in the middle of a 10GB
17 > file changes, you end up with a few bytes in your archive and not 10GB
18 > (pre-compression).
19 > 3. It has a ton of cloud/remote backends, so it is real easy to store
20 > the data on AWS/Google/whatever. When operating this way it can keep
21 > local copies of the metadata, and if for some reason those are lost it
22 > can just pull that only down from the cloud to resync without a huge
23 > bill.
24 > 4. It can do all the backup rotation logic (fulls, incrementals,
25 > retention, etc).
26 > 5. It can prefix files so that on something like AWS you can have the
27 > big data archive files go to glacier (cheap to store, expensive to
28 > restore), and the small metadata stays in a data class that is cheap
29 > to access.
30 > 6. By default local metadata is kept unencrypted, and anything on the
31 > cloud is encrypted. This means that you can just keep a public key in
32 > your keyring for completely unattended backups, without fear of access
33 > to the private key. Obviously if you need to restore your metadata
34 > from the cloud you'll need the private key for that.
35 >
36 > If you like the more tar-like process another tool you might want to
37 > look at is dar. It basically is a near-drop-in replacement for tar
38 > but it stores indexes at the end of every file, which means that you
39 > can view archive contents/etc or restore individual files without
40 > scanning the whole archive. tar was really designed for tape where
41 > random access is not possible.
42 >
43
44
45 Curious question here.  As you may recall, I backup to a external hard
46 drive.  Would it make sense to use that software for a external hard
47 drive?  Right now, I'm just doing file updates with rsync and the drive
48 is encrypted.  Thing is, I'm going to have to split into three drives
49 soon.  So, compressing may help.  Since it is video files, it may not
50 help much but I'm not sure about that.  Just curious. 
51
52 Dale
53
54 :-)  :-) 

Replies

Subject Author
RE: [gentoo-user] How to compress lots of tarballs Laurence Perkins <lperkins@×××××××.net>
Re: [gentoo-user] How to compress lots of tarballs Frank Steinmetzger <Warp_7@×××.de>