| 1 |
Am Wed, Sep 29, 2021 at 03:04:41PM -0500 schrieb Dale: |
| 2 |
|
| 3 |
> Curious question here. As you may recall, I backup to a external hard |
| 4 |
> drive. Would it make sense to use that software for a external hard |
| 5 |
> drive? |
| 6 |
|
| 7 |
Since you are using LVM for everything IIRC, it would be a very efficient |
| 8 |
way for you to make incremental backups with snapshots. But I have no |
| 9 |
knowledge in that area to give you hints. |
| 10 |
|
| 11 |
|
| 12 |
But I do use Borg. It’s been my primary backup tool for my systems for |
| 13 |
almost two years now. Before that I used rsnapshot (i.e. rsync with |
| 14 |
versioning through hard links) for my home partion and simple rsync for the |
| 15 |
data partition. Rsnapshot is quite slow, because it has to compare at least |
| 16 |
the inodes of all files on the source and destination. Borg uses a cache, |
| 17 |
which speeds things up drastically. |
| 18 |
|
| 19 |
I have one Borg repo for the root fs, one for ~ and one for the data |
| 20 |
partition, and each repo receives the partition from two different hosts, |
| 21 |
but which have most of their data mirrored daily with Unison. A tool like |
| 22 |
Borg can deduplicate all of that and create snapshots of it. This saves |
| 23 |
oogles of space, but also allows me to restore an entire host with a simple |
| 24 |
rsync from a mounted Borg repo. (only downside: no hardlink support, AFAIK). |
| 25 |
|
| 26 |
Borg saves its data in 500 MB files, which makes it very SMR-friendly. |
| 27 |
Rsnapshot will create little holes in the backup FS over time with the |
| 28 |
deletion of old snapshots. And as we all know, this will bring SMR drives |
| 29 |
down to a crawl. If you back-up only big video files, then this may not be a |
| 30 |
huge problem. But it will be with the ~ partition, with its thousands of |
| 31 |
little files. In Borg, little changes do not trickle down to many random |
| 32 |
writes. If a data file becomes partially obsolete, it is rewritten into a |
| 33 |
new file and the old one deleted as a whole. Thanks to that, I have no worry |
| 34 |
using 2.5″ 4 TB drives as main backup drive (as we all know, everything 2.5″ |
| 35 |
above 1 TB is SMR). |
| 36 |
|
| 37 |
Those big data files also make it very efficient to copy a Borg repo (for |
| 38 |
example to mirror the backup drive to another drive for off-site storage), |
| 39 |
because it uses a very small number of files itself: |
| 40 |
|
| 41 |
$ borg info data |
| 42 |
... |
| 43 |
------------------------------------------------------------------------------ |
| 44 |
Original size Compressed size Deduplicated size |
| 45 |
All archives: 18.09 TB 17.60 TB 1.23 TB |
| 46 |
|
| 47 |
Unique chunks Total chunks |
| 48 |
Chunk index: 722096 10888890 |
| 49 |
|
| 50 |
$ find data -type f | wc -l |
| 51 |
2498 |
| 52 |
|
| 53 |
I have 21 snapshots in that repo, which amount to 18 TB of backed-up data, |
| 54 |
deduped down to 1.23 TB, spread over only 2498 files including metadata. |
| 55 |
|
| 56 |
> Right now, I'm just doing file updates with rsync and the drive |
| 57 |
> is encrypted. |
| 58 |
|
| 59 |
While Borg has an encryption feature, I chose not to use it and rely on the |
| 60 |
underlying LUKS. Because then I can use KDE GUI stuff to mount the drive and |
| 61 |
run my Borg wrapper script without ever having to enter a passphrase. |
| 62 |
|
| 63 |
> Thing is, I'm going to have to split into three drives soon. So, |
| 64 |
> compressing may help. Since it is video files, it may not help much but |
| 65 |
> I'm not sure about that. |
| 66 |
|
| 67 |
Of my PC’s data partition, almost 50 % is music, 20 % is my JPEG pictures |
| 68 |
library, 15 % is video files and the rest is misc stuff like Kerbal Space |
| 69 |
Program, compressed archives of OpenStreetMap files and VM images. |
| 70 |
This is the statistics of my last snapshot: |
| 71 |
|
| 72 |
Original size Compressed size Deduplicated size |
| 73 |
730.80 GB 698.76 GB 1.95 MB |
| 74 |
|
| 75 |
Compression gain is around 4 %. Much of which probably comes from empty |
| 76 |
areas in VM images and 4 GB of pdf and html files. On my laptop, whose data |
| 77 |
partition has fewer VM stuff, but a lot more videos, it looks thus: |
| 78 |
|
| 79 |
Original size Compressed size Deduplicated size |
| 80 |
1.01 TB 1.00 TB 1.67 MB |
| 81 |
|
| 82 |
So only around 1 % of savings. However, compression is done using lz4 (by |
| 83 |
default, you can choose other algos), which is extremely fast but not very |
| 84 |
strong. In fact, Borg tries to compress all chunks, but if it finds that |
| 85 |
compressing a chunk doesn’t yield enough benefit, it actually discards it |
| 86 |
and uses the uncompressed data to save on CPU load later on. |
| 87 |
|
| 88 |
-- |
| 89 |
Grüße | Greetings | Qapla’ |
| 90 |
Please do not share anything from, with or about me on any social network. |
| 91 |
|
| 92 |
Some people are so tired, they can’t even stay awake until falling asleep. |
Archives