Re: [gentoo-user] How to compress lots of tarballs

[Rich Freeman]

On Mon, Sep 27, 2021 at 9:30 AM Peter Humphrey <peter@××××××××××××.uk> wrote:
>
> Thanks to all who've helped. I can't avoid feeling, though, that the main
> bottleneck has been missed: that I have to read and write on a USB-3 drive.
> It's just taken 23 minutes to copy the current system backup from USB-3 to
> SATA SSD: 108GB in 8 .tar files.

You keep mentioning USB3, but I think the main factor here is that the
external drive is probably a spinning hard drive (I don't think you
explicitly mentioned this but it seems likely esp with the volume of
data).  That math works out to 78MB/s.  Hard drive transfer speeds
depend on the drive itself and especially whether there is more than
one IO task to be performed, so I can't be entirely sure, but I'm
guessing that the USB3 interface itself is having almost no adverse
impact on the transfer rate.

The main thing to avoid is doing other sustained read/writes from the
drive at the same time.

It looks like you ended up doing the bulk of the compression on an
SSD, and obviously those don't care nearly as much about IOPS.

I've been playing around with lizardfs for bulk storage and found that
USB3 hard drives actually work very well, as long as you're mindful
about what physical ports are on what USB hosts and so on.  A USB3
host can basically handle two hard drives with no loss of performance.
I'm not dealing with a ton of IO though so I can probably stack more
drives with pretty minimal impact unless there is a rebuild (in which
case the gigabit ethernet is probably still the larger bottleneck).
Even a Raspberry Pi 4 has two USB3 hosts, which means you could stack
4 hard drives on one and get basically the same performance as SATA.
When you couple that with the tendency of manufacturers to charge less
for USB3 drives than SATA drives of the same performance it just
becomes a much simpler solution than messing with HBAs and so on and
limiting yourself to hardware that can actually work with an HBA.

-- 
Rich