Re: [gentoo-user] How to compress lots of tarballs

[Peter Humphrey]

On Tuesday, 28 September 2021 12:38:42 BST Rich Freeman wrote:

> You keep mentioning USB3, but I think the main factor here is that the
> external drive is probably a spinning hard drive (I don't think you
> explicitly mentioned this but it seems likely esp with the volume of
> data).  That math works out to 78MB/s.  Hard drive transfer speeds
> depend on the drive itself and especially whether there is more than
> one IO task to be performed, so I can't be entirely sure, but I'm
> guessing that the USB3 interface itself is having almost no adverse
> impact on the transfer rate.

I'm sure you're right Rich, and yes, this is 2.5" drive. I'm seeing about 
110MB/s reading from USB feeding into zstd.

> The main thing to avoid is doing other sustained read/writes from the
> drive at the same time.

Quite so.

> It looks like you ended up doing the bulk of the compression on an
> SSD, and obviously those don't care nearly as much about IOPS.

Yes, input from USB and output to SSD.

> I've been playing around with lizardfs for bulk storage and found that
> USB3 hard drives actually work very well, as long as you're mindful
> about what physical ports are on what USB hosts and so on.  A USB3
> host can basically handle two hard drives with no loss of performance.
> I'm not dealing with a ton of IO though so I can probably stack more
> drives with pretty minimal impact unless there is a rebuild (in which
> case the gigabit ethernet is probably still the larger bottleneck).
> Even a Raspberry Pi 4 has two USB3 hosts, which means you could stack
> 4 hard drives on one and get basically the same performance as SATA.
> When you couple that with the tendency of manufacturers to charge less
> for USB3 drives than SATA drives of the same performance it just
> becomes a much simpler solution than messing with HBAs and so on and
> limiting yourself to hardware that can actually work with an HBA.

-- 
Regards,
Peter.