Re: [gentoo-user] Getting maximum space out of a hard drive

[Rich Freeman]

On Fri, Aug 26, 2022 at 7:26 AM Dale <rdalek1967@×××××.com> wrote:
>
> I looked into the Raspberry and the newest version, about $150 now, doesn't even have SATA ports.

The Pi4 is definitely a step up from the previous versions in terms of
IO, but it is still pretty limited.  It has USB3 and gigabit, and they
don't share a USB host or anything like that, so you should get close
to full performance out of both.  The CPU is of course pretty limited,
as is RAM.  Biggest benefit is the super-low power consumption, and
that is something I take seriously as for a lot of cheap hardware that
runs 24x7 the power cost rapidly exceeds the purchase price.  I see
people buying old servers for $100 or whatever and those things will
often go through $100 worth of electricity in a few months.

How many hard drives are you talking about?  There are two general
routes to go for something like this.  The simplest and most
traditional way is a NAS box of some kind, with RAID.  The issue with
these approaches is that you're limited by the number of hard drives
you can run off of one host, and of course if anything other than a
drive fails you're offline.  The other approach is a distributed
filesystem.  That ramps up the learning curve quite a bit, but for
something like media where IOPS doesn't matter it eliminates the need
to try to cram a dozen hard drives into one host.  Ceph can also do
IOPS but you're talking 10GbE + NVMe and big bucks, and that is how
modern server farms would do it.

I'll describe the traditional route since I suspect that is where
you're going to end up.  If you only had 2-4 drives total you could
probably get away with a Pi4 and USB3 drives, but if you want
encryption or anything CPU-intensive you're probably going to
bottleneck on the CPU.  It would be fine if you're more concerned with
capacity than storage.

For more drives than that, or just to be more robust, then any
standard amd64 build will be fine.  Obviously a motherboard with lots
of SATA ports will help here.  However, that almost always is a
bottleneck on consumer gear, and the typical solution to that for SATA
is a host bus adapter.  They're expensive new, but cheap on ebay (I've
had them fail though, which is probably why companies tend to sell
them while they're still working).  They also use a ton of power -
I've measured them using upwards of 60W - they're designed for servers
where nobody seems to care.  A typical HBA can provide 8-32 SATA
ports, via mini-SAS breakout cables (one mini-SAS port can provide 4
SATA ports).  HBAs tend to use a lot of PCIe lanes - you don't
necessarily need all of them if you only have a few drives and they're
spinning disks, but it is probably easiest if you get a CPU with
integrated graphics and use the 16x slot for the HBA.  That or get a
motherboard with two large slots (they usually aren't 16x, but getting
4-8x slots on a consumer motherboard isn't super-common).

For software I'd use mdadm plus LVM.  ZFS or btrfs are your other
options, and those can run on bare metal, but btrfs is immature and
ZFS cannot be reshaped the way mdadm can, so there are tradeoffs.  If
you want to use your existing drives and don't have a backup to
restore or want to do it live, then the easiest option there is to add
one drive to the system to expand capacity.  Put mdadm on that drive
as a degraded raid1 or whatever, then put LVM on top, and migrate data
from an existing disk live over to the new one, freeing up one or more
existing drives.  Then put mdadm on those and LVM and migrate more
data onto them, and so on, until everything is running on top of
mdadm.  Of course you need to plan how you want the array to look and
have enough drives that you get the desired level of redundancy.  You
can start with degraded arrays (which is no worse than what you have
now), then when enough drives are freed up they can be added as pairs
to fill it out.

If you want to go the distributed storage route then CephFS is the
canonical solution at this point but it is RAM-hungry so it tends to
be expensive.  It is also complex, but there are ansible playbooks and
so on to manage that (though playbooks with 100+ plays in them make me
nervous).  For something simpler MooseFS or LizardFS are probably
where I'd start.  I'm running LizardFS but they've been on the edge of
death for years upstream and MooseFS licensing is apparently better
now, so I'd probably look at that first.  I did a talk on lizardfs
recently: https://www.youtube.com/watch?v=dbMRcVrdsQs

-- 
Rich