Re: [gentoo-user] Rasp-Pi-4 Gentoo servers

[Rich Freeman]

On Fri, Feb 28, 2020 at 6:09 AM Wols Lists <antlists@××××××××××××.uk> wrote:
>
> On 27/02/20 21:49, Rich Freeman wrote:
> > A fairly cheap amd64 system can run a ton of services in containers
> > though, and it is way simpler to maintain that way.  I still get quick
> > access to snapshots/etc, but now if I want to run a gentoo container
> > it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one
> > core, but once a month it needs 4GB of RAM and 100% of 6 cores.  As
> > long as I'm not doing an emerge -u world on half a dozen containers at
> > once it is no big deal at all.
>
> Do all your containers have the same make options etc? Can't remember
> which directory it is, but I had a shared emerge directory where it
> stored this stuff and I emerged with -bk options (use binary if it's
> there, create binary if it isn't).
>

They're probably not too far off in general, but not exact.  I only
run one instance of any particular container, so I haven't tried to do
parallel builds.  If portage had support for multiple binary packages
co-existing with different build options I might.  If I ever get
really bored for a few weeks I could see playing around with that.  It
seems like it ought to be possible to content-hash the list of build
options and stick that hash in the binary package filename, and then
have portage search for suitable packages, using a binary package if
one matches, and doing a new build if not.

Many of my containers don't even run Gentoo.  I have a few running
Arch, Ubuntu Server, or Debian.  If some service is well-supported in
one of those and is poorly supported in Gentoo I will tend to go that
route.  I'll package it if reasonable but some upstreams are just not
very conducive to this.

There was a question about ARM-based NAS in this thread which I'll go
ahead and tackle to save a reply.  I'm actually playing around with
lizardfs (I might consider moosefs instead if starting from scratch -
or Ceph if I were scaling up but that wouldn't be practical on ARM).
I have a mix of chunkservers but my target is to run new ones on ARM.
I'm using RockPro64 SBCs with LSI HBAs (this SBC is fairly unique in
having PCIe).  There is some issue with the lizardfs code that causes
performance issues on ARM though I understand they're working on this,
so that could change.  I'm using it for multimedia and I care more
about static space than iops, so it is fine for me.  The LSI HBA pulls
more power than the SBC does, but overall the setup is very low-power
and fairly inexpensive (used HBAs on ebay).  I can in theory get up to
16 drives on one SBC this way.  The SBC also supports USB3 so that is
another option with a hub - in fact I'm mostly shucking USB3 drives
anyway.

Main issue with ARM SBCs in general is that they don't have much RAM,
so IMO that makes Ceph a non-starter.  Otherwise that would probably
be my preferred option.  Bad things can happen on rebuilds if you
don't have 1GB/TB as they suggest, and even with the relatively
under-utilized servers I have now that would be a LOT of RAM for ARM
(really, it would be expensive even on amd64).  Lizardfs/moosefs
chunkservers barely use any RAM at all.  The master server does need
more - I have shadow masters running on the SBCs but since I'm using
this for multimedia the metadata server only uses about 100MB of RAM
and that includes processes, libraries, and random minimal service
daemons like sshd.  I'm running my master on amd64 though to get
optimal performance, shadowed on the chunkservers so that I can
failover if needed, though in truth the amd64 box with ECC is the
least likely thing to die and runs all the stuff that uses the storage
right now anyway.

The other suggestion to consider USB3 instead of SATA for storage
isn't a bad idea.  Though going that route means wall warts and drives
as far as the eye can see.  Might still be less messy than my setup,
which has a couple of cheap ATX PSUs with ATX power switches, 16x PCIe
powered risers for the HBAs (they pull too much power for the SBC),
and rockwell drive cages to stack the drives in (they're meant for a
server chasis but they're reasonably priced and basically give you an
open enclosure with a fan).  I'd definitely have a lot fewer PCBs
showing if I used USB3 instead.  I'm not sure how well that would
perform though - that HBA has a lot of bandwidth if the node got busy
with PCIe v2 x4 connectivity (SAS9200-16E) and with USB3 it would all
go through 1-2 ports.  Though I doubt I'd ever get THAT many drives on
a node and if I needed more space I'd probably expand up to 5
chunkservers before I'm putting more than about 3 drives on each - you
get better performance and more fault-tolerance that way.

One big reason I went the distributed filesystem approach was that I
was getting tired of trying to cram as many drives as I could into a
single host and then dealing with some of the inflexibilities of zfs.
The inflexibility bit is improving somewhat with removable vdevs,
though I'm not sure how much residue those leave behind if you do it
often.  But, zfs is still limited to however many drives you can cram
into one host, while a distributed filesystem lets you expand
outwards.  Plus it is fault-tolerant at the host level instead of the
drive level.

-- 
Rich