Re: [gentoo-user] Rasp-Pi-4 Gentoo servers

[William Kenworthy]

I am going to answer multiple points below:


On 1/3/20 11:36 pm, Daniel Frey wrote:
> On 3/1/20 6:33 AM, Rich Freeman wrote:
>> On Sun, Mar 1, 2020 at 2:13 AM William Kenworthy <billk@×××××××××.au>
>> wrote:
>>>
>>> Keep in mind that rpi are not the only cheap, capable arm hardware out
>>> there.
>>>
>>
I am in Oz, delivery from HardKernel (the South Korean company behind 
the Odroid line) takes ~1 week.  Shipping is mostly via FedEx, who are a 
bit pricy and for me means a 30m drive to get it from a delivery centre 
as I cant be home to take delivery - though the last delivery used a 
different, much more flexible delivery company.

lizardfs and moosefs are very similar (originally a fork) - I went with 
moosefs as the community was better a few months ago - but lizardfs 
sounds like its getting back on track after some infighting (moosefs 
apparently went through the same thing)  I find the moosefs 
documentation and community help are adequate, but ultimately they are 
are a commercial project with a free taster offering hence some 
limitations in design for the community version (such as single 
master/no shadow masters) that lizardfs doesn't have.  I will move to 
lizardfs when I am satisfied they have their act together because of 
this - having a single master means taking the whole cluster offline 
when the master needs maintenance or fails which is painful

For those wanting to run a lot of drives on a single host - that defeats 
the main advantage of using a chunkserver based filesystem - 
redundancy.  Its far more common to have a host fail than a disk drive.  
Losing the major part of your storage in one go means the cluster is 
effectively dead - hence having a lot of completely separate systems is 
much more reliable - yes, I did try having 4 sata drives on an atom 
board and found it was easy to justify two more HC2's for the 
reliability. (note, its not the effect of the atom boards reliability I 
am pointing out, but the effect on the whole storage system of losing 
such a large percentage of capacity in one go - think maintenance, fail 
to startup etc. - its more common than you may think)

Dale: there is no single reference - I just designed on the fly and did 
what was necessary to move from my existing KVM/Qemu architecture on a 
two powerfull intel systems, each with 8TB storage to an lxc container 
based system backed by moosefs.  It uses an odroid arm based n2 for lxc, 
an arm based xu4 for management and software compilation (its not that 
powerfull, but is adequate), the HC2's (very similar to the xu4 - same 
arm architecture) and an H2 to run the mfsmaster and I am moving to 
ansible for management.  Overall, I am finding less maintenance due to 
better backups and better reliability, considerably lower power 
consumption while overall performance seems better.

Gotchas:

1. I originally got the 4gb ram N2 to run the mfsmaster software - tests 
were excellent - until I added multiple copies of a mailserver with 
nearly 20 years of history - hit swap and slowed to a crawl.  So I now 
have an intel based Odroid H2 with 32GB ram - currently is using ~4.2GB 
ram for 7TB of 20TB used, but has hit over 32GbB and well into swap 
while converging.  One admin in our local LUG is using 4xHC2 with the 
master running on one of the HC2's as a media server with no problems - 
its millions of small files added ina short time that causes the grief - 
there is a formula in the moosefs documentation allowing resources to be 
estimated - on the mailing list I saw a mention of a data centre using 
something like 150G ram and having problems!

2. The Odroid HC2 has a single sata port and a single usb port - I have 
5 of them, two have a sata + a usb3 drive attached.  I did try (on the 
N2) using multiple usb3 drives on the internal hub - disaster with way 
too much traffic through the hub - don't do it!

3. Storage options are an SD card (the faster the better - swap on an SD 
card ... sucks!) or eMMC (5xfaster than even a good SD card.) for the N2 
and xu4 - HC2's can only do sdcard, or sata. The H2 can do sd card, 
sata, eMMC, or m2 NVME - this last really flies!  Note that almost all 
arm system max out at 2 to 4GB of ram, so swap is usually needed for 
safety - depending on OOM killer resource management on a SAN type 
storage system is asking for corruption like I saw in one recent gentoo 
email.

4. xu4/HC2 are 32 bit arm v7, the N2 is 64 bit and runs aarch64 nicely - 
I copied my rpi images across repurposed them (hooray for 
emerge/portage!).  I do not use, or have done any work on their 
graphics/multimedia capabilities

5. I want to move to all gentoo-sources kernels - the xu4/HC2's are 
still on the odroid kernel until I get around to it.  The N2 was a lot 
of work, but ultimately successful, the H2 is standard amd64 EFI.


Have fun!

BillK