Re: [gentoo-user] Rasp-Pi-4 Gentoo servers

[William Kenworthy]

On 2/3/20 10:40 am, Rich Freeman wrote:
> On Sun, Mar 1, 2020 at 8:52 PM William Kenworthy <billk@×××××××××.au> wrote:
>> For those wanting to run a lot of drives on a single host - that defeats
>> the main advantage of using a chunkserver based filesystem -
>> redundancy.  Its far more common to have a host fail than a disk drive.
>> Losing the major part of your storage in one go means the cluster is
>> effectively dead - hence having a lot of completely separate systems is
>> much more reliable
> Of course.  You should have multiple hosts before you start putting
> multiple drives on a single host.
>
> However, once you have a few hosts the performance improves by adding
> more, but you're not really getting THAT much additional redundancy.
> You would get faster rebuild times by having more hosts since there
> would be less data to transfer when one fails and more hosts doing the
> work.
>
> So, it is about finding a balance.  You probably don't want 30 drives
> on 2 hosts.  However, you probably also don't need 15-30 hosts for
> that many drives either.  I wouldn't be putting 16 drives onto a
> single host until I had a fair number of hosts.
>
> As far as the status of lizardfs goes - as far as I can tell it is
> mostly developed by a company and they've wavered a bit on support in
> the last year.  I share your observation that they seem to be picking
> up again.  In any case, I'm running the latest stable and it works
> just fine, but it lacks the high availability features.  I can have
> shadow masters, but they won't automatically fail over, so maintenance
> on the master is still a pain.  Recovery due to failure of the master
> should be pretty quick though even if manual - just have to run a
> command on each shadow to determine which has the most recent
> metadata, then adjust DNS for my master CNAME to point to the new
> master, and then edit config on the new master to tell it that it is
> the master and no longer a shadow, and after restarting the daemon the
> cluster should be online again.
>
> The latest release candidate has the high availability features (used
> to be paid, is now free), however it is still a release candidate and
> I'm not in that much of a rush.  There was a lot of griping on the
> forums/etc by users who switched to the release candidate and ran into
> bugs that ate their data.  IMO that is why you don't go running
> release candidates for distributed filesystems with a dozen hard
> drives on them - if you want to try them out just run them in VMs with
> a few GB of storage to play with and who cares if your test data is
> destroyed.  It is usually wise to be conservative with your
> filesystems.  Makes no difference to me if they take another year to
> do the next release - I'd like the HA features but it isn't like the
> old code goes stale.
>
> Actually, the one thing that it would be nice if they fixed is the
> FUSE client - it seems to leak RAM.
>
> Oh, and the docs seem to hint at a windows client somewhere which
> would be really nice to have, but I can't find any trace of it.  I
> only normally run a single client but it would obviously perform well
> as a general-purpose fileserver.
>
> There has been talk of a substantial rewrite, though I'm not sure if
> that will actually happen now.  If it does I hope they do keep the RAM
> requirements low on the chunkservers.  That was the main thing that
> turned me off from ceph - it is a great platform in general but
> needing 1GB RAM per 1TB disk adds up really fast, and it basically
> precludes ARM SBCs as OSDs as you can't get those with that much RAM
> for any sane price - even if you were only running one drive per host
> good luck finding a SBC with 13GB+ of RAM.  You can tune ceph to use
> less RAM but I've heard that bad things happen if you have some hosts
> shuffle during a rebuild and you don't have gobs of RAM - all the OSDs
> end up with an impossible backlog and they keep crashing until you run
> around like Santa Claus filling every stocking with a handful of $60
> DIMMs.
>
> Right now lizardfs basically uses almost no ram at all on
> chunkservers, so an ARM SBC could run dozens of drives without an
> issue.
>
Everything bad you hear about ceph is true ... and then some! I did try, 
but this was some years ago so hopefully its better now. The two biggies 
were excessive network requirements (bandwidth, separation) and recovery 
times with frequent crash and burn. There are ceph features I would 
really like to use (rbd, local copies with much simpler config, ...) but 
moosefs is a lot more bullet proof on lesser resource requirements 
though I did find properly pruned vlans on a smartswitch separating the 
intra-cluster from external requests made a noticeable difference.

moosefs has a windows client but its only available with the paid 
version.  The master/shadow-master and auto failover is only available 
through the paid version - for the community you have to stop the 
master, copy the files then change DNS etc. before restarting the new 
master - cant really do it online even when scripted - its painful with 
downtime and I had dns caching issues that took time to work their way 
out of the system.  I thought lizardfs was much more community minded 
but you are characterising it as similar to moosefs - a taster offering 
by a commercial company holding back some of the non-essential but 
jucier features for the paid version - is that how you see them?

By the way, to keep to the rpi subject, I did have a rpi3B with a usb2 
sata drive attached but it was hopeless as a chunkserver impacting the 
whole cluster.  Having the usb data flow and network data flow through 
the same hub just didn't go well - I started with the odroids before the 
rpi4 was released or I might have experimented with it first (using a 
sata HAT) - anyone with a comment on how that compares with a HC2?

BillK