Re: [gentoo-user] Hard drive storage questions

[Rich Freeman]

On Fri, Nov 9, 2018 at 3:17 AM Bill Kenworthy <billk@×××××××××.au> wrote:
>
> I'll second your comments on ceph after my experience - great idea for
> large scale systems, otherwise performance is quite poor on small
> systems. Needs at least GB connections with two networks as well as only
> one or two drives per host to work properly.
>
> I think I'll give lizardfs a go - an interesting read.
>

So, ANY distributed/NAS solution is going to want a good network
(gigabit or better), if you care about performance.  With Ceph and the
rebuilds/etc it probably makes an even bigger difference, but lizardfs
still shuttles data around.  With replication any kind of write is
multiplied so even moderate use is going to use a lot of network
bandwidth.  If you're talking about hosting OS images for VMs it is a
big deal.  If you're talking about hosting TV shows for your Myth
server or whatever, it probably isn't as big a deal unless you have 14
tuners and 12 clients.

Lizardfs isn't without its issues.  For my purposes it is fine, but it
is NOT as robust as Ceph.  Finding direct comparisons online is
difficult, but here are some of my observations (having not actually
used either, but having read up on both):

* Ceph (esp for obj store) is designed to avoid bottlenecks.  Lizardfs
has a single master server that ALL metadata requests have to go
through.  When you start getting into dozens of nodes that will start
to be a bottleneck, but it also eliminates some of the rigidity of
Ceph since clients don't have to know where all the data is.  I
imagine it adds a bit of latency to reads.

* Lizardfs defaults to acking writes after the first node receives
them, then replicates them.  Ceph defaults to acking after all
replicas are made.  For any application that takes transactions
seriously there is a HUGE data security difference, but it of course
will lower write latency for lizardfs.

* Lizardfs makes it a lot easier to tweak storage policy at the
directory/file level.  Cephfs basically does this more at the
mountpoint level.

* Ceph CRUSH maps are much more configurable than Lizardfs goals.
With Ceph you could easily say that you want 2 copies, and they have
to be on hard drives with different vendors, and in different
datacenters.  With Lizardfs combining tags like this is less
convenient, and while you could say that you want one copy in rack A
and one in rack B, you can't say that you don't care which 2 as long
as they are different.

* The lizardfs high-availability stuff (equiv of Ceph monitors) only
recently went FOSS, and probably isn't stabilized on most distros.
You can have backup masters that are ready to go, but you need your
own solution for promoting them.

* Lizardfs security seems to be non-existent.  Don't stick it on your
intranet if you are a business.  Fine for home, or for a segregated
SAN, maybe, or you could stick it all behind some kind of VPN and roll
your own security layer.  Ceph security seems pretty robust, but
watching what the ansible playbook did to set it up makes me shudder
at the thought of doing it myself.  Lots of keys that all need to be
in sync so that everything can talk to each other.  I'm not sure if
for clients whether it can outsource authentication to kerberos/etc -
not a need for me but I wouldn't be surprised if this is supported.
The key syncing makes a lot more sense within the cluster itself.

* Lizardfs is MUCH simpler to set up.  For Ceph I recommend the
ansible playbook, though if I were using it in production I'd want to
do some serious config management as it seems rather complex and it
seems like the sort of thing that could take out half a datacenter if
it had a bug.  For Lizardfs if you're willing to use the suggested
hostnames about 95% of it is auto-configuring as storage nodes just
reach out to the default master DNS name and report in, and everything
trusts everything (not just by default - I don't think you even can
lock it down unless you stick every node behind a VPN to limit who can
talk to who).

-- 
Rich