[gentoo-user] Re: btrfs fails to balance

[James]

Bill Kenworthy <billk <at> iinet.net.au> writes:


> > The main thing keeping me away from CephFS is that it has no mechanism
> > for resolving silent corruption.  Btrfs underneath it would obviously
> > help, though not for failure modes that involve CephFS itself.  I'd
> > feel a lot better if CephFS had some way of determining which copy was
> > the right one other than "the master server always wins."


The "Giant" version 0.87 is a major release with many new fixes;
it may have the features you need. Currently the ongoing releases are
up to : v0.91. The readings look promissing, but I'll agree it
needs to be tested with non-critical data.

http://ceph.com/docs/master/release-notes/#v0-87-giant

http://ceph.com/docs/master/release-notes/#notable-changes


> Forget ceph on btrfs for the moment - the COW kills it stone dead after
> real use.  When running a small handful of VMs on a raid1 with ceph -
> sloooooooooooow :)

I'm staying away from VMs. It's spark on top of mesos I'm after. Maybe
docker or another container solution, down the road.

I read where some are using a SSD with raid 1 and bcache to speed up
performance and stability a bit. I do not want to add SSD to the mix right
now, as the (3) node development systems all have 32 G of ram.



> You can turn off COW and go single on btrfs to speed it up but bugs in
> ceph and btrfs lose data real fast!

Interesting idea, since I'll have raid1 underneath each node. I'll need to
dig into this idea a bit more.


> ceph itself (my last setup trashed itself 6 months ago and I've given
> up!) will only work under real use/heavy loads with lots of discrete
> systems, ideally 10G network, and small disks to spread the failure
> domain.  Using 3 hosts and 2x2g disks per host wasn't near big enough :(
>  Its design means that small scale trials just wont work.

Huh. My systems are FX8350 (8)processors running at 4GHz with 32 G ram.
Water coolers will allow me to crank up the speed (when/if needed) to
5 or 6 GHz. Not intel but  low end either.


> Its not designed for small scale/low end hardware, no matter how
> attractive the idea is :(

Supposedly there are tool to measure/monitor ceph better now. That is
one of the things I need to research. How to manage the small cluster
better and back off the throughput/load while monitoring performance
on a variety of different tasks. Definitely not a production usage.

I certainly appreciate your ceph_experiences. I filed a but with the
version request for Giant v0.87. Did your run the 9999 version ?
What versions did you experiment with?

I hope to set up Anisble to facilitate rapid installations of a variety
of gentoo systems used for cluster or ceph testing. That way configurations
should be able to "reboot" after bad failures.  Did your experienced
failures with Ceph require the gentoo-btrfs based systems to be complete
reinstalled from scratch, or just purge the disk of Ceph and reconfigure Ceph?

I'm hoping to "configure ceph" in such a way that failures do not corrupt
the gentoo-btrfs installation and only require repair to ceph; so your
comments on that strategy are most welcome.




> BillK


James


>