Re: [gentoo-user] Re: btrfs fails to balance

[Bill Kenworthy]

On 21/01/15 00:03, Rich Freeman wrote:
> On Tue, Jan 20, 2015 at 10:07 AM, James <wireless@×××××××××××.com> wrote:
>> Bill Kenworthy <billk <at> iinet.net.au> writes:
>>
>>> You can turn off COW and go single on btrfs to speed it up but bugs in
>>> ceph and btrfs lose data real fast!
>>
>> Interesting idea, since I'll have raid1 underneath each node. I'll need to
>> dig into this idea a bit more.
>>
>
> So, btrfs and ceph solve an overlapping set of problems in an
> overlapping set of ways.  In general adding data security often comes
> at the cost of performance, and obviously adding it at multiple layers
> can come at the cost of additional performance.  I think the right
> solution is going to depend on the circumstances.
>
> if ceph provided that protection against bitrot I'd probably avoid a
> COW filesystem entirely.  It isn't going to add any additional value,
> and they do have a performance cost.  If I had mirroring at the ceph
> level I'd probably just run them on ext4 on lvm with no
> mdadm/btrfs/whatever below that.  Availability is already ensured by
> ceph - if you lose a drive then other nodes will pick up the load.  If
> I didn't have robust mirroring at the ceph level then having mirroring
> of some kind at the individual node level would improve availability.
>
> On the other hand, ceph currently has some gaps, so having it on top
> of zfs/btrfs could provide protection against bitrot.  However, right
> now there is no way to turn off COW while leaving checksumming
> enabled.  It would be nice if you could leave the checksumming on.
> Then if there was bitrot btrfs would just return an error when you
> tried to read the file, and then ceph would handle it like any other
> disk error and use a mirrored copy on another node.  The problem with
> ceph+ext4 is that if there is bitrot neither layer will detect it.
>
> Does btrfs+ceph really have a performance hit that is larger than
> btrfs without ceph?  I fully expect it to be slower than ext4+ceph.
> Btrfs in general performs fairly poorly right now - that is expected
> to improve in the future, but I doubt that it will ever outperform
> ext4 other than for specific operations that benefit from it (like
> reflink copies).  It will always be faster to just overwrite one block
> in the middle of a file than to write the block out to unallocated
> space and update all the metadata.
>

answer to both you and James here:

I think it was pre 8.0 when I dropped out.  Its Ceph that suffers from
bitrot - I use the "golden master" approach to generating the VM's so
corruption was obvious.  I did report one bug in the early days that
turned out to be btrfs, but I think it was largely ceph which has been
born out by consolidating the ceph trial hardware and using it with
btrfs and the same storage - rare problems and I can point to
hardware/power when it happened.

The performance hit was not due to lack of horsepower (cpu, ram etc) but
due to I/O - both network bandwidth and internal bus on the hosts.  That
is why a small number of systems no matter how powerful wont work well.
 For real performance, I saw people using SSD's and large numbers of
hosts in order to distribute the data flows - this does work and I saw
some insane numbers posted.  It also requires multiple networks
(internal and external) to separate the flows (not VLAN but dedicated
pipes) due to the extreem burstiness of the traffic.  As well as VM
images, I had backups (using dirvish) and thousands of security camera
images.  Deletes of a directory with a lot of files would take many
hours.  Same with using ceph for a mail store (came up on the ceph list
under "why is it so slow") - as a chunk server its just not suitable for
lots of small files.

Towards the end of my use, I stopped seeing bitrot on a system with data
but idle to limiting it to occurring during heavy use.  My overall
conclusion is lots of small hosts with no more than a couple of drives
each and multiple networks with lots of bandwidth is what its designed for.

I had two reasons for looking at ceph - distributed storage where data
in use was held close to the user but could be redistributed easily
with multiple copies (think two small data stores with an intermittent
WAN link storing high and low priority data) and high performance with
high availability on HW failure.

Ceph was not the answer for me with the scale I have.

BillK