Re: [gentoo-user] Re: ceph on gentoo?

[Rich Freeman]

On Wed, Dec 24, 2014 at 5:16 AM, Holger Hoffstätte
<holger.hoffstaette@××××××××××.com> wrote:
>
> There's light and deep scrub; the former does what you described,
> while deep does checksumming. In case of mismatch it should create
> a quorum. Whether that actually happens and/or works is another
> matter. ;)
>

If you have 10 copies of a file and 9 are identical and 1 differs,
then there is little risk in this approach.  The problem is that if
you have two copies of the file and they are different, all it can do
is pick one, which is what I believe it does.  So, not only isn't this
as efficient as n+2 raid, or 2*n raid, but you end up needing 3-4*n
redundancy.  That is a LOT of wasted space simply to avoid having a
checksum.

> Unfortunately a full point-in-time deep scrub and the resulting creation
> of checksums is more or less economically unviable with growing amounts
> of data; this really should be incremental.

Since checksums aren't stored anywhere, you end up having to scan
every node and compare all the checksums across them.  Depending on
how that works it is likely to be a fairly synchronous operation,
which makes it much harder to deal with file access during the
operation.  If they just sequentially scan each disk, create an index,
sort the index, and then pass it on to some central node to do all the
comparisions that would be better than doing it completely
synchronously.

>
> I know how btrfs scrub works, but it too (and in fact every storge system)
> suffers from the problem of having to decide which copy is "good"; they
> all have different points in their timeline where they need to make a
> decision at which a checksum is considered valid. When we're talking
> about preventing bitrot, just having another copy is usually enough.
>
> On top of that btrfs will at least tell you which file is suspected,
> thanks to its wonderful backreferences.

btrfs maintains checksums for every block on the disk apart from those
blocks.  Sure, if your metadata and data all gets corrupted at once
you could have problems, but you'll at least know that you have
problems.  A btrfs scrub is asynchronous - each disk can be checked
independently of the others as there is no need to compare checksums
for files across disks, since the checksums are pre-calculated.  If a
bad extent is found, it is re-copied from one of the good disks (which
of course is synchronous).

Since the scans are asynchronous it performs a lot better than a RAID
scrub, since a read against a mirror can be allowed to disrupt just
one of the device scrubs while the other proceeds.  Indeed, you could
just scrub the devices one at a time and then only writes or parallel
reads take a hit (for mirrored mode).

Btrfs is of course immature and can't recover errors for raid5/6
modes, and of course those raid modes would not perform as well when
being scrubbed since a read requires access to n disks and a write
requires access to n+1/2 disks (note though that the use of checksums
makes it safe to do a read without reading full parity - I have no
idea if the btrfs implementation takes advantage of this).

For a single-node system btrfs (and of course zfs) have a much more
robust design IMHO.  Now, of course the node itself becomes the
bottleneck and that is what ceph is intended to handle.  The problem
is that like pre-zfs RAID it handles total failure well, and data
corruption less-well.  Indeed, unless it always checks multiple nodes
on every read a silent corruption is probably not going to be detected
without a scrub (while btrfs and zfs compare checksums on EVERY read,
since that is much less expensive than reading multiple devices).

I'm sure this could be fixed in ceph, but it doesn't seem like anybody
is prioritizing that.

--
Rich