Re: [gentoo-user] Re: [offtopic] Copy-On-Write ?

[Rich Freeman]

On Sat, Sep 16, 2017 at 9:43 AM, Kai Krakow <hurikhan77@×××××.com> wrote:
>
> Actually, I'm running across 3x 1TB here on my desktop, with mraid1 and
> draid 0. Combined with bcache it gives confident performance.
>

Not entirely sure I'd use the word "confident" to describe a
filesystem where the loss of one disk guarantees that:
1.  You will lose data (no data redundancy).
2.  But the filesystem will be able to tell you exactly what data you
lost (as metadata will be fine).

>
> I was very happy a long time with XFS but switched to btrfs when it
> became usable due to compression and stuff. But performance of
> compression seems to get worse lately, IO performance drops due to
> hogged CPUs even if my system really isn't that incapable.
>

Btrfs performance is pretty bad in general right now.  The problem is
that they just simply haven't gotten around to optimizing it fully,
mainly because they're more focused on getting rid of the data
corruption bugs (which is of course the right priority).  For example,
with raid1 mode btrfs picks the disk to use for raid based on whether
the PID is even or odd, without any regard to disk utilization.

When I moved to zfs I noticed a huge performance boost.

Fundamentally I don't see why btrfs can't perform just as well as the
others.  It just isn't there yet.

> What's still cool is that I don't need to manage volumes since the
> volume manager is built into btrfs. XFS on LVM was not that flexible.
> If btrfs wouldn't have this feature, I probably would have switched
> back to XFS already.

My main concern with xfs/ext4 is that neither provides on-disk
checksums or protection against the raid write hole.

I just switched motherboards a few weeks ago and either a connection
or a SATA port was bad because one of my drives was getting a TON of
checksum errors on zfs.  I moved it to an LSI card and scrubbed, and
while it took forever and the system degraded the array more than once
due to the high error rate, eventually it patched up all the errors
and now the array is working without issue.  I didn't suffer more than
a bit of inconvenience but with even mdadm raid1 I'd have had a HUGE
headache trying to recover from that (doing who knows how much
troubleshooting before realizing I had to do a slow full restore from
backup with the system down).

I just don't see how a modern filesystem can get away without having
full checksum support.  It is a bit odd that it has taken so long for
Ceph to introduce it, and I'm still not sure if it is truly
end-to-end, or if at any point in its life the data isn't protected by
checksums.  If I were designing something like Ceph I'd checksum the
data at the client the moment it enters storage, then independently
store the checksum and data, and then retrieve both and check it at
the client when the data leaves storage.  Then you're protected
against corruption at any layer below that.  You could of course have
additional protections to catch errors sooner before the client even
sees them.  I think that the issue is that Ceph was really designed
for object storage originally and they just figured the application
would be responsible for data integrity.

The other benefit of checksums is that if they're done right scrubs
can go a lot faster, because you don't have to scrub all the
redundancy data synchronously.  You can just start an idle-priority
read thread on every drive and then pause it anytime a drive is
accessed, and an access on one drive won't slow down the others.  With
traditional RAID you have to read all the redundancy data
synchronously because you can't check the integrity of any of it
without the full set.  I think even ZFS is stuck doing synchronous
reads due to how it stores/computes the checksums.  This is something
btrfs got right.

>
>>  For the moment I'm
>> relying more on zfs.
>
> How does it perform memory-wise? Especially, I'm currently using bees[1]
> for deduplication: It uses a 1G memory mapped file (you can choose
> other sizes if you want), and it picks up new files really fast, within
> a minute. I don't think zfs can do anything like that within the same
> resources.

I'm not using deduplication, but my understanding is that zfs deduplication:
1.  Works just fine.
2.  Uses a TON of RAM.

So, it might not be your cup of tea.  There is no way to do
semi-offline dedup as with btrfs (not really offline in that the
filesystem is fully running - just that you periodically scan for dups
and fix them after the fact, vs detect them in realtime).    With a
semi-offline mode then the performance hits would only come at a time
of my choosing, vs using gobs of RAM all the time to detect what are
probably fairly rare dups.

That aside, I find it works fine memory-wise (I don't use dedup).  It
has its own cache system not integrated fully into the kernel's native
cache, so it tends to hold on to a lot more ram than other
filesystems, but you can tune this behavior so that it stays fairly
tame.

-- 
Rich