Re: [gentoo-user] snapshots?

[Rich Freeman]

On Fri, Jan 1, 2016 at 5:42 AM, lee <lee@××××××××.de> wrote:
> "Stefan G. Weichinger" <lists@×××××.at> writes:
>
>> btrfs offers RAID-like redundancy as well, no mdadm involved here.
>>
>> The general recommendation now is to stay at level-1 for now. That fits
>> your 2-disk-situation.
>
> Well, what shows better performance?  No btrfs-raid on hardware raid or
> btrfs raid on JBOD?

I would run btrfs on bare partitions and use btrfs's raid1
capabilities.  You're almost certainly going to get better
performance, and you get more data integrity features.  If you have a
silent corruption with mdadm doing the raid1 then btrfs will happily
warn you of your problem and you're going to have a really hard time
fixing it, because btrfs only sees one copy of the data which is bad,
and all mdadm can tell you is that the data is inconsistent with no
idea which one is right.  You'd end up having to try to manipulate the
underlying data to figure out which one is right and fix it (the data
is all there, but you'd probably end up hex-editing your disks).  If
you were using btrfs raid1 you'd just run a scrub and it would
detect/fix the problem, since btrfs would see both copies and know
which one is right.  Then if you ever move to raid5 when that matures
you eliminate the write hole with btrfs.

>>
>> I would avoid converting and stuff.
>>
>> Why not try a fresh install on the new disks with btrfs?
>
> Why would I want to spend another year to get back to where I'm now?

I wouldn't do a fresh install.  I'd just set up btrfs on the new disks
and copy your data over (preserving attributes/etc).  Before I did
that I'd create any subvolumes you want to have on the new disks and
copy the data into them.  The only way to convert a directory into a
subvolume after the fact is to create a subvolume with the new name,
copy the directory into it, and then rename the directory and
subvolume to swap their names, then delete the old directory.  That is
time-consuming, and depending on what directory you're talking about
you might want to be in single-user or boot from a rescue disk to do
it.

I wouldn't do an in-place ext4->btrfs conversion.  I know that there
were some regressions in that feature recently and I'm not sure where
it stands right now.

>> I never had /boot on btrfs so far, maybe others can guide you with this.
>>
>> My /boot is plain extX on maybe RAID1 (differs on
>> laptops/desktop/servers), I size it 500 MB to have space for multiple
>> kernels (especially on dualboot-systems).
>>
>> Then some swap-partitions, and the rest for btrfs.
>
> There you go, you end up with an odd setup.  I don't like /boot
> partitions.  As well as swap partitions, they need to be on raid.  So
> unless you use hardware raid, you end up with mdadm /and/ btrfs /and/
> perhaps ext4, /and/ multiple partitions.

With grub2 you can boot from btrfs.  I used to use a separate boot
partition on ext4 with btrfs for the rest, but now my /boot is on my
root partition.  I'd still partition space for a boot partition in
case you move to EFI in the future but I wouldn't bother formatting it
or setting it up right now.  As long as you're using grub2 you really
don't need to do anything special.

You DO need to partition your disks though, even if you only have one
big partition for the whole thing.  The reason is that this gives
space for grub to stick its loaders/etc on the disk.

I don't use swap.  If I did I'd probably set up an mdadm array for it.
According to the FAQ btrfs still doesn't support swap from a file.

There isn't really anything painful about that setup though.  Swap
isn't needed to boot, so openrc/systemd will start up mdadm and
activate your swap.  I'm not sure if dracut will do that during early
boot or not, but it doesn't really matter if it does.

If you have two drives I'd just set them up as:
sd[ab]1 - 1GB boot partition unformatted for future EFI
sd[ab]2 - mdadm raid1 for swap
sd[ab]3 - btrfs


> When you use hardware raid, it
> can be disadvantageous compared to btrfs-raid --- and when you use it
> anyway, things are suddenly much more straightforward because everything
> is on raid to begin with.

I'd stick with mdadm.  You're never going to run mixed
btrfs/hardware-raid on a single drive, and the only time I'd consider
hardware raid is with a high quality raid card.  You'd still have to
convince me not to use mdadm even if I had one of those lying around.

>> Create your btrfs-"pool" with:
>>
>> # mkfs.btrfs -m raid1 -d raid1 /dev/sda3 /dev/sdb3
>>
>> Then check for your btrfs-fs with:
>>
>> # btrfs fi show
>>
>> Oh: I realize that I start writing a howto here ;-)
>
> That doesn't work without an extra /boot partition?

It works fine without a boot partition if you're using grub2.  If you
want to use grub legacy you'll need a boot partition.

>
> How's btrfs's performance when you use swap files instead of swap
> partitions to avoid the need for mdadm?

btrfs does not support swap files at present.  When it does you'll
need to disable COW for them (using chattr) otherwise they'll be
fragmented until your system grinds to a halt.  A swap file is about
the worst case scenario for any COW filesystem - I'm not sure how ZFS
handles them.

>
> Now I understand that it's apparently not possible to simply make a
> btrfs-raid1 from the two raw disks, copy the system over, install grub
> and boot from that.  (I could live with swap files instead of swap
> partitions.)

Even if you used no swap and no boot like I have right now, you'd
still want to create a single large partition for better grub2
support.  Without space between the partition table and the first
partition (which you'll want to start at 2048 or whatever the default
is these days) it has to resort to blocklists.  That means that if for
any reason the files in /boot/grub move on disk your system won't
boot.  That isn't a btrfs thing - it holds just as true if you're
using ext4 and is generally frowned upon.

>
>> As mentioned here several times I am using btrfs on >6 of my systems for
>> years now. And I don't look back so far.
>
> And has it always been reliable?
>

I've never had an episode that resulted in actual data loss.  I HAVE
had an episode or two which resulted in downtime.

When I've had btrfs issues I can generally mount the filesystem
read-only just fine.  The problem was that cleanup threads were
causing kernel BUGs which cause the filesystem to stop syncing (not a
full panic, but when all your filesystems are effectively read-only
there isn't much difference in many cases).  If I rebooted the system
would BUG within a few minutes.  In one case I was able to boot from a
more recent kernel on a rescue disk and fix things by just mounting
the drive and letting it sit for 20min to finish cleaning things up
while the disk was otherwise idle (some kind of locking issue most
likely) - maybe I had to run btrfsck on it.  In the other case it was
being really fussy and I ended up just restoring from a backup since
that was the path of least resistance.  I could have probably
eventually fixed the problem, and the drive was mountable read-only
the entire time so given sufficient space I could have copied all the
data over to a new filesystem with no loss at all.

Things have been pretty quiet for the last six months though, and I
think it is largely due to a change in strategy around kernel
versions.  Right now I'm running 3.18.  I'm starting to consider a
move to 4.1, but there is a backlog of btrfs fixes for stable that I'm
waiting for Greg to catch up on and maybe I'll wait for a version
after that to see if things settle down.  Around the time of
3.14->3.18 btrfs maturity seemed to settle in a bit, and at this point
I think newer kernels are more likely to introduce regressions than
fix problems.  The pace of btrfs patching seems to have increased as
well in the last year (which is good in the long-term - most are
bugfixes - but in the short term even bugfixes can introduce bugs).
Unless I have a reason not to at this point I plan to run only
longterm kernels, and move to them when they're about six months
mature.

If I had done that in the past I think I would have completely avoided
that issue that required me to restore from backups.  That happened in
the 3.15/3.16 timeframe and I'd have never even run those kernels.
They were stable kernels at the time, and a few versions in when I
switched to them (I was probably just following gentoo-sources stable
keywords back then), but they still had regressions (fixes were
eventually backported).

I think btrfs is certainly usable today, though I'd be hesitant to run
it on production servers depending on the use case (I'd be looking for
a use case that actually has a significant benefit from using btrfs,
and which somehow mitigates the risks).

Right now I keep a daily rsnapshot (rsync on steroids - it's in the
Gentoo repo) backup of my btrfs filesystems on ext4.  I occasionally
debate whether I still need it, but I sleep better knowing I have it.
This is in addition to my daily duplicity cloud backups of my most
important data (so, /etc and /home are in the cloud, and mythtv's
/var/video is just on a local rsync backup).

Oh, and don't go anywhere near btrfs raid5/6 (btrfs on top of mdadm
raid5/6 is fine, but you lose the data integrity features).  I
wouldn't go anywhere near that for at least a year, and probably
longer.

Overall I'm very happy with btrfs though.  Snapshots and reflinks are
very handy - I can update containers and nfs roots after snapshotting
them and it gives me a trivial rollback solution, and while I don't
use snapper I do manually rotate through snapshots weekly.  If you do
run snapper I'd probably avoid generating large numbers of snapshots -
one of my BUG problems happened as a result of snapper deleting a few
hundred snapshots at once.

Btrfs's deferred processing of the log/btrees can cause the kinds of
performance issues associated with garbage collection (or BUGs due to
thundering herd problems).  I use ionice to try to prioritize my IO so
that stuff like mythtv recordings will block less realtime activities,
and in the past that hasn't always worked with btrfs.  The problem is
that btrfs would accept too much data into its log, and then it would
block all writes while it tried to catch up.  I haven't seen that as
much recently, so maybe they're getting better about that.  As with
any other scheduling problem it only works if you correctly block
writes into the start of the pipeline (I've heard of similar problems
with TCP QoS and such if you don't ensure that the bottleneck is the
first router along the route - you can let in too much low-priority
traffic and then at that point you're stuck dealing with it).

I'd suggest looking at the btrfs mailing list to get a survey for what
people are dealing with.  Just ignore all the threads marked as
patches and look at the discussion threads.

If you're getting the impression that btrfs isn't quite
fire-and-forget, you're getting the right impression.  Neither is
Gentoo, so I wouldn't let that alone scare you off.  But, I see no
reason to not give you fair warning.

-- 
Rich