Re: [gentoo-user] [OT] SMR drives (WAS: cryptsetup close and device in use when it is not)

[Rich Freeman]

On Sat, Jul 31, 2021 at 8:59 AM William Kenworthy <billk@×××××××××.au> wrote:
>
> I tried using moosefs with a rpi3B in the
> mix and it didn't go well once I started adding data - rpi 4's were not
> available when I set it up.

Pi2/3s only have USB2 as far as I'm aware, and they stick the ethernet
port on that USB bus besides.  So, they're terrible for anything that
involves IO of any kind.

The Pi4 moves the ethernet off of USB, upgrades it to gigabit, and has
two USB3 hosts, so this is just all-around a missive improvement.
Obviously it isn't going to outclass some server-grade system with a
gazillion PCIe v4 lanes but it is very good for an SBC and the price.

I'd love server-grade ARM hardware but it is just so expensive unless
there is some source out there I'm not aware of.  It is crazy that you
can't get more than 4-8GiB of RAM on an affordable arm system.

> I think that SMR disks will work quite well
> on moosefs or lizardfs - I don't see long continuous writes to one disk
> but a random distribution of writes across the cluster with gaps between
> on each disk (1G network).

So, the distributed filesystems divide all IO (including writes)
across all the drives in the cluster.  When you have a number of
drives that obviously increases the total amount of IO you can handle
before the SMR drives start hitting the wall.  Writing 25GB of data to
a single SMR drive will probably overrun its CMR cache, but if you
split it across 10 drives and write 2.5GB each, there is a decent
chance they'll all have room in the cache, take the write quickly, and
then as long as your writes aren't sustained they can clear the
buffer.

I think you're still going to have an issue in a rebalancing scenario
unless you're adding many drives at once so that the network becomes
rate-limiting instead of the disks.  Having unreplicated data sitting
around for days or weeks due to slow replication performance is
setting yourself up for multiple failures.  So, I'd still stay away
from them.

If you have 10GbE then your ability to overrun those disks goes way
up.  Ditto if you're running something like Ceph which can achieve
higher performance.  I'm just doing bulk storage where I care a lot
more about capacity than performance.  If I were trying to run a k8s
cluster or something I'd be on Ceph on SSD or whatever.

> With a good adaptor, USB3 is great ... otherwise its been quite
> frustrating :(  I do suspect linux and its pedantic correctness trying
> to deal with hardware that isn't truly standardised (as in the
> manufacturer probably supplies a windows driver that covers it up) is
> part of the problem.  These adaptors are quite common and I needed to
> apply the ATA command filter and turn off UAS using the usb tweaks
> mechanism to stop the crashes and data corruption.  The comments in the
> kernel driver code for these adaptors are illuminating!

Sometimes I wonder.  I occasionally get errors in dmesg about
unaligned writes when using zfs.  Others have seen these.  The zfs
developers seem convinced that the issue isn't with zfs but it simply
is reporting the issue, or maybe it happens under loads that you're
more likely to get with zfs scrubbing (which IMO performs far worse
than with btrfs - I'm guessing it isn't optimized to scan physically
sequentially on each disk but may be doing it in a more logical order
and synchronously between mirror pairs).  Sometimes I wonder if there
is just some sort of bug in the HBA drivers, or maybe the hardware on
the motherboard.  Consumer PC hardware (like all PC hardware) is
basically a black box unless you have pretty sophisticated testing
equipment and knowledge, so if your SATA host is messing things up how
would you know?

-- 
Rich