Re: [gentoo-user] [OT] SMR drives (WAS: cryptsetup close and device in use when it is not)

[William Kenworthy]

On 31/7/21 9:30 pm, Rich Freeman wrote:
> On Sat, Jul 31, 2021 at 8:59 AM William Kenworthy <billk@×××××××××.au> wrote:
>> I tried using moosefs with a rpi3B in the
>> mix and it didn't go well once I started adding data - rpi 4's were not
>> available when I set it up.
> Pi2/3s only have USB2 as far as I'm aware, and they stick the ethernet
> port on that USB bus besides.  So, they're terrible for anything that
> involves IO of any kind.
>
> The Pi4 moves the ethernet off of USB, upgrades it to gigabit, and has
> two USB3 hosts, so this is just all-around a missive improvement.
> Obviously it isn't going to outclass some server-grade system with a
> gazillion PCIe v4 lanes but it is very good for an SBC and the price.
>
> I'd love server-grade ARM hardware but it is just so expensive unless
> there is some source out there I'm not aware of.  It is crazy that you
> can't get more than 4-8GiB of RAM on an affordable arm system.
Checkout the odroid range.  Same or only slightly $$$ more for a much
better unit than a pi (except for the availability of 8G ram on the pi4)
None of the pi's I have had have come close though I do not have a pi4
and that looks from reading to be much closer in performance.  The
Odroid sites includes comparison charts of odroid aganst the rpi and it
also shows it getting closer in performance.  There are a few other
companies out there too.  I am hoping the popularity of the pi 8G will
push others to match it. I found the supplied 4.9 or 4.14 kernels
problematic with random crashes, espicially if usb was involved.  I am
currently using the 5.12 tobetter kernels and aarch64 or arm32 bit
gentoo userlands.
>
>> I think that SMR disks will work quite well
>> on moosefs or lizardfs - I don't see long continuous writes to one disk
>> but a random distribution of writes across the cluster with gaps between
>> on each disk (1G network).
> So, the distributed filesystems divide all IO (including writes)
> across all the drives in the cluster.  When you have a number of
> drives that obviously increases the total amount of IO you can handle
> before the SMR drives start hitting the wall.  Writing 25GB of data to
> a single SMR drive will probably overrun its CMR cache, but if you
> split it across 10 drives and write 2.5GB each, there is a decent
> chance they'll all have room in the cache, take the write quickly, and
> then as long as your writes aren't sustained they can clear the
> buffer.
Not strictly what I am seeing.  You request a file from MFS and the
first first free chunkserver with the data replies.  Writing is similar
in that (depending on the creation arguments) a chunk is written
wherever responds fastest then replicated.  Replication is done under
control of an algorithm that replicates a set number of chunks at a time
between a limited number of chunkservers in a stream depending on
replication status.  So I am seeing individual disk activity that is
busy for a few seconds, and then nothing for a short period - this
pattern has become more pronounced as I added chunkservers and would
seem to match the SMR requirements.  If I replace/rebuild (resilver) a
chunkserver, that one is a lot busier, but still not 100% continuous
write or read.  Moosefs uses a throttled replication methodology.  This
is with 7 chunkservers and 9 disks - more is definitely better for
performance.
> I think you're still going to have an issue in a rebalancing scenario
> unless you're adding many drives at once so that the network becomes
> rate-limiting instead of the disks.  Having unreplicated data sitting
> around for days or weeks due to slow replication performance is
> setting yourself up for multiple failures.  So, I'd still stay away
> from them.
I think at some point I am going to have to add an SMR disk and see what
happens - cant do it now though.
>
> If you have 10GbE then your ability to overrun those disks goes way
> up.  Ditto if you're running something like Ceph which can achieve
> higher performance.  I'm just doing bulk storage where I care a lot
> more about capacity than performance.  If I were trying to run a k8s
> cluster or something I'd be on Ceph on SSD or whatever.
Tried ceph - run away fast :) I have a lot of nearly static data but
also a number of lxc instances (running on an Odroid N2) with both the
LXC instance and data stored on the cluster.  These include email,
calendaring, dns, webservers etc. all work well.  The online borgbackup
repos are also stored on it. Limitations on community moosefs is the
single point of failure that is the master plus the memory resource
requirements on the master.  I improved performance and master memory
requirements considerably by pushing the larger data sets (e.g., Gib of
mail files) into a container file stored on MFS and loop mounted onto
the mailserver lxc instance.  Convoluted but very happy with the
improvement its made.
>> With a good adaptor, USB3 is great ... otherwise its been quite
>> frustrating :(  I do suspect linux and its pedantic correctness trying
>> to deal with hardware that isn't truly standardised (as in the
>> manufacturer probably supplies a windows driver that covers it up) is
>> part of the problem.  These adaptors are quite common and I needed to
>> apply the ATA command filter and turn off UAS using the usb tweaks
>> mechanism to stop the crashes and data corruption.  The comments in the
>> kernel driver code for these adaptors are illuminating!
> Sometimes I wonder.  I occasionally get errors in dmesg about
> unaligned writes when using zfs.  Others have seen these.  The zfs
> developers seem convinced that the issue isn't with zfs but it simply
> is reporting the issue, or maybe it happens under loads that you're
> more likely to get with zfs scrubbing (which IMO performs far worse
> than with btrfs - I'm guessing it isn't optimized to scan physically
> sequentially on each disk but may be doing it in a more logical order
> and synchronously between mirror pairs).  Sometimes I wonder if there
> is just some sort of bug in the HBA drivers, or maybe the hardware on
> the motherboard.  Consumer PC hardware (like all PC hardware) is
> basically a black box unless you have pretty sophisticated testing
> equipment and knowledge, so if your SATA host is messing things up how
> would you know?
>
BillK