Re: [gentoo-user] Is there a way to misconfigure USB ports in the kernel?

[Michael]

On Tuesday, 30 March 2021 18:11:56 BST Dr Rainer Woitok wrote:
> On Saturday, 2020-12-05 19:07:51 +0100, I myself wrote:
>
> ("> >" refers to Michael <confabulate@××××××××.com>)
>
> > Michael,
> >
> > On Friday, 2020-11-27 19:07:17 +0000, you wrote:
> > > ...
> > > A 4k block size is recommended for ntfs-3g which is the default sector
> > > created by fdisk and friends on Linux these days.  This will align your
> > > partition optimally.  In addition, mkfs.ntfs will use 4096 bytes as the
> > > default cluster size, so you should be good in that respect.
> > >
> > > Another setting you may want to try is mounting the USB with
> > > 'big_writes' -
> > > check the man page.  This should help particularly with large files,
> > > which
> > > will use larger blocks up to 128KB when copying data to the NTFS.
> >
> > Both, the VeraCrypt command line (--fs-options=big_writes) and the Vera-
> > Crypt GUI  (under "Settings  --> Preferences")  allow setting this mount
> > option.  But
> >
> >    $ mount | grep veracrypt
> >
> > never shows it,  initially causing me  to erroneously believe  it wasn't
> > set and to try finding  on the web another way  of setting it.   By pure
> > chance I finally found out that
> >
> >    $ ps -ef | grep veracrypt
> >
> > lists a  "/usr/sbin/mount.ntfs" task  which shows the  options really in
> > effect.  However,  I haven't yet had the time to test the effect of this
> > option when writing  plenty of really big files.   I will report on that
> > later.
>
> Well,  it's been quite a while,  due to my being almost permanently con-
> fronted with more pressing tasks ... :-(
>
> To sum up my experience with my new 128 GB Philips USB 3.0 sticks: while
> the Philips sticks  are significantly faster for reading operations than
> my old 64 GB Verbatim ones (probably USB 2.0), writing operations to the
> Philips sticks  are unbearably slow,  regardless of whether  I created a
> normal unencrypted NTFS filesystem on them or an encrypted NTFS filesys-
> tem using VeraCrypt.   Writing to  the USB stick  while at the same time
> reading from it in a different terminal window caused commands like "cd"
> or "ls" to simply stall.  Thus while running
>
>    $ cp --preserve=timestamps -ru $source_dir .
>
> in one terminal window, I ran
>
>    $ while true
>
>    > do n=$(ps -ef|g 'cp --preserve'|g -v grep)
>    >
>    >    if [[ "$n" = "${o-}" ]]
>    >    then sleep 10
>    >    else o="$n"
>    >
>    >         echo "$n"
>    >
>    >    fi
>    >
>    > done
>
> in another, to get the  wall clock times  when copying a new file began.
> That way I found that copying a 30 MB file took about 40 minutes.

OK, unless you made a typo and the "minutes" were meant to say seconds, this 
is ridiculously slow.

You could run some tests to see what is causing the delay.  The veracrypt 
algos & cipher iterations, the fuse based ntfs-3g, or the USB stick's 
controller.

However if, as I understand it, all other variables are the same and the only 
change was to replace your Verbatim  64G USB 2.0 sticks with Philips 128G USB 
3.0 sticks, then the slow writes point to the Philips devices being the 
culprit.

Some years ago I tested some USB 2.0 sticks of various sizes, from 256M up to 
32G and recall the smaller the USB stick the faster the write performance, so 
differences in writing speed are normal.  The writing speed you're describing 
however is a clear indication of something being wrong.


> So what are my options?
>
>    - Stay away from Philips USB 3.0 sticks?
>
>    - Stay away from Philips USB sticks in general?

Without knowing the internals, a brand may offer only an unwarranted 
assumption of performance.  We saw Western Digital disks being sold as CMR, 
while having SMR internals.  A brand could switch OEM suppliers, or 
components, making benchmarking unreliable.


>    - Stay away from USB 3.0 sticks in general?

USB 3.0 is faster and USB 3.2 when available will be even faster.  So use 
whatever the USB ports on your PC offer.


>    - Stay away from Filesystem in User Space  using a non-stable 5.10 or
>      5.11 kernel (currently I'm using stable 5.4.97)?
>
>    - Stay away from Gentoo?
>
>    - Stay away from Linux in general  and go back to OTOS  (aka the Only
>      True Operating System aka Windoze)?
>
>    - ...?

In-kernel fs drivers are measurably faster than fuse based fs for well 
understood reasons.  However, if needs must and the fs you require is not 
available on Linux, then some compromise will be required.


> Any ideas and comments welcome ...
>
> Sincerely,
>   Rainer

You may want to run some tests on the sticks you have, if only to bottom out 
what their performance is on different PCs and USB ports:

dd if=/dev/zero of=/run/media/<USER>/<STICK_LABEL>/TESTFILE bs=512 
count=600000 oflag=direct conv=notrunc,fsync status=progress

Use a large enough file to make sure the USB controller cache gets saturated.

You could use a ramdisk/tmpfs as an input file.

If you write directly to the device as Dale suggested it will wipe data, so 
keep a backup of anything you need first.

You can experiment with different filesystems and in the first opportunity 
with a different make of USB 3.0 stick.  You'd soon be able to determine how 
good the real world performance can get and if the Philips or something else 
is causing the problem you've experienced.

A note about UDF:  it works and it is versatile - but ensuring 
interoperability between different OSs can be tricky.  Check some suggestions 
here:

https://askubuntu.com/questions/27936/can-and-should-udf-be-used-as-a-hard-drive-format

and here:

https://superuser.com/questions/39942/using-udf-on-a-usb-flash-drive

Ultimately such tests are an attempt to eliminate methodically all other 
factors, until you isolate the cause of the problem you are experiencing.