Re: [gentoo-user] Slightly OT - My new SSD.

[Alan Mackenzie]

Hello, Adam.

On Mon, Mar 20, 2017 at 17:09:15 +1100, Adam Carter wrote:
> > That, indeed, seems to to be the case.  When I do cat /proc/interrupts |
> > egrep '(CPU/nvm)', I get just the header line with one data line:

> >            CPU0       CPU1       CPU2       CPU3
> >  17:          0          0         15      14605   IO-APIC 17-fasteoi ehci_hcd:usb1, nvme0q0, nvme0q1

> > I'm kind of feeling a bit out of my depth here.  What are the nvme0q0,
> > etc.?  "Queues" of some kind?  You appear to have nine of these things,
> > I've just got two.  I'm sure there's a fine manual I ought to be
> > reading.  Do you know where I might find this manual?


> Can't remember where i read up on this. Might have been troubleshooting
> poor small packet performance on a firewall (some network drivers can have
> multiqueue too). Maybe start with this;
> https://www.thomas-krenn.com/en/wiki/Linux_Multi-Queue_Block_IO_Queueing_Mechanism_(blk-mq)

Interesting article.

> It looks like the nvme driver was made "mutliqueue" in kernel 3.19.

> FWIW my system is 8 core (AMD 8350). Its odd having two queues on the same
> interrupt, but I have the same for q0 and q1, ....

I think q0 is the "administrative" queue, and the other 8 are ordinary
queues.  (Sorry, I read that somewhere, but can't remember where).

> .... but on your system i'd say there should be some queues on other
> interrupts to they can be serviced by other cores, so that doesnt look
> right.

It wasn't right.

> Do you have MSI enabled? Bus options -> PCI Support -> Message Signaled
> Interrupts (MSI and MSI-X)

I didn't have MSI enabled, but do now.  I now get 5 queues (nvmeq[0-4])
on four interrups, and the interrupts are spread over the four cores
(I've got a 4 core Athlon).

> If your system is not too old you may get more interrupts or a better
> spread with that enabled.

Yes, that is the case.  From a "warm start", I was able to copy my 1.4GB
file in 2.76s.  This is a bit more like it!

> When I look at the entire /proc/interrupts, there are just 30 lines
> > listed, and I suspect there are no more than 32 interrupt numbers
> > available.  Is there any way I can configure Linux to give my SSD more
> > than one interrupt line to work with?

> > > FWIW
> > > # hdparm -tT /dev/nvme0n1

> > > /dev/nvme0n1:
> > >  Timing cached reads:   9884 MB in  2.00 seconds = 4945.35 MB/sec
> > >  Timing buffered disk reads: 4506 MB in  3.00 seconds = 1501.84 MB/sec

> > I get:

> > /dev/nvme0n1:
> >  Timing cached reads:   4248 MB in  2.00 seconds = 2124.01 MB/sec
> >  Timing buffered disk reads: 1214 MB in  3.00 seconds = 404.51 MB/sec

> > So my "cached reads" speed is (a little under) half of yours.  This is
> > to be expected, since my PCIe lanes are only version 2 (and yours are
> > probably version 3).


> FWIW the motherboard manual says it has PCIe 2.0 x16 slots. Agree that
> cache speed is likely a hardware issue.


> > But the "buffered disk read" are much slower.  Is
> > this just the age of my PC, or might I have something suboptimally
> > configured?


> You look like you're getting SATA speeds, but since you have the nvme
> device, i guess that implies you havent fallen back to SATA.

I don't think the SSD has a SATA interface.

> Could well be older hardware or less PCIe slots/lanes.

My $ hdparm -tT /dev/nvme0n1 speeds haven't improved since enabling MSI
in the kernel.

But I'm intending to build a new machine "soon" (depending on when I can
get a suitable Ryzen motherboard), and will put this NVMe SSD into the
new box.  I have to decide whether to get another one of these NVMe SSDs
to run as RAID-1, or whether to put swap, /usr, etc. on it, and build a
RAID-1 from two cheaper SATA SSDs.

-- 
Alan Mackenzie (Nuremberg, Germany).