Re: [gentoo-server] Atrocious NFS performance

[Matthias Bethke]

Hi Jeroen,

To make a long story short (I'll tell it anyway for the archives' sake),
the problem seems solved so far. After recompiling the kernel with 100Hz
ticks I get a slight increase in latency when doing the same things as
yesterday on the server but it feels just as responsive as before.
Lacking any benchmarks from before the change, that's the best measure I
have. It might have been the firmware upgrade, but I doubt it. If I have
to reboot any time soon, maybe I'll do another test to verify it.

on Wednesday, 2006-04-19 at 20:42:09, you wrote:
> >The slowness is the same on SuSE and Gentoo based clients. The previous
> >installation handled the same thing without any problems, which I'd
> >certainly expect from a dual Xeon @3 GHz with $ GB RAM, a Compaq
> >SmartArray 642 U320 hostadpater and some 200 GB in a RAID5, connected
> >to the clients via GBit ethernet.
> >
> RAID-5? Ouch.
> RAID-10 offers a much better raw performance; since individual mirrors
> are striped, you get at least 4/3 the seek performance of a 4-disk

Yeah, but also at 2/3 the capacity. I know RAID5 isn't exactly
top-notch, but as long as the controller takes care of the checksumming
and distribution and the CPU doesn't have to, it's good enough for our
site. That's mostly students doing their exercises, webbrowsing, some
programming, usually all with small datasets. The biggest databases are
about two gigs and the disks write at just above 40 MB/s.

> >Definitely not good for GBit, but not so bad either considering it will
> >have taken half a minute just to open that file. The file is complete
> >despite the I/O error but the error is definitely related to the server
> >load, it never happens normally (and I get 9-11s for the 100 MB).
> >
> LoadAvg of over 10 for I/O only ? That is a serious problem.
> I repeat, that is a *problem*, not bad performance.

Huh? No, 9 to 11 seconds, i.e. ~10 MB/s. I don't see a way how this
benchmark could possibly bring my load up that much, after all it's just
one process on the client and one on the server.

> Since you say the box has 4GB of RAM, what happens when you do a linear
> read of 2 or 3 GB of data, first uncached and then cached ?
> That should not be affected by the I/O subsystem at all.

Writing gives me said 40 MB/s, reading it back (dd to /dev/null in 1 MB
chunks) is 32 MB/s uncached (*slower* than writes? Hm. controller
caching maybe...) , ~850 MB/s cached.

> Also, test your network speed by running netperf or ioperf between
> client and server.
> Get some baseline values for maximum performance first!

I didn't test it as the only thing I changed was the server software and
it was just fine before. And it *is* fine as long as the server disks
aren't busy. Theoretically it could be that the Broadcom NIC driver
started sucking donkey balls in kernel 2.6, but ssh and stuff are just
fine and speedy (~30 MB/s for a single stream of zeroes).

> And more bla I don't understand about NFS - what about the basics ?
> Which versions are the server and client running ?
> Since both could run either v2 or v3 and in-kernel or userspace, that's
> 4 x 4 = 16 possible combinations right there - and that is assuming they
> both run the *same* minor versions of the NFS software.

It's v3, that's why I snipped the unused v2 portions of nfsstat output.
Both server and client are in-kernel---the client could only be
userspace via FUSE, right?---and the latest stable versions,
nfs-utils-1.0.6-r6, gentoo-sources-2.6.15-r1 on the client and
hardened-sources-2.6.14-r7 on the server.

> >And one parameter I haven't tried to tweak is the IO scheduler. I seem
> >to remember a recommendation to use noop for RAID5 as the cylinder
> >numbers are completely virtual anyway so the actual head scheduling
> >should be left to the controller. Any opinions on this?
> >
> I have never heard of the I/O scheduler being able to influence or get
> data directly from disks.
> In fact, as far as I know that is not even possible with IDE or SCSI,
> which both have their own abstraction layers.
> What you probably mean is the way the scheduler is allowed to interface
> with the disk subsystem - which is solely determined by the disk
> subsystem itself.

OK, that was a bit misleading, I meant that even assuming things about
the flat file the scheduler sees of the disk like that offsets in the
file sort of linearly correspond to cylinders---which is what it does to
implement things like the elevator algorithm---are virtually always
right for simple drives but may not be for a RAID.

> I'd recommend reading the specs for the raid controller - twice.
> Also dive into the module source if you're up for it - it can reveal a
> lot more than just plugging it in and adding disks.

Ugh...I read the O'Reilly book on Linux device drivers so I know some
basics of it (up to kernel 2.4 that is) but I'd rather not touch the
200+ KB of the cciss source as my first real project, especially not
when my only test hardware is the production server...

cheers!
	Matthias
-- 
I prefer encrypted and signed messages. KeyID: FAC37665
Fingerprint: 8C16 3F0A A6FC DF0D 19B0  8DEF 48D9 1700 FAC3 7665