Re: [gentoo-amd64] NFS on freebsd

[P.V.Anthony]

Steve Herber wrote:
> Could you document you setup and test procedure?
> I use NFS for most disk storage but have never tried to tune it much.
> Any performance suggestions are welcome.
>
> Thanks,
>
> Steve Herber    herber@×××××.com        work: 206-221-7262
> Security Engineer, UW Medicine, IT Services    home: 425-454-2399

Hi,

Let me say a big sorry for this late reply. Sorry.

Now for the details about the setup. The links for the optimizations that I got
from the net is at the bottom.

Hardware.

Motherboard: MSI Neo4 Platinum

Ram: 2GB

CPU: AMD 64 +3000
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 47
model name      : AMD Athlon(tm) 64 Processor 3000+
stepping        : 0
cpu MHz         : 1809.307
cache size      : 512 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt lm 3dnowext 3dnow
pni lahf_lm
bogomips        : 3622.42
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc


IDE drives: 80GD x 2 Raid 1 using Software Raid for the os.

Data Drives: 8 x 300GB Maxtor Sata drives on Promise EX8350 raid card set to
Raid 5 with stripe size set to 64k. The drives are split in 2 groups of 4 so
each group is a logical drive. The Promise raid card is on a pci-express slot.

Ethernet: Two on board gigabit ethernet. One Marvell and the other is Nvidea.
One more Intel gigabit ethernet on pci slot.

Here is the output from "lspci" command.
0000:00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
0000:00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
0000:00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
0000:00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
0000:00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3)
0000:00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2)
0000:00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2)
0000:00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3)
0000:00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
0000:00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
0000:00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
0000:00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
0000:01:06.0 VGA compatible controller: S3 Inc. 86c325 [ViRGE] (rev 06)
0000:01:08.0 Ethernet controller: Intel Corporation 82544GC Gigabit Ethernet
Controller (Copper) (rev 02)
0000:02:00.0 PCI bridge: Intel Corporation 80333 Segment-A PCI Express-to-PCI
Express Bridge
0000:02:00.2 PCI bridge: Intel Corporation 80333 Segment-B PCI Express-to-PCI
Express Bridge
0000:03:0e.0 RAID bus controller: Promise Technology, Inc.: Unknown device 8350
0000:05:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8053 Gigabit
Ethernet Controller (rev 15)



Software.

OS: Gentoo for 64bit
Linux version 2.6.14-gentoo (root@videoserver) (gcc version 3.4.4 (Gentoo
3.4.4-r1, ssp-3.4.4-1.0, pie-8.7.8)) #1 Tue Nov 1 19:01:57 SGT 2005

Filesystem: ext3 for OS and ext3 for data drives. Tried XFS but there was some
problems with the earlier kernel so I went with ext3. Some test with kernel
2.6.14 with XFS was good. I will go into it when I can get the better speed with
NFS.

This is what I have set on /etc/exports
/VideoDisk01    *(async,insecure,no_root_squash,rw,nohide)
/VideoDisk02    *(async,insecure,no_root_squash,rw,nohide)

Here are some test I did on the drives on the Promise Raid Card.
# time dd if=/dev/zero of=/VideoDisk02/test/lala bs=8192k count=512
512+0 records in
512+0 records out

real    0m46.754s
user    0m0.008s
sys     0m16.321s

That is 4G in 47 seconds. So that is 85MB per second.

More test the raid drives.
# hdparm -Tt /dev/sda

/dev/sda:
  Timing cached reads:   3004 MB in  2.00 seconds = 1501.91 MB/sec
  Timing buffered disk reads:  336 MB in  3.01 seconds = 111.70 MB/sec

# hdparm -Tt /dev/sdb

/dev/sdb:
  Timing cached reads:   3032 MB in  2.00 seconds = 1515.91 MB/sec
  Timing buffered disk reads:  356 MB in  3.00 seconds = 118.66 MB/sec

Test on the ethernet connection.
# nttcp -T 10.0.0.100
      Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  8388608    0.09    0.02    788.3382   2918.2842    2048  24058.17   89059.0
1  8388608    0.09    0.08    776.0673    838.8084    2195  25383.65   27435.8

another test done a second later.
# nttcp -T 10.0.0.100
      Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  8388608    0.50    0.05    134.7476   1369.7643    2048   4112.17   41801.9
1  8388608    0.52    0.07    129.5928    986.8370    3355   6478.79   49335.3

It looks like my network speed is like a bell curve. Going up steady to about 
700MBit then slowing coming down to 129MBit. Then it starts to go up then down 
again.

Could be my network. Need to do more checking.

Please do comment or make some suggestions

Yes one more thing is that kernel 2.6.15 is out and I was looking at the change 
log and there seems to be many changes with regards to nfs. Maybe 2.6.15 is 
better. I will do more test.

Here are all the links that I got for the optimizations.

http://www.enterpriseitplanet.com/networking/features/article.php/3497796
http://www-didc.lbl.gov/TCP-tuning/linux.html
http://libarynth.f0.am/cgi-bin/twiki/view/Libarynth/NFSonOsX
http://www.metaconsultancy.com/whitepapers/nfs.htm
http://naeblis.cx/rtomayko/2004/08/09/NFSAutomountOSX
http://astcomm.net/tech/nfs_howto/client/
http://dawuss.student.utwente.nl/blog/entry/38
http://lists.freebsd.org/pipermail/freebsd-net/2004-December/005997.html
http://sial.org/howto/osx/automount/
http://datatag.web.cern.ch/datatag/howto/tcp.html
http://bio3d.colorado.edu/tor/sadocs/filesys/nfs.html
http://www.psc.edu/networking/projects/tcptune/
http://people.redhat.com/alikins/system_tuning.html#tcp







-- 
gentoo-amd64@g.o mailing list