Re: [gentoo-server] iSCSI on Linux

[Kashani]

I just joined gentoo-server in the past few days and missed the
original post, but I'll post some of our experiences so far with Linux
iSCSI on the client side.

	Server side is Cisco iSCSI 5400 with RAID Inc backend. All
volume management is in the Cisco gear and is roughly 8TB usable. I've got
a single client machine, Redhat 9.0, using the RPMs from the sourceforge
project. All interfaces are GigE.

	The client is running Helix Streaming server 9.02 and Apache
2.0.49. Total outbound traffic is 70Mb/s at peak with 45Mb/s being
average. All content is served from the iSCSI array on a single LUN at
this time. By this time next week I should have the other 3 servers
coverted over and be serving 250Mb/s off the iSCSI SAN. All servers are
dual PIII 1Ghz or better, have a dedicated interface for the iSCSI lan,
and 4GB of RAM.

	Tested with jumbo frames and without on the iSCSI lan. Jumbo
frames increased performance 5-10% with regards to throughput and
decreased CPU overhead by at least 60%. CPU on the Cisco side did drop
slightly, but was generally unaffected. I'm assuming they do most of the
TCP stuff in hardware for performance. I also used ethtool to set "autoneg
on" which enabled flow control. Cisco recommended that as a way to
decrease segment retransmits, shown by netstat -s | grep seg. This did
almost eliminate bad segments and dropped segment retrans by a factor or
10. Service has been stable other than Real Server problems with the Helix
build on Redhat 9.

Some musing about our system and how it may not resemble yours:

	Our system is 100% reads, long ones as well, off the SAN. I have 
noticed that when using the current client machine to rsync new data to 
the SAN that iSCSI CPU usage increases dramatically. Also doing lots of 
reads with significant writes also increase CPU usage more then the data 
transfer would suggest.
	I currently have one client going to one lun out of four. The plan 
is to spread content evenly on the SAN. I'm not sure how this will effect 
the client and server sides. Especially once I add the other 3 client 
machines.
	None of the file we're serving are smaller than 300MB so caching 
content in RAM never happens. I don't know if webserving off a SAN with 
smaller files would see less SAN traffic when Linux/Apache caches content.
	We're doing almost no volume management. I need to talk to the 
admin to set it up so I have no idea if it's easier under Cisco and how 
the client reacts to changes.

	This was probably a bit outside the data you were looking for, but 
if you want to hear more shoot me an email. I'll probably have another 
update early next week.

kashani

On Tue, 20 Apr 2004, Bryn Hughes wrote:

> Doesn't look like I've had any bites out there so I'll share my
> findings at least for any future googlers...
>
> The good news is iSCSI on Linux works.  I have been able to export
> block devices from one Linux machine and mount them on another.
> Apparently the Windows drivers are well tested too but I don't have any
> Windows machines to test with.  The initiator sees the exported iSCSI
> LUN as if it were a locally attached SCSI disk.  If you export an LVM
> volume it appears to the remote machine as a local SCSI disk and you
> can do all the usual stuff to it - i.e. you need to either create a
> partition table or use lvm on the remote machine to portion out your
> virtual device.
>
> The bad news is the iSCSI target driver is lacking a few key functions.
>   It doesn't seem to handle resizing volumes very well - I tried
> resizing an exported LVM volume and all hell broke loose.  The biggest
> problem however is that it doesn't appear to let you make configuration
> changes without restarting the target daemon.  This is a major problem
> as it means you have to unmount _ALL_ iSCSI volumes before you can add
> a new one on the target server.  This wouldn't make for much of a SAN
> if every time you wanted to make a configuration change you had to down
> all your servers!
>
> So here's hoping that more developers will join in on the iSCSI target
> project listed below at ardistech.com.  There's definitely something
> brewing here which will make SAN's a whole lot cheaper and easier to
> implement for the average environment.  Give it some more development
> time and it will actually be viable!
>
> Bryn
>
>
> On Apr 15, 2004, at 6:48 PM, Bryn Hughes wrote:
>
> > I'm looking for experiences implementing iSCSI on Linux - not just as
> > an initiator but as a target as well.  I've found these two projects:
> >
> > This is the iSCSI initiator project started by Cisco
> > http://sourceforge.net/projects/linux-iscsi
> >
> > This is the ardistech iSCSI target project
> > http://www.ardistech.com/iscsi/
> >
> > In theory it is possible to create iSCSI volumes using EVMS or similar
> > and then mount them as block devices over ethernet to another host
> > (could be Windows, Novell, Linux, OS X, whatever).  Basically you can
> > build yourself a SAN using free software and normal ethernet
> > (obviously gigabit preferred).
> --
> ~
> ~
> :wq
>
>