Re: [gentoo-server] iSCSI on Linux

[Bryn Hughes]

Excellent... this is the kind of info I'm looking for.  Your  setup 
isn't actually that different from what I'm looking at.  I have about 
200 gigs of video that need to be stored along with another 300 gigs of 
various other multimedia - the video is all training video to be 
streamed off our web server and the other stuff is our marketing and 
media department's work files - basically their job archive.  I'm not 
looking at running a mail server or a database off of this right now, I 
figured I'd start with something simpler and less i/o intensive.  This 
is our 'proof of concept' SAN for all intents and purposes.

With regards to volume management I found that I could use all the 
normal Linux tricks with the exported LUNs - the client machine really 
doesn't know it isn't a direct attached SCSI disk.  I'm considering 
FalconStor IPStor for our SAN controller as it is Linux based (Red Hat 
unfortunately) and will handle everything from replication to snapshots 
and backups.  It basically has the same function as the Cisco storage 
router except it's more open and will work with pretty much everything 
under the sun.

Bryn


On Apr 20, 2004, at 3:14 PM, Kashani wrote:

> 	I just joined gentoo-server in the past few days and missed the
> original post, but I'll post some of our experiences so far with Linux
> iSCSI on the client side.
>
> 	Server side is Cisco iSCSI 5400 with RAID Inc backend. All
> volume management is in the Cisco gear and is roughly 8TB usable. I've
> got
> a single client machine, Redhat 9.0, using the RPMs from the
> sourceforge
> project. All interfaces are GigE.
>
> 	The client is running Helix Streaming server 9.02 and Apache
> 2.0.49. Total outbound traffic is 70Mb/s at peak with 45Mb/s being
> average. All content is served from the iSCSI array on a single LUN at
> this time. By this time next week I should have the other 3 servers
> coverted over and be serving 250Mb/s off the iSCSI SAN. All servers are
> dual PIII 1Ghz or better, have a dedicated interface for the iSCSI lan,
> and 4GB of RAM.
>
> 	Tested with jumbo frames and without on the iSCSI lan. Jumbo
> frames increased performance 5-10% with regards to throughput and
> decreased CPU overhead by at least 60%. CPU on the Cisco side did drop
> slightly, but was generally unaffected. I'm assuming they do most of
> the
> TCP stuff in hardware for performance. I also used ethtool to set
> "autoneg
> on" which enabled flow control. Cisco recommended that as a way to
> decrease segment retransmits, shown by netstat -s | grep seg. This did
> almost eliminate bad segments and dropped segment retrans by a factor
> or
> 10. Service has been stable other than Real Server problems with the
> Helix
> build on Redhat 9.
>
> Some musing about our system and how it may not resemble yours:
>
> 	Our system is 100% reads, long ones as well, off the SAN. I have
> noticed that when using the current client machine to rsync new data to
> the SAN that iSCSI CPU usage increases dramatically. Also doing lots of
> reads with significant writes also increase CPU usage more then the
> data
> transfer would suggest.
> 	I currently have one client going to one lun out of four. The plan
> is to spread content evenly on the SAN. I'm not sure how this will
> effect
> the client and server sides. Especially once I add the other 3 client
> machines.
> 	None of the file we're serving are smaller than 300MB so caching
> content in RAM never happens. I don't know if webserving off a SAN with
> smaller files would see less SAN traffic when Linux/Apache caches
> content.
> 	We're doing almost no volume management. I need to talk to the
> admin to set it up so I have no idea if it's easier under Cisco and how
> the client reacts to changes.
>
> 	This was probably a bit outside the data you were looking for, but
> if you want to hear more shoot me an email. I'll probably have another
> update early next week.
>
> kashani
>
> On Tue, 20 Apr 2004, Bryn Hughes wrote:
>
>> Doesn't look like I've had any bites out there so I'll share my
>> findings at least for any future googlers...
>>
>> The good news is iSCSI on Linux works.  I have been able to export
>> block devices from one Linux machine and mount them on another.
>> Apparently the Windows drivers are well tested too but I don't have
>> any
>> Windows machines to test with.  The initiator sees the exported iSCSI
>> LUN as if it were a locally attached SCSI disk.  If you export an LVM
>> volume it appears to the remote machine as a local SCSI disk and you
>> can do all the usual stuff to it - i.e. you need to either create a
>> partition table or use lvm on the remote machine to portion out your
>> virtual device.
>>
>> The bad news is the iSCSI target driver is lacking a few key
>> functions.
>>   It doesn't seem to handle resizing volumes very well - I tried
>> resizing an exported LVM volume and all hell broke loose.  The biggest
>> problem however is that it doesn't appear to let you make
>> configuration
>> changes without restarting the target daemon.  This is a major problem
>> as it means you have to unmount _ALL_ iSCSI volumes before you can add
>> a new one on the target server.  This wouldn't make for much of a SAN
>> if every time you wanted to make a configuration change you had to
>> down
>> all your servers!
>>
>> So here's hoping that more developers will join in on the iSCSI target
>> project listed below at ardistech.com.  There's definitely something
>> brewing here which will make SAN's a whole lot cheaper and easier to
>> implement for the average environment.  Give it some more development
>> time and it will actually be viable!
>>
>> Bryn
>>
>>
>> On Apr 15, 2004, at 6:48 PM, Bryn Hughes wrote:
>>
>>> I'm looking for experiences implementing iSCSI on Linux - not just as
>>> an initiator but as a target as well.  I've found these two projects:
>>>
>>> This is the iSCSI initiator project started by Cisco
>>> http://sourceforge.net/projects/linux-iscsi
>>>
>>> This is the ardistech iSCSI target project
>>> http://www.ardistech.com/iscsi/
>>>
>>> In theory it is possible to create iSCSI volumes using EVMS or
>>> similar
>>> and then mount them as block devices over ethernet to another host
>>> (could be Windows, Novell, Linux, OS X, whatever).  Basically you can
>>> build yourself a SAN using free software and normal ethernet
>>> (obviously gigabit preferred).
>> --
>> ~
>> ~
>> :wq
>>
>>
>>
--
~
~
:wq