| 1 |
Hi folks, |
| 2 |
|
| 3 |
this message is rather lengthy. If you don't feel like reading all of it |
| 4 |
please don't bother to answer. You'll need the whole lot to get the |
| 5 |
picture. ;-) |
| 6 |
|
| 7 |
I have run into a weird network problem with 1Gb NICs. It involves these two |
| 8 |
boxes: |
| 9 |
|
| 10 |
Box A |
| 11 |
P4 2.8Ghz HT |
| 12 |
512GB ram |
| 13 |
Tigon Gb NIC (module tg3) |
| 14 |
IDE drives |
| 15 |
|
| 16 |
Box B |
| 17 |
Xeon 2.6Ghz HT |
| 18 |
512GB ram |
| 19 |
Intel Pro/1000 Gb NIC (module e1000) |
| 20 |
SCSI RAID5 |
| 21 |
|
| 22 |
The two of them are connected by a cross-over cable. So nothing else is on |
| 23 |
that network, kinda peer-to-peer connection. Both boxes are running *exactly* |
| 24 |
the same gentoo software. I emerged it on one box, tarred it up, copied it |
| 25 |
over to the other one and made the config changes like IP addresses, names |
| 26 |
and such. Kernel is 2.6.12-gentoo-r6. Of course, box B loads the SCSI |
| 27 |
modules. All file transfers I am talking about are done with a file |
| 28 |
"all.tar.bz2" of the size of 1088MB. Both boxes are idle otherwise. Neither |
| 29 |
box runs services like FTP or HTTP. So I have to resort to other protocols to |
| 30 |
transfer files. Both do run NFS and SSH. |
| 31 |
|
| 32 |
Case 1: |
| 33 |
I log into A and NFS mount B's /tmp on A's /mnt/floppy and cd to /tmp. |
| 34 |
"cp /mnt/floppy/all.tar.bz2 ." (receiving on A) as well as "cp |
| 35 |
all.tar.bz2 /mnt/floppy" (sending from A) result in a sustained transfer rate |
| 36 |
of 2xMB/s. That's to be expected because it involves an IDE drive on A, and |
| 37 |
that's about the limit of current IDE drives (though 1Gb NICs can transfer |
| 38 |
data at about 4 to 5 times that rate). It also confirms that both Gb NICs are |
| 39 |
performing though it doesn't confirm they are getting near their theoretical |
| 40 |
limits (the latter unimportant in this case). |
| 41 |
|
| 42 |
Case 2: |
| 43 |
I log into A and sftp into B. "get all.tar.bz2" (receiving on A) transfers the |
| 44 |
file at 2xMB/s, same as in case 1. CPU utilisation is up to 40-50% due to |
| 45 |
encryption. Still, encryption does not slow down the transfer rate by any |
| 46 |
significant amount. This can be expected with the CPUs involved. |
| 47 |
|
| 48 |
Case 3: |
| 49 |
I log into A and sftp into B. "put all.tar.bz2" (sending from A) transfers the |
| 50 |
file at 3.7MB/s!!!!! This is far slower than on a 100baseT network where I |
| 51 |
get transfer rates of about 10MB/s with the network being the bottleneck |
| 52 |
rather than the harddisks. CPU utilisation is down to about 10%, indicating |
| 53 |
that something else than encryption is throttling the transfer. This is odd! |
| 54 |
|
| 55 |
Case 4: |
| 56 |
I log into B and try to NFS mount A's /tmp to B's /mnt/floppy. It returns with |
| 57 |
an RPC timeout. So I can't do the "cp" test from B. |
| 58 |
|
| 59 |
Case 5: |
| 60 |
I log into B and sftp into A. It sits there for about 10 seconds before |
| 61 |
presenting me with a password prompt. ???? After, I get transfer rates close |
| 62 |
to case 2 and case 3, just the other way round. |
| 63 |
|
| 64 |
I am puzzled. First I thought that the Gb NIC on box A is somehow kaput but |
| 65 |
case 1 surely shows it is performing. What the heck is going on here? I would |
| 66 |
be deeply indebted to any person on this list that could shed some light on |
| 67 |
this. Any hint what to investigate would be highly appreciated. Really. This |
| 68 |
has troubled me for the last three days and I would go as far as ship you a |
| 69 |
Windhoek Lager. ;-) |
| 70 |
|
| 71 |
Uwe |
| 72 |
|
| 73 |
-- |
| 74 |
95% of all programmers rate themselves among the top 5% of all software |
| 75 |
developers. - Linus Torvalds |
| 76 |
|
| 77 |
http://www.uwix.iway.na (last updated: 20.06.2004) |
| 78 |
-- |
| 79 |
gentoo-user@g.o mailing list |
Archives