Low FTP transfer rate when enabling Jumbo frames.
Bookmark this page:
 Yahoo!
 Google
 Windows Live
 del.icio.us
 digg
 Netscape
|
|
|||||||||||||
|
Posted by fj on July 14, 2006, 5:46 am
Please log in for more thread options Hi I have a problem with FTP and Jumbo frames, and would like to hear if anybody else has experienced the same problem. The problem is that when I enables jumbo frames support the ftp transfer rate becomes very low compared with non-jumbo frames ftp transferring. The actually numbers are : Non-jumbo frames ftp transfer rate : 16.5 MB/s. Jumbo frames ftp transfer rate : 30 kB/s. When I do a simple drag-and-drop of a file between the 2 computers the transfer time is the same regardless of enabling jumbo frames or not. The problem isn't a switch problem, since I see the same thing using a "crossed cable". Is there anybody there have seen the same, or is there a configuration I have forgot to setup ?. Regards Flemming P.S. My operating system is windows 2000, and I am using the "build-in" ftp server. The FTP client is "core FTP". | |||||||||||||
|
Posted by Rick Jones on July 14, 2006, 1:46 pm
Please log in for more thread options > Non-jumbo frames ftp transfer rate : 16.5 MB/s.
> Jumbo frames ftp transfer rate : 30 kB/s. > When I do a simple drag-and-drop of a file between the 2 computers the
> transfer time is the same regardless of enabling jumbo frames or not. > The problem isn't a switch problem, since I see the same thing using > a "crossed cable". > Is there anybody there have seen the same, or is there a configuration
> I have forgot to setup ?. Some thoughs: *) does _everything_ between the two systems (inclusive) support JumboFrame? *) what is the TCP window size being used for the FTP data connection (you may need to do some packet tracing to find that) *) are there any TCP retransmissions (however one sees that in windows) *) IIRC there are times when Windows TCP will decide to do "direct I/O" (as it were) and hold-onto the user's buffer for DMA. Depending on the call the user made that may block the user until the data is ACKed by the remote, effectively turning the sliding window of TCP into stop-start. rick jones -- No need to believe in either side, or any side. There is no cause. There's only yourself. The belief is in your own precision. - Jobert these opinions are mine, all mine; HP might not want them anyway... :) feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH... | |||||||||||||
|
Posted by fj on July 17, 2006, 11:08 am
Please log in for more thread options
Hi Rick Thanks for the feedback. *) Yes, all the equipment supports jumbo frames. *) The TCP windows size is 64K. *) I am not quite sure about what you are saying about "user's buffer for DMA". Could you elaborate on that? Another thing I have discovered is that I do not have any problem with TCP traffic when the MTU size is 4088 bytes, and I do not have any problems with UDP traffic ( MTU = 9014 bytes ). I have tried to capture the traffic with ethereal, and there I can see that it takes 0.2 Sec to get an acknowledge from the receiving part when MTU=9014 bytes. Traffic with MTU=1518 bytes gives acknowledge within a few msec, so I expect that this is my problem, but don't know what to do about it. /Flemming Jahn Rick Jones skrev: > > I have a problem with FTP and Jumbo frames, and would like to hear
> > if anybody else has experienced the same problem. The problem is > > that when I enables jumbo frames support the ftp transfer rate > > becomes very low compared with non-jumbo frames ftp > > transferring. The actually numbers are : >
> > Non-jumbo frames ftp transfer rate : 16.5 MB/s.
> > Jumbo frames ftp transfer rate : 30 kB/s. >
> > When I do a simple drag-and-drop of a file between the 2 computers the
> > transfer time is the same regardless of enabling jumbo frames or not. > > The problem isn't a switch problem, since I see the same thing using > > a "crossed cable". >
> > Is there anybody there have seen the same, or is there a configuration
> > I have forgot to setup ?. >
> Some thoughs: > > *) does _everything_ between the two systems (inclusive) support > JumboFrame? > > *) what is the TCP window size being used for the FTP data connection > (you may need to do some packet tracing to find that) > > *) are there any TCP retransmissions (however one sees that in windows) > > *) IIRC there are times when Windows TCP will decide to do "direct > I/O" (as it were) and hold-onto the user's buffer for DMA. Depending > on the call the user made that may block the user until the data is > ACKed by the remote, effectively turning the sliding window of TCP > into stop-start. > > rick jones > > -- > No need to believe in either side, or any side. There is no cause. > There's only yourself. The belief is in your own precision. - Jobert > these opinions are mine, all mine; HP might not want them anyway... :) > feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH... | |||||||||||||
|
Posted by Rick Jones on July 17, 2006, 12:47 pm
Please log in for more thread options
> *) Yes, all the equipment supports jumbo frames.
> *) The TCP windows size is 64K.
> *) I am not quite sure about what you are saying about "user's buffer
> for DMA". Could you elaborate on that? "Normally" when an application calls send, the kernel/stack makes a copy of the user's buffer, and then schedules DMA from that copy. The user is then free to keep running, doing whatever it wishes with his buffer. However, sometimes the MS TCP stack decides it does not want to copy. Instead it will simply reference the user's buffer directly. Since allowing the user to manipulat that buffer while the stack is sending from it would lead to doubplusungoodness like data corruption, the user is precluded from accessing the buffer. In the case of which I'm thinking the user application is blocked from running entirely until the data in that buffer has been ACKed by the remote. So, you lose parallelism between the user and the stack/remote and things go slowly. > Another thing I have discovered is that I do not have any problem
> with TCP traffic when the MTU size is 4088 bytes, and I do not have > any problems with UDP traffic ( MTU = 9014 bytes ). While "Jumbo Frame" is not standardized, (I've seen all sorts of examples of "Jumbo Frame" support asserted for what I wouldn't call "real" Jumbo Frames), I thought the JF IP MTU was 9000 bytes, not 9014. I've heard of stacks calling 2048 bytes JF or 8192 but this would be the first time I've seen 9014. > I have tried to capture the traffic with ethereal, and there I can see
> that it takes 0.2 Sec to get an acknowledge from the receiving part > when MTU=9014 bytes. Traffic with MTU=1518 bytes gives acknowledge > within a few msec, so I expect that this is my problem, but don't > know what to do about it. 0.2 seconds sounds like waiting for the standalone ACK timer. Are you certain that the receiving TCP is advertising a 64KB window? And that the sending TCP is sending multiple MSS segments at a time, keeping that window full? rick jones -- Process shall set you free from the need for rational thought. these opinions are mine, all mine; HP might not want them anyway... :) feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH... | |||||||||||||
|
Posted by fj on July 18, 2006, 7:34 am
Please log in for more thread options
Hi Rick My NIC is an Intel Pro 1000, and the possible setting for Jumbo frames is 9014 and 4088 bytes. I have used Ethereal to determine the TCP window size, and it says that the TCP windows size is 64K. I have even tried to increase the TCP window size to 128K and that doesn't change anything. That said, you did point me in the right direction, and I found description of the 200 ms problem at Microsoft ( http://support.microsoft.com/kb/321098/ ). It turned out that it is possible to disabled the delayed standalone ACK timer. When turned off, the transfer rate increases to approx. 16 Mbytes/s. Thanks for your help Flemming Jahn Rick Jones skrev: > > *) Yes, all the equipment supports jumbo frames.
>
> > *) The TCP windows size is 64K.
>
> > *) I am not quite sure about what you are saying about "user's buffer
> > for DMA". Could you elaborate on that? >
> "Normally" when an application calls send, the kernel/stack makes a > copy of the user's buffer, and then schedules DMA from that copy. The > user is then free to keep running, doing whatever it wishes with his > buffer. > > However, sometimes the MS TCP stack decides it does not want to copy. > Instead it will simply reference the user's buffer directly. Since > allowing the user to manipulat that buffer while the stack is sending > from it would lead to doubplusungoodness like data corruption, the > user is precluded from accessing the buffer. In the case of which I'm > thinking the user application is blocked from running entirely until > the data in that buffer has been ACKed by the remote. So, you lose > parallelism between the user and the stack/remote and things go > slowly. > > > Another thing I have discovered is that I do not have any problem
> > with TCP traffic when the MTU size is 4088 bytes, and I do not have > > any problems with UDP traffic ( MTU = 9014 bytes ). >
> While "Jumbo Frame" is not standardized, (I've seen all sorts of > examples of "Jumbo Frame" support asserted for what I wouldn't call > "real" Jumbo Frames), I thought the JF IP MTU was 9000 bytes, not > 9014. I've heard of stacks calling 2048 bytes JF or 8192 but this > would be the first time I've seen 9014. > > > I have tried to capture the traffic with ethereal, and there I can see
> > that it takes 0.2 Sec to get an acknowledge from the receiving part > > when MTU=9014 bytes. Traffic with MTU=1518 bytes gives acknowledge > > within a few msec, so I expect that this is my problem, but don't > > know what to do about it. >
> 0.2 seconds sounds like waiting for the standalone ACK timer. Are you > certain that the receiving TCP is advertising a 64KB window? And that > the sending TCP is sending multiple MSS segments at a time, keeping > that window full? > > rick jones > -- > Process shall set you free from the need for rational thought. > these opinions are mine, all mine; HP might not want them anyway... :) > feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH... | |||||||||||||
> if anybody else has experienced the same problem. The problem is
> that when I enables jumbo frames support the ftp transfer rate
> becomes very low compared with non-jumbo frames ftp
> transferring. The actually numbers are :