Low FTP transfer rate when enabling Jumbo frames.

Some thoughs:

*) does _everything_ between the two systems (inclusive) support JumboFrame?

*) what is the TCP window size being used for the FTP data connection (you may need to do some packet tracing to find that)

*) are there any TCP retransmissions (however one sees that in windows)

*) IIRC there are times when Windows TCP will decide to do "direct I/O" (as it were) and hold-onto the user's buffer for DMA. Depending on the call the user made that may block the user until the data is ACKed by the remote, effectively turning the sliding window of TCP into stop-start.

rick jones

Hi Rick

Thanks for the feedback.

*) Yes, all the equipment supports jumbo frames.

*) The TCP windows size is 64K.

*) I am not quite sure about what you are saying about "user's buffer for DMA". Could you elaborate on that?

Another thing I have discovered is that I do not have any problem with TCP traffic when the MTU size is 4088 bytes, and I do not have any problems with UDP traffic ( MTU = 9014 bytes ).

I have tried to capture the traffic with ethereal, and there I can see that it takes 0.2 Sec to get an acknowledge from the receiving part when MTU=9014 bytes. Traffic with MTU=1518 bytes gives acknowledge within a few msec, so I expect that this is my problem, but don't know what to do about it.

/Flemming Jahn

Rick Jones skrev:

"Normally" when an application calls send, the kernel/stack makes a copy of the user's buffer, and then schedules DMA from that copy. The user is then free to keep running, doing whatever it wishes with his buffer.

However, sometimes the MS TCP stack decides it does not want to copy. Instead it will simply reference the user's buffer directly. Since allowing the user to manipulat that buffer while the stack is sending from it would lead to doubplusungoodness like data corruption, the user is precluded from accessing the buffer. In the case of which I'm thinking the user application is blocked from running entirely until the data in that buffer has been ACKed by the remote. So, you lose parallelism between the user and the stack/remote and things go slowly.

While "Jumbo Frame" is not standardized, (I've seen all sorts of examples of "Jumbo Frame" support asserted for what I wouldn't call "real" Jumbo Frames), I thought the JF IP MTU was 9000 bytes, not

9014. I've heard of stacks calling 2048 bytes JF or 8192 but this would be the first time I've seen 9014. 0.2 seconds sounds like waiting for the standalone ACK timer. Are you certain that the receiving TCP is advertising a 64KB window? And that the sending TCP is sending multiple MSS segments at a time, keeping that window full?

rick jones

Hi Rick

My NIC is an Intel Pro 1000, and the possible setting for Jumbo frames is 9014 and 4088 bytes.

I have used Ethereal to determine the TCP window size, and it says that the TCP windows size is 64K. I have even tried to increase the TCP window size to 128K and that doesn't change anything.

That said, you did point me in the right direction, and I found description of the 200 ms problem at Microsoft (

). It turned out that it is possible to disabled the delayed standalone ACK timer. When turned off, the transfer rate increases to approx. 16 Mbytes/s.

Thanks for your help

Flemming Jahn

Rick Jones skrev:

Are they presented as "MTU's" or "frame sizes?" 9014 makes "sense" as a JumboFrame Frame size as it would result in an IP MTU of 9000 bytes which IMO is the de facto "standard" for JumboFrame, going back to the first Alteon GbE NICs. I don't consider smaller values to be "real" Jumbo Frames :)

I may not have grokked that URL fully, but it sounds like a kludge. IMO often as not, shortening or disabling the standalone ACK timer is a kludge.

Sounds like something was stuck waiting for the standalone ACK timer to fire on the receiver, so the sender could send more, which then begs the question - why? I wonder if the larger MTU is exposing a situation where one side is trying to send "logically associated data" in more than one call to send() and those sends, while larger than

1460 bytes are not larger than 8960.

My pleasure.

rick jones