Speed Mismatch?!?

mschunk · 2006-05-07T23:43:21+00:00

I've been banging my head on this one for weeks.Can someone help me understand why network performance between Windows2k/xp/2k3 hosts with a gigabit interface, and those with a 100/fullinterface are slow so?Now, 100/full to 100/full transfers are plenty fast. Very. And 1000to 1000 transfers outright smoke...as gigabit should.But, for large TCP transfers from a 1gbit host to a 100mbit host itgoes very Slow...1) No matter what NIC used. Intel, Broadcom, D-Link, 3COM....or driverversion.2) No matter what version of windows to other version of windows...ordifferent motherboards, etc...the slowness is very predictable nomatter what hardware involved.3) No matter what type of gigabit switch....Cisco 3750, severalNetgears, some managed others not.4) No matter how the switches involved are used...host and server insame switch, slow. Host and server in different switches...linkedstraight...routed between them...I even tried trunks between switchestogether w/ link ag....always slow. Always the same.5) No matter how many times I try different Ethernet cables...Cat5, 5e,6...some pre-made, others made by hand.6) No matter what TCP window sizes are set to anywhere, be it client orserver side or both.I put gbit adapters in the servers, thinking they could use them tosever files to all the workstations that have 100mbit NIC's. I mean,it was logical...a gbit server "should" be able to handle 10x thethroughput to 100mbit clients, right???? Nope.Note again that the performance problem is only one way. When I pushfiles FROM the 100mbit hosts, to the 1gbit servers, speed is expected:fast, about 80mbit/sec. When I pull files form the 1bitg servers, tothe 100mbit PC's...speed is absolutely terrible. I'm taking5mbit/sec, if that.Looking at the packets...I'm seeing gobs and gobs of TCP retransmitframes. It's like the switches (even the fancy Cisco one) aresending data to the PC's faster than the PC's can take it....butonly when moving data from 1gbit to 100mbit. (So much for...

M

mschunk 20 years ago

TAC has been "...[looking] at the captures..." for the last day+.

No word yet.

This _is_ a clue. It's buried above, but the reason it took me a year and a half to realise that somthing was horribly wrong, not just slow servers etc - was because I have an extra switch in my path compared to the rest of the users. I can get 30mbit/sec to Gi. Still crapy and asym....80 going to Gi, 30 coming from.

I have not tried exactly what you have above, I did this:

source-gi-->buffer-sw-Gi-->Stack-->Fa-sink.

IOW, I did not try buffering w/ a switch that had an Fa link to the stack...I stayed on Gi.

I also took this a little further and setup an 4x etherchannel, still all gi...I _WAS_ trying to buffer somthing somwhere, not really knowing how but wondering if packet A overran packet B to the point that the stack could not see the etherframe signature of A anymore...reports no drop becuase it saw no packet in the first place. I know there is a drop somwhere, where is this drop occuring? Packet drop counters increment if and only if IOS is aware that a drop has occured...I doubt interface counters care what goes on between the guts that contect other interfaces. Stack ring counters? Still don't know how to get any.

Anyway, I have time right now to try what you suggest.

< time passes >

It works at 65 mbit/sec.

Gi-source-->buffer-sw-Fa-->stack-->Fa-sink

iperf...

Recall that I'm unable to conect from fa-client to gi-server to run the test. I get "connection failed" If I fail to lanuch "ipsef -s" on Gi, I get "Connection refused" I have the same problem with pcattcp. Thing is client-fa-->server-Gi is not the slow path. It's slower that it should be using robocoy.

Gi-server-->stack-->Gi-sink

...kicks so much ass. Very fast.

I got no clue about the inerds of a 3750...is there supposed to be a buffer/queue dedicated for moving data between fa and gi?!?!?!?

I'm been thinkg about the odds of a gi-host overruning an fa-host. I don't see it. TCP does send some segments out...but do not most TCP stacks "wait" at some point for some ACKs beofre sending more data? In UDP, I could see Gi sending so much that packets just drop in the switch once egrees queues filled up. Or does it not work that way? I'm just making assumption based on prior data; not knowing how much value said prior data/experiance has in this case.

Vote

A

anybody43 20 years ago

I have just remembered something. GI Ethernet has a feature called "Pause Frames"

############ # Begin guesswork ############ These are designed to prevent or reduce the number of 'overrun frames' (my terminology) sent. Some network protocols can send wire rate bursts of frames that can overflow the buffers in intermediate or in end stations. Such overflow packets are lost.

Pause frames are sent by the station that is experiencing the overflowing buffers towards the source and in some way signal to the source to slow down a bit.

############ # End guesswork ############

I have no real idea if you network _needs_ pause frames to be operational to function (I would strongly doubt it) however what I was wondering was if you did in fact have partially operational pause frames and maybe there might be a bug in the way that the pause frames were operating.

If this model is correct you will NOT be seeing lost packets on the network.

I know nothing about pause frames. They could for example not show up on a sniffer 'cos they could quite reasonably be implemented at Layer 1 and a cheap and cheerful sniffer could quite reasonably not be able to see them.

You could quite reasonably post this issue on comp.dcom.lans.ethernet. Leave in the cisco references. Someone there who is familiar with the pause frame implementation for example may just see the issue right away.

Out of interest, you aren't using huge TCP windows are you? i.e. you haven't got TCP Window scaling turned on? Consider all of your hosts.

If you were getting buffer overflows you should I would imaging be seeing output drops on the Fa interfaces. sh int | inc drops If there are none then maybe the counters are broken or maybe the drops are internal to the switch or maybe there is a 'pause frame bug'.

Vote

M

mschunk 20 years ago

I thought about Pause frames too.

I'm certain I now even less about them than yourself.

I do know that I have all flow-control turned off on the stack, as is the default. Useing braodcoms diag tools for thier NIC's...During link negotiation, the stack is saying "I don't do flow control" and the NIC, who has flow control set to auto, noted this.

I'm watiing for TAC to say _somthing_ but at this point they have not responded to what I set or asked any more questions. I will bring up flow control with them.

I looked into flow control because, befor I realized that _ALL_ my GigE NIC's are affected, I was thinking "bad driver" and the RTM driver for the broadcom NIC's in question did, arrounding to broadcom, have a bu in them with reguards to flow control...the older driver would revice a pause frame and them pause for too long.

Vote

R

rdymek 20 years ago

What brand/models do you have this experience with? I've currently got two 6509's curreently deployed with Gig hard coded on both ends of the link (auto disabled). I've also got 2 2948G-48 switches deployed with gig hard coded on both ends of the link. We get a full 1gb throughput without a single error on the port.

Ryan

Vote

R

rdymek 20 years ago

Personally, my experiences have been just the opposite. Honestly, I can't quote the IEEE standard on this one as I've never looked it up and just assumed that you could set it or put it on auto just like

100mb.

I've currently got 6500's using the WS-X6148-GE-TX copper 48 port blades, as well as 2948G's that are hard coded to gig. Both on the server and the switch.

I'm not saying its not against IEEE, but If it were against IEEE standards, then why would the server as well as the switch allow this?

Ryan

Vote