Speed Mismatch?!?

Oh,it is very interesting. I will try it. Do you affirm that NIC was full duplex?

Yes. In fact I played around with duplex settings quite a bit. I'm very certain that I've got 100/full on the 100mib side, and 1000/full (can't do 1/2 on gbit) on the gibt side.

On the gig interfaces on the servers, do you have them set to auto or

1000/full? You can set 1000/full, but it does NOT work. Gig ports MUST be set to auto ON BOTH SIDES on gigabit.

Oh, really? i did't suppose it...... it must AUTO..... "thrill5" ?ÈëÏû?ÐÂÎÅ : snipped-for-privacy@comcast.com...

Thank you for the "duplex" warnings.

I'm confidant that duplex issuess are not involved. I left out gobs of information about different duplex settings I have tried.

And yes, we all know that the _only_ way to get 1000mbit/sec is in "Auto" as IEEE standards require "auto" as the _only_ way to get

1000/full.

For completness, I did test transfers between 1gbit and 100/half. Results: From 100/half to 1gbit = 60MBit/sec. From 1gbit to 100/half was so bad I canceled the test.

I'm aware that all 100mbit devices on a fabric must operate at the same duplex...right down to every last devices, printers too, or nothing works well.

My ehternet fabric concisits of either 1) 100/full, or 2) 1000/full (auto.)

Recap: 100/full --> 1000/full at 80mbtis/sec. 1000/full --> 100/full = 5mbit/sec.

Can anybody out there confirm, that just as there is a huge perforamce problem going between 100mbit and (older) 10mbit hosts...that the same really does apply to gbit and 100mbit?

Most of our servers are connected via gig, and all of the workstations are

100/full. I have not seen the problem you are describing except when there is a duplex mismatch. Have you made sure you don't have a duplex mismatch with other devices that are in the traffic path? You could have an uplink or router port with a duplex issue. I have run into problems with NIC drivers, but this was a few years ago. I know there were issues with drivers on Sun Gig boards, 3Com 3C590 embedded, and Broadcom 10/100 drivers on System V.

The one sure way to diagnose you problem is to sniff the traffic as leaves the server and at the same time sniff where it enters the client. This will tell you if packets are disappearing and at which end. You can keep moving the sniffers to sniff the paths in-between to find out where the packets are being dropped.

I'm 99% sure you have a duplex mismatch problem. I know you felt this wasn't the case but everything you have described points to this.

Here are the top things that will happen with duplex mismatch:

1. Collisions on a port shat should be full duplex (a properly operating port at full duplex should never have a collision).
2. Problems (errors and performance issues) more in one direction than the other (duplex problems are asymmetrical).
3. Retransmissions (these are due to the collisions and errors on the port

Don't forget that you need the switch to be set the same as the host. You cannot have auto on one side and it set on the other.

Turn on debugging on your switch - chances are you are getting a ton of duplex mismatch errors.

If the duplex is fine then the only other way you can get collisions is due to a bad nic, bad switch port, bad cable or too long of a cable.

Otherwise, all your TCP connections are throttled at the TCP layer from host to host. The switch mearly sends what its sent. If the Gig server is overrunning the 100mb port, it wouldn't be the switch that's the problem, but rather the Gig server itself. The switch does not do any throttling.

Store and forward is simply a switching method - not a buffering mechanism. Its to provide 100% CRC reliability, but its also least efficient. It used to be in older days that you needed this reliability. With today's Ethernet standards, this is not necessary and cut-through tends to provide the same level of reliability, while providing much faster performance. But end result is that store-and-forward has absolutely nothing to do with "buffering" packets to help from higher speed links to lower speed ones.

Ryan

I find that in some heavy traffic conditions auto negotiation can flap and flow control on gigabit links can throttle connections.

Now for server a gigabit over cooper when using multiple speed devices, it is recommended to set at auto, however I set all switch and server to

1000/full anyway and seem to get better reliability.

I always set switch-switch links hard set at 1000/full and no auto negotiation and not flow control regardless.

I have issues with flow control on switch-switch links causing traffic to halt intermittently, however never seem that issue with host connections at gigabit. Flow control may be needed on host based systems as may not be able to handle the entire bandwidth, most host system bus can not handle full throttle gigabit traffic.

Thanks Rdymek, I know it smells like a duplex problem.

sh interfaces and eye-balling NIC config....no, it's not. I've got a good variety of switches, but have been using mostly just the cisco ones for working through this....many different nics of both kinds and PC's w/ different Windows versions (2k/xp/2k3) on them to play with.

These Speed results are consistent...from Gi --> Fa, throughput really sucks.

The collisions are not consistent. Today, I tried things out on a 2801 that's not needed for the moment. No collisions detected today. Not one. No CRC errors, Just really bad performance between NIC's at same duplex but different speeds...What I find odd, is that the switch reports NO packet drops on either side. "sh interface "

I can't get good debug events going. Packets have to switch by the processor for the debugger to catch them, right? Anyway....no duplex mismatch on interface state change.

I have not tried sniffing both sides at the same time. Thanks for setting me straight on the "store and forward." I'll try "buffering" too...I have only one router, a 3825, that has both gi and fa interfaces on it. I'll stick it between two switches and see what happens. I won't be able to play with it until Sunday, during off-hours. If this works though....gbit overrunning the 100mbit host....how would I stop that even if true? It's my worst fear, here...if this works it confirms that I really do need to put 1000 and

100 hosts on separate L2 fabrics.

...Till Sunday

What switch do you have that allows you to manually set "1000/full" It's aginst IEEE standards and I have no NIC or switch in the building that lets me.

If you have one, please give us the model number?

Please post show version from each Cisco switch in the traffic path

Interesting issue, but it does make some sense.. How about get one of thos DOS client/server Bandwidth tools that use UDP (not TCP). Check the ports utilisation on CLI, and you will see nothing wrong with the Switch/Duplex settings. Think about how TCP slow start works - one dropped packet resets the TCP window.. then ramps up again exponentially until another dropped packet. What is happening is that from 1000 -> 100 port packets gets dropped in the fabric - whilst I bet the 100 -> 1000 port, the packets gets buffered which introduces delay however the overall throughput is better. Simply, having a server on a gigport should not be servicing just one client. the idea is that you have at least 10 x 100mb ports/clients using the server on GIG. Even smarter if you have a linux server to cap to

1. TCP Answer no.

I have here a PC on 100Mpbs and a couple of hops away a server on 1Gbps.

The intermediate hops are all GBE or maybe a 4x GBE Etherchannel.

################## ## 1G bps Server end ################## C:\\Documents and Settings\\admin>iperf -s

------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 8.00 KByte (default)

------------------------------------------------------------ [1896] local 192.168.5.33 port 5001 connected with 192.168.67.249 port

2172 [ ID] Interval Transfer Bandwidth [1896] 0.0-10.0 sec 112 MBytes 94.2 Mbits/sec

C:\\Documents and Settings\\admin>iperf -c 192.168.67.249

------------------------------------------------------------ Client connecting to 192.168.67.249, TCP port 5001 TCP window size: 8.00 KByte (default)

------------------------------------------------------------ [1908] local 192.168.5.33 port 2404 connected with 192.168.67.249 port

5001 [ ID] Interval Transfer Bandwidth [1908] 0.0-10.0 sec 110 MBytes 92.0 Mbits/sec

################# ## 100Mbps PC end. ################# M:\\Documents and Settings\\admin>iperf -c 192.168.5.33

------------------------------------------------------------ Client connecting to 192.168.5.33, TCP port 5001 TCP window size: 63.0 KByte (default)

------------------------------------------------------------ [1704] local 192.168.67.249 port 2172 connected with 192.168.5.33 port

5001 [ ID] Interval Transfer Bandwidth [1704] 0.0-10.0 sec 112 MBytes 94.1 Mbits/sec

M:\\Documents and Settings\\admin>iperf -s

------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 8.00 KByte (default)

------------------------------------------------------------ [1668] local 192.168.67.249 port 5001 connected with 192.168.5.33 port

2404 [ ID] Interval Transfer Bandwidth [1668] 0.0-10.0 sec 110 MBytes 92.1 Mbits/sec

This works in this case. No tuning, no optimisation just perfect performnce right out of the box.

If it didn't work we would be in BIG trouble. All of our servers are on GBE and a lot of the clients are on FE.

Millions and millions of people are using just such a setup as you describe.

OK, yours isn't working.

Maybe this is the clue?

"Late collisions" are your problem, or at least one of them.

They are caused by two things in modern switched networks. Firstly collisions can only occur on Half Duplex ports.

Late collisions are a consequence of a duplex missmatch. They occur on the HD end since the FD end just transmitts when it feels like it and sometimes there is an out of window collision.

I have found (a while back) that WIndows drivers can report duplex settings incorrectly. In dicfficult cases I trust ONLY the port counters on Cisco networking kit.

Use the counters to determine if you have a duplex missmatch.

If there is a missmatch AND there is enough traffic (let's say) there WILL BE late collisions on the HD end.

The FD end will be seeing CRC and Align errors.

So, very carefully determine what to do to emiminate the duplex missmatch and then roll it out onto the various kit.

Pick a couple of ports on one switch set up 1000 on one and 100 on another.

Clear the counters

run some traffic if the performance is poor send

sh run int fa xx sh run int gi xx

sh int fa xx sh int gi xx

If you download iperf and use it to create traffic in both directions simultaneously you will for sure show up any problems.

For some reason when I do both directions simultaneously I get only 90Mbps one way and 25Mbps he other way. I don't know why but I think that if it was a duplex missmatch it would be worse.

Changing the iperf "buffer size" from the default 8k to 32k fixed it.

85Mbps each way now.

I expect that this is the amount of data sent before waiting on an acknowledgement.

With 500k I get 90Mbps each way. In this acse we will be running into the TCP Rx window which seems to be 64k. No scaling.

Thank you everybody for your support, how about I show some numbers? You will soon see that there is no duplex issue being reported by the switches.

Both client and server in this test running Win2k3, SP1...I don't have a single linux box in the building :(

Client = Nivida nForce2 on-board 10/100 Server = Nivida nForce4 on-board 10/100/1000

I'll run tests aginst server twice:

1) both NIC and switchport manually set at 100/full 2) both NIC and switchport set at "auto"

Although I'ved gotten the same speed results on many different switches, I'll run this today

on the production stack of 3750's.

I'll use iperf, and run a set of tests in TCP, and another in UDP.

------------------------- Stack Info

-------------------------

3750Stack# sh ver

Cisco Internetwork Operating System Software IOS (tm) C3750 Software (C3750-I5-M), Version 12.1(19)EA1d, RELEASE SOFTWARE (fc1) Copyright (c) 1986-2004 by cisco Systems, Inc. Compiled Mon 05-Apr-04 22:06 by antonino Image text-base: 0x00003000, data-base: 0x009206D8

ROM: Bootstrap program is C3750 boot loader BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.1(14r)EA1a, RELEASE SOFTWARE (fc1)

(...)

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

• 1 28 WS-C3750G-24TS 12.1(19)EA1d C3750-I5-M 2 52 WS-C3750-48P 12.1(19)EA1d C3750-I9-M 3 52 WS-C3750-48P 12.1(19)EA1d C3750-I9-M 4 52 WS-C3750-48P 12.1(19)EA1d C3750-I9-M 5 52 WS-C3750-48P 12.1(19)EA1d C3750-I9-M 6 52 WS-C3750-48P 12.1(19)EA1d C3750-I5-M

(...)

3750Stack# sh run

(...) ! ip subnet-zero ip routing ! ip host vg 192.168.8.3 ip host rld-1760 192.168.8.2 ip host rld-3725 192.168.8.1 ip name-server 192.168.1.5 mls qos map cos-dscp 0 8 16 26 34 46 48 56 mls qos map ip-prec-dscp 0 8 16 26 34 46 48 56 mls qos ! ! spanning-tree mode pvst spanning-tree loopguard default no spanning-tree optimize bpdu transmission spanning-tree extend system-id ! (...) ! interface GigabitEthernet1/0/24 description ------------ SERVER PORT ----- switchport access vlan 105 switchport mode access no ip address no mdix auto storm-control broadcast level 2.00 storm-control multicast level 2.00 ! (...) ! interface FastEthernet5/0/24 description ------------ CLIENT PORT ----- switchport access vlan 100 switchport mode access switchport voice vlan 200 no ip address no mdix auto storm-control broadcast level 2.00 storm-control multicast level 2.00 spanning-tree portfast spanning-tree bpduguard enable ! (...) ! interface Vlan1 no ip address shutdown ! interface Vlan100 ip address 192.168.0.1 255.255.248.0 ip helper-address 192.168.1.5 ! interface Vlan105 ip address 10.30.5.1 255.255.255.0 ip helper-address 192.168.1.5 ! interface Vlan200 ip address 192.168.8.4 255.255.248.0 ip helper-address 192.168.1.5 ! (...)

----------------- Stack Notes

-----------------

There are VoIP phones in use. The stack does inter-vlan rounting:

3750Stack# sh ip route

(...) C 10.30.5.0 is directly connected, Vlan105 S* 0.0.0.0/0 [1/0] via 192.168.0.2 C 192.168.8.0/21 is directly connected, Vlan200 C 192.168.0.0/21 is directly connected, Vlan100 (...)

Client IP address = 192.168.1.6 Server IP Address = 10.30.5.5

------------------ iperf, 100/full 100/full

------------------

The results of this test will surprize no one:

C:\\>iperf -f m -B 192.168.1.6 -c 10.30.5.5 -r

------------------------------------------------------------ Server listening on TCP port 5001 Binding to local address 192.168.1.6 TCP window size: 0.01 MByte (default)

------------------------------------------------------------

------------------------------------------------------------ Client connecting to 10.30.5.5, TCP port 5001 Binding to local address 192.168.1.6 TCP window size: 0.01 MByte (default)

------------------------------------------------------------ [1192] local 192.168.1.6 port 3523 connected with 10.30.5.5 port 5001 [ ID] Interval Transfer Bandwidth [1192] 0.0-10.0 sec 96.8 MBytes 81.1 Mbits/sec [1948] local 192.168.1.6 port 5001 connected with 10.30.5.5 port 6025 [ ID] Interval Transfer Bandwidth [1948] 0.0-10.0 sec 98.2 MBytes 82.4 Mbits/sec

See, TCP is about 80Mbit/sec, and symetrical Looks good. Let's try UDP:

C:>iperf -f m -B 192.168.1.6 -c 10.30.5.5 -r -u

------------------------------------------------------------ Server listening on UDP port 5001 Binding to local address 192.168.1.6 Receiving 1470 byte datagrams UDP buffer size: 0.01 MByte (default)

------------------------------------------------------------

------------------------------------------------------------ Client connecting to 10.30.5.5, UDP port 5001 Binding to local address 192.168.1.6 Sending 1470 byte datagrams UDP buffer size: 0.01 MByte (default)

------------------------------------------------------------ [1192] local 192.168.1.6 port 3601 connected with 10.30.5.5 port 5001 [ ID] Interval Transfer Bandwidth [1192] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec [1192] Server Report: [1192] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 0.000 ms 0/ 893 (0%) [1192] Sent 893 datagrams [1244] local 192.168.1.6 port 5001 connected with 10.30.5.5 port 6258 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [1244] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 0.977 ms 0/ 894 (0%)

Whoa! Only 1.05Mbit/sec for UDP? (This is still 100/full

100/full) Maybe I just don't know enough about UDP and/or iperf? (First time I'ved used iperf was today, up till now I had been using robocopy.)

----------------------------- Let's now do the (server) GigabitEthernet (client) FastEthernet tests.

-------------------------

But first, clear switch counter.

3750Stack# clear counters gi1/0/24 3750Stack# clear counters fa5/0/24

...uh-oh. running iperf as client from "client" results in connection timeout. (not consection refused, as happens when I forget to start iperf as server)

...ok, I'll do iperf -s on the client instead...we are only goint to see the "slow" path.

C:\\>iperf -f m -c 192.168.1.6

------------------------------------------------------------ Client connecting to 192.168.1.6, TCP port 5001 TCP window size: 0.01 MByte (default)

------------------------------------------------------------ [1908] local 10.30.5.5 port 4608 connected with 192.168.1.6 port 5001 [ ID] Interval Transfer Bandwidth [1908] 0.0-10.1 sec 1.74 MBytes 1.45 Mbits/sec

C:\\>

Yup, slow as ethernet over carrier pigeon. Now, lets see some interface counters:

3750Stack#sh interfaces gi1/0/24 GigabitEthernet1/0/24 is up, line protocol is up (connected) Hardware is Gigabit Ethernet, address is 0011.bb99.2598 (bia 0011.bb99.2598) MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 1000Mb/s, media type is RJ45 output flow-control is off, input flow-control is off ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output 00:00:01, output hang never Last clearing of "show interface" counters 00:05:31 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 2126 packets input, 2261304 bytes, 0 no buffer Received 11 broadcasts (0 multicast) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 0 multicast, 0 pause input 0 input packets with dribble condition detected 1690 packets output, 300868 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out 3750Stack# 3750Stack#sh interfaces fa5/0/24 FastEthernet5/0/24 is up, line protocol is up (connected) Hardware is Fast Ethernet, address is 0011.20a7.6b1a (bia 0011.20a7.6b1a) MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 100Mb/s input flow-control is off, output flow-control is off ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:21, output 00:00:57, output hang never Last clearing of "show interface" counters 00:17:29 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 1000 bits/sec, 1 packets/sec 5 minute output rate 48000 bits/sec, 14 packets/sec 3232 packets input, 437440 bytes, 0 no buffer Received 31 broadcasts (0 multicast) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 18 multicast, 0 pause input 0 input packets with dribble condition detected 13545 packets output, 7019546 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out

I'm no CCNA....but the above tells me there is no duplex issue. Or droped packets?!? In fact, I'm starting to think the collisions I had been seeing were generated by some other device. (I found out yesterday there are some old printers around that operate at 10/half.)

I don't understand WHY I can't get iperf to run in this, the test I wanted you all to see.

I'll skip the UDP test this time, you can already see how bad the situation is already with the server transmitting tcp to the client at a rate of 1.45Mbits/sec

I want to demo asymitry...shame iperf cound not connect both ways...Let me flip back to robocopy:

------------------------------------------------------------------------------- ROBOCOPY :: Robust File Copy for Windows :: Version XP010

-------------------------------------------------------------------------------

Started : Tue May 09 13:11:59 2006

Source : \\Foo Dest : \\Foo

Files : *.*

Options : *.* /S /E /COPY:DAT /MOVE /R:1000000 /W:30

------------------------------------------------------------------------------

New Dir 1 \\Foo

100% New File 32.6 m BigFile.mp3

------------------------------------------------------------------------------

Total Copied Skipped Mismatch FAILED Extras Dirs : 1 1 0 0 0 0 Files : 1 1 0 0 0 0 Bytes : 32.68 m 32.68 m 0 0 0 0 Times : 0:00:04 0:00:04 0:00:00 0:00:00

Speed : 6920528 Bytes/sec. Speed : 395.995 MegaBytes/min.

Ended : Tue May 09 13:12:04 2006

------------------------------------------------------------------------------- ROBOCOPY :: Robust File Copy for Windows :: Version XP010

-------------------------------------------------------------------------------

Started : Tue May 09 13:12:04 2006

Source : \\Foo Dest : \\Foo

Files : *.*

Options : *.* /S /E /COPY:DAT /MOVE /R:1000000 /W:30

------------------------------------------------------------------------------

New Dir 1 \\Foo

100% New File 32.6 m BigFile.mp3

------------------------------------------------------------------------------

Total Copied Skipped Mismatch FAILED Extras Dirs : 1 1 0 0 0 0 Files : 1 1 0 0 0 0 Bytes : 32.68 m 32.68 m 0 0 0 0 Times : 0:01:19 0:01:19 0:00:00 0:00:00

Speed : 433561 Bytes/sec. Speed : 24.808 MegaBytes/min.

Ended : Tue May 09 13:13:23 2006

So while iperf could not transfer both ways, windows file system could.

Client --> server = 395.995 MegaBytes/min. Server --> Client = 24.808 MegaBytes/min.

Note that when I do robocopy between 100/fll and 100/full...robocoy gets 591 MBytes/min both directions. So, even though the FastE -->

GigE doen't suck as much as GigE -> FastE.....FastE -> FastE is giving me the best performace.

Comments?

Merv, if you are a "couple hope away" than you are further away then I. Remember, in the above there are NO ROUTED HOPS between client and server, it's all switched at L2. and there is no "packet buffering" as was pointed out to me above. I'll still be placing a L3 router between the two on Sunday and see what happens. My goal here is aviod a seperate L2 fabric dedicated to gigabit....that's seams to be what you already have, no?

I think that the following description of iperf is OK but I am not 100% sure.

When using iperf TCP tests are very different from UDP tests.

TCP uses TCP (?) but does modify the behaviour slightly since by default iperf has a buffer size of 8k.

It sends 8k and then waits until all of the data has been acknowledged before sending another 8k. In your case on one switch (or stack) and at 100Mbps this is not significant and you saw ~100Mbps.

In the case of UDP there are no acknowledgements. It just blasts the data into the network at the configured rate. There is a default rate that is quite low and this acn be changed with commnd line parameters.

The observed behaviour is anomalous. Your network should not be behaving in this way.

You appear to have eliminated the duplex issue (if the counters are to be trusted).

As a next step I would like to see packet captures from both ends of the link.

You can download Ethereal (and of course winpcap) and install it on both ends. If you want to mount the resultant dump files on a server on the internet I could have look.

I would also suggest eliminating the "stack" from the equation by using ports on the same switch.

Can you get any stacking-link statistics?

On the 4500 when the backplane is oversubscribed (to keep it simple) you seem to get.

sh int Gi3/22 GigabitEthernet3/22 is up, line protocol is up (connected) ..... 109 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

sh int Gi3/22 cou det ..... Port Rx-No-Pkt-Buff RxPauseFrames TxPauseFrames PauseFramesDrop Gi3/22 109 0 0 0

I don't supose you have a spare switch? Ideally this should be reproduced off line.

Do you have Cisco support? If I was seeing this I would raise a TAC case.

You could get this problem if there were no buffers in the path between the 1000Mbps port and the 100Mbps port.

TCP sends (small-ish:-) bursts of packets at wire rate back to back and the switch is REQUIRED to buffer them.

Maybe for some reason yours is not doing this. This could be due to a bug, due to the buffer memory being use by something else, perhaps due to miss-configuration.

I see that you have storm control enabled, how about turning that off?

I notice that you software is a bit aged? You could check the release notes for likely bugs. You have - 12.1(19)EA1d --- 12.1.19-EA1 is "Deferred" which I think means that a serious problem was found with it.

I would consider changing the software. Note that in a substantial network that this is not something to be undertaken lightly.

Implementation plan Test plan Backout plan.

Good luck.

Thanks "anybod"

I feel somewhat vindicated in that you are able to look at this and say to yourself, "WTF"

I really appreciate your time; I won't waste it w/ packet dumps quite yet.

I do have TAC support...I prefer Usenet over these idiots but I degrees. I keep smartnet mostly for RMA's and Software updates. I knew this IOS was older, I'll look into this releases "caveats" and then upgrading it before going further.

As you say, you don't just do that on a dime. I don't think I'll have the switch up to date in less than three Sundays.

I'll post back in time, let you all know what happens.

Thanks again.

Hmmmm.

You do need to understand how to drive TAC but I find them pretty good. You, of course, have to jump through a few hoops and send a bit of info but if you have a clearly defined issue like this it should be quite easy to get the results that you want in quite a short period.

My impression is that Cisco really, really do want TAC to provide a good service.

A while back I was a network consultant with a gold partner and I got a lot of experience with TAC. The stuff we were sending then had already been picked over pretty thoroughly (there were 10 CCIEs in the office - not me) and it was very rare not to get at least a decent explanation of why we were being stupid:-)

The TAC is a key part of why I still recommend and use Cisco kit pretty much exclusively.

Since the software that you are using is deferred it will be difficult to resist an upgrade recommendation though. It may of course be deferred for a security reason or something and so they may let you keep it.

Back to the issue.

Maybe the switch really is full up with traffic?

Take off the storm control.

Check the CPU when the problem is ocurring just in case for some reason the packets are being software switched. This would be indicated by high CPU .

See if the behaviour is the same with both ports on one module. (Just as a simplification of the problem.)

Good luck again.

Thanks Anybod.

Bad TAC experience...CallManager and IPCC support is how I developed a bad taste for them.

Storm-control. I did try removing it. It had no impact...so I put it back before I forgot. That's there for a very good reason...a coworker ran "ghost" in broadcast mode during peak hours...not only did it flood the switch, it literally destroyed two T1 ports on an attached 3725. (that's right! a packet storm that caused hardware failure. The gateway was RMA'd a few months later too; never was the same again.)

For a while I played with mls - I didn't set this switch up - least I know more about mls than I did before but toying with that didn't do anything either.

Both on same Module.... yes. Did that a few weeks ago. I get the same results going from gi1/0/24 to gi1/0/23 as I do in the results I posted.

The switches are linked through a stack ring cable...sh switch neigh, detail...ummm...are there stats for the ring?

I mentioned that 1gig 1gig performance smokes, right?

732mbit/sec. (same server I've been testing w/ one that's in production.) (which reminds me...a few servers I still have at gig as they only talk to each other, the file and database servers are set to manual 100/full for the moment.)

CPU utilization....I've never seen it above 10% on the "5 min" interval....hey, my routers do a "sh proc cpu hist" for a dandy graph...catalyst IOS can't do that???

There is another, very odd, side to this story I have not mentioned.

99% of users access servers like this:

PC-->IP Phone-->Stack-->Server

I'm set up a bit different for various reasons:

PC-->Switch-->Switch-->Stack-->Server

Apparently, the performance issue has been with us since this network was put together...about 1.5 years. I never really noticed. Poeple said "this is slow" and - the in-house apps are big so I brushed it off, then one day I saw what they had been talking about and paid attention.

I said that 100/full to 100/full (both ports on stack) got about

80mbits/sec....and I get aobut that too with the extra switch in my patch. I was getting about 30mbit/sec to gbit the server in my special, extra-switch-than-most path. Still very asymmetrical, though. I discovered that, to a point, the more switches I placed between the 100/full and gbit hosts...things get noticeably better, but still not what I think it should be. (Plus such a "fix" is impractical to implement for all.)

I'll open a TAC case.