I have a problem with a Cisco Catalyst 3750G switch/router.
In my tests, I could not get the 3750G to reliably route more than 3400 IPv4 packets per second when these packets contained IP options.
The packets were UDP/IPv4. I tried both small (payload = 100 bytes) and large (payload = 1450 bytes) packets.
The IP options consisted of 4 or 8 NOPs (option type 1). Thus there were no alignment issues, and nothing extra for the router to process.
3570 pps && payload= 100 : 1.1% packet loss
3570 pps && payload=1450 : 2.3% packet loss
5000 pps && payload=1450 : 30.2% packet loss
The following page does not mention my situation:
formatting link
I ran show process cpu:
CPU utilization for five seconds: 87%/29%; one min: 89%; five min: 73% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process [...] 150 7678123 5843620 1313 54.63% 53.74% 43.74% 0 IP Input
A few entries show less than 0.15% CPU usage. All other show 0%.
Where does the 87% figure come from?
Is it 87% overall with 29% IRQ servicing => 58% other than IRQ.
Why is the router dropping packets if the CPU is not maxed out?
Is this a known problem?
Has it perhaps been fixed in a more recent firmware revision?
For reference: >show version Cisco IOS Software, C3750 Software (C3750-IPSERVICES-M), Version
12.2(25)SEC, RELEASE SOFTWARE (fc2) Copyright (c) 1986-2005 by Cisco Systems, Inc. Compiled Thu 14-Jul-05 21:33 by antonino
ROM: Bootstrap program is C3750 boot loader BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(25r)SE, RELEASE SOFTWARE (fc)
cyrus uptime is 1 week, 2 days, 18 hours, 44 minutes System returned to ROM by power-on System image file is "flash:/c3750-ipservices-mz.122-25.SEC.bin"
cisco WS-C3750G-24TS-1U (PowerPC405) processor (revision A0) with
118784K/12280K bytes of memory. Processor board ID FOC0902U0GY Last reset from power-on
17 Virtual Ethernet interfaces
28 Gigabit Ethernet interfaces The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory. Base ethernet MAC Address : 00:13:19:86:CA:00 Motherboard assembly number : 73-9637-07 Power supply part number : 341-0098-01 Motherboard serial number : FOC09020A2U Power supply serial number : DCA084701XH Model revision number : A0 Motherboard revision number : A0 Model number : WS-C3750G-24TS-S1U System serial number : FOC0902U0GY SFP Module assembly part number : 73-7757-02 SFP Module revision Number : A0 SFP Module serial number : CAT085207B5 Top Assembly Part Number : 800-25730-01 Top Assembly Revision Number : A0 Version ID : 01 CLEI Code Number : CNMWS00ARA Hardware Board Revision Number : 0x05
The 3750 is primarily a hardware basd router but can fall back to CPU routing under certain circumstances. You seem to have hit thse curcumstances.
It will doo wire rate hardware routing but the software routing performance is poor. Further there are two kinds of software routing (fast switched and process switched) and you are getting a lot of process switching.
CPU utilization for five seconds: 87%/29%; one min: 89%; five min:
73%
87% total CPU of which 29% is fast switching packets.
The "IP Input" process is the one that does the process switching.
If you look for documents that cover troubleshooting high cpu that ar relevant to you platform you may get something that explains all of the details.
So whatever your options are doing is causing the switch to fall back to CPU based routing.
All interfaces involved are 100 Mbit/s and full duplex.
On the sender: # ethtool eth2 Settings for eth2: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: MII PHYAD: 1 Transceiver: internal Auto-negotiation: on Current message level: 0x000020c1 (8385) Link detected: yes
That interface is connected to port 17 on the 3750: Name : Gi1/0/17 Administrative Speed: auto Administrative Duplex: auto Administrative Auto-MDIX: on Administrative Power Inline: N/A Operational Speed: 100 Operational Duplex: full Operational Auto-MDIX: on
On the receiver: Settings for eth2: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: MII PHYAD: 1 Transceiver: internal Auto-negotiation: on Current message level: 0x000020c1 (8385) Link detected: yes
That interface is connected to port 19 on the 3750: Name : Gi1/0/19 Administrative Speed: auto Administrative Duplex: auto Administrative Auto-MDIX: on Administrative Power Inline: N/A Operational Speed: 100 Operational Duplex: full Operational Auto-MDIX: on
What problem do you suspect?
I failed to mention that the router doesn't drop any packets when packets do not carry any IP option.
Vlan17 is up, line protocol is up Hardware is EtherSVI, address is 0013.1986.ca48 (bia 0013.1986.ca48) Internet address is 10.1.17.254/24 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 10/255 Encapsulation ARPA, loopback not set ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:01, output hang never Last clearing of "show interface" counters never Input queue: 21/75/35000/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 41465000 bits/sec, 3474 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 43540876 packets input, 3580507536 bytes, 0 no buffer Received 0 broadcasts (364590 IP multicast) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 52036 packets output, 3613284 bytes, 0 underruns 0 output errors, 0 interface resets 0 output buffer failures, 0 output buffers swapped out
Vlan19 is up, line protocol is up Hardware is EtherSVI, address is 0013.1986.ca4a (bia 0013.1986.ca4a) Internet address is 10.1.19.254/24 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, reliability 255/255, txload 10/255, rxload 1/255 Encapsulation ARPA, loopback not set ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:27:22, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 41577000 bits/sec, 3469 packets/sec 289 packets input, 17454 bytes, 0 no buffer Received 0 broadcasts (0 IP multicast) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 33807273 packets output, 3212600286 bytes, 0 underruns 0 output errors, 0 interface resets 0 output buffer failures, 0 output buffers swapped out
GigabitEthernet1/0/17 is up, line protocol is up (connected) Hardware is Gigabit Ethernet, address is 0013.1986.ca11 (bia
0013.1986.ca11) MTU 9000 bytes, BW 100000 Kbit, DLY 100 usec, reliability 255/255, txload 1/255, rxload 122/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX input flow-control is off, output flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 48059000 bits/sec, 4019 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 56540589 packets input, 382292988 bytes, 0 no buffer Received 745135 broadcasts (0 multicast) 0 runts, 0 giants, 0 throttles 1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 745042 multicast, 0 pause input 0 input packets with dribble condition detected 434362 packets output, 35087682 bytes, 0 underruns 0 output errors, 0 collisions, 1 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out
GigabitEthernet1/0/19 is up, line protocol is up (connected) Hardware is Gigabit Ethernet, address is 0013.1986.ca13 (bia
0013.1986.ca13) MTU 9000 bytes, BW 100000 Kbit, DLY 100 usec, reliability 255/255, txload 105/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX input flow-control is off, output flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 41515000 bits/sec, 3471 packets/sec 7177 packets input, 4052852 bytes, 0 no buffer Received 81 broadcasts (0 multicast) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 18 multicast, 0 pause input 0 input packets with dribble condition detected 45385038 packets output, 1476776494 bytes, 0 underruns 0 output errors, 0 collisions, 1 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out
probably the switch goes to software forwarding as some people have already said. I suppose that you have not configured any of the "ip accounting, access-lists etc)"
what you have sent shows that probably the packets are dropped in the input queue of SVI "Vlan17" (Input queue: 21/75/35000/0 (size/max/ drops/flushes).
You could try (as a workaround): a. raise the input queue (eg hold-queue 1000 in) b. try using l3 ports instead of SVI. (conf t; int fa0/17 ; no switchport; ip address 10.1.17.254 255.255.255.0)
I guess that the best thing you can do is to upgrade to latest software and if the problems perists, open a case with TAC.
Cabling-Design.com Forums website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.