CPU utilization on the router.

YES.

The first line of sh proc cpu shows xy/za.

This represents

xy total cpu.

za Interrupt level CPU.

The latter is NOT accounted for in the rest of the displayed details since I guess the machine is too busy forwarding packets to do accounting. This is the reason you bought it.

If this does not explain your observations please come back for more:-)

Capture the output of

term len 0 sh proc cpu

Import into excel, sort by the field runtime(ms)whi will show which processes are using the largest amount of CPU

Like the others said you have a problem with interrupt cpu utilization. Check if you have CEF enabled on this router... If you do then there is a lot of "CEF unsupported" packets, so they have to be fast switched using CPU interrupts. This from my experience could be caused by high number of NAT translations if this router is performing NAT...

CEF must be enabled on ingress interface(s)...

Let me know what you find out or/and if you need IOS command support...

B.R. Igor

Post the output of

sh version

sh int stats

sh ip cef

and post your config

Many thanks....

------------------------------------------------------------------------------------------------------------------------

Cisco Internetwork Operating System Software IOS (tm) 3600 Software (C3620-IK9O3S7-M), Version 12.3(13a), RELEASE SOFTWARE (fc2) [CUT] ROM: System Bootstrap, Version 11.1(7)AX [kuong (7)AX], EARLY DEPLOYMENT RELEASE SOFTWARE (fc2) ROM: 3600 Software (C3620-IK9O3S7-M), Version 12.3(13a), RELEASE SOFTWARE (fc2)

Borderline uptime is 4 days, 58 minutes System returned to ROM by reload System image file is "flash:c3620-ik9o3s7-mz.123-13a.bin"

[CUT]

cisco 3620 (R4700) processor (revision 0x81) with 61440K/4096K bytes of memory. Processor board ID R5S03T1A R4700 CPU at 80MHz, Implementation 33, Rev 1.0 Bridging software. X.25 software, Version 3.0.0. Basic Rate ISDN software, Version 1.1.

2 Ethernet/IEEE 802.3 interface(s) 1 Serial network interface(s) 4 ISDN Basic Rate interface(s) DRAM configuration is 32 bits wide with parity disabled. 29K bytes of non-volatile configuration memory. 32768K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

------------------------------------------------------------------------------------------------------------------------

Ethernet0/0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 866391 61385831 832953 114705969 Route cache 2408528 1045733747 1911060 397805101 Total 3274919 1107119578 2744013 512511070 Interface Serial0/0 is disabled

Ethernet0/1 Switching path Pkts In Chars In Pkts Out Chars Out Processor 342236 80784405 73984 8393092 Route cache 1301711 340221922 1799144 982834951 Total 1643947 421006327 1873128 991228043

-------------------------------------------------------------------------------------------------------------------------

Prefix Next Hop Interface

0.0.0.0/0 xxxxxxxxxxxxx Ethernet0/0 0.0.0.0/32 receive 10.174.231.0/24 192.168.38.137 Ethernet0/1 192.168.38.136/29 attached Ethernet0/1 192.168.38.136/32 receive 192.168.38.137/32 192.168.38.137 Ethernet0/1 192.168.38.142/32 receive 192.168.38.143/32 receive 192.168.38.192/28 192.168.38.137 Ethernet0/1 Eth0/0 LAN---/24 attached Ethernet0/0 Eth0/0 LAN---/32 receive 1st ISP's GW /32 1st ISP's GW Ethernet0/0 2nd ISP's GW /32 2nd ISP's GW Ethernet0/0 Eth0 public IP/32 receive Eth0/0 bcast/32 receive 224.0.0.0/4 drop 224.0.0.0/24 receive 255.255.255.255/32 receive

-------------------------------------------------------------------------------------------------------------------------

! version 12.3 service timestamps debug datetime msec service timestamps log datetime msec service password-encryption ! hostname Borderline ! boot-start-marker boot-end-marker ! no logging on enable secret 5 ffffffffffffffffffffffffff ! no aaa new-model ip subnet-zero ! ! ip cef no ip domain lookup ip domain name fffffffff ip name-server xxxxxxxxxxxxxxx ip name-server xxxxxxxxxxxxxxx ! ip audit po max-events 100 ! ! username foofoo1 password 7 fffffffffffffffff username foofoo2 password 7 ggggggggggggggggggggg ! ! ! ! ! interface Ethernet0/0 ip address xxxxxxxxxxxxxx 255.255.255.0 ip nat outside full-duplex ! interface Serial0/0 no ip address shutdown ! interface Ethernet0/1 ip address 192.168.38.142 255.255.255.248 ip nat inside full-duplex ! interface BRI1/0 no ip address shutdown ! interface BRI1/1 no ip address shutdown ! interface BRI1/2 no ip address shutdown ! interface BRI1/3 no ip address shutdown ! ip nat inside source list 112 interface Ethernet0/0 overload ip nat inside source static tcp 192.168.38.137 443 interface Ethernet0/0 443 ip nat inside source static 192.168.38.193 xxxxxxxxxxxxxxx ip nat inside source static 192.168.38.137 xxxxxxxxxxxxxxx no ip http server no ip http secure-server ip classless ip route 0.0.0.0 0.0.0.0 xxxxxxxxxxxxxxxxx ip route 10.174.231.0 255.255.255.0 192.168.38.137 ip route 192.168.38.192 255.255.255.240 192.168.38.137 ! ! ! ip access-list extended vty-access permit tcp 10.174.231.0 0.0.0.255 any eq 22 permit tcp 10.174.231.0 0.0.0.255 any eq telnet permit tcp xxxxxxxxxxxxxx 0.0.1.255 any eq 22 permit tcp xxxxxxxxxxxxxx 0.0.0.31 any eq 22 access-list 1 permit 192.168.79.0 0.0.0.255 access-list 1 deny 10.0.0.0 0.255.255.255 access-list 1 deny 172.0.0.0 0.31.255.255 access-list 1 deny 192.168.0.0 0.0.255.255 access-list 1 permit xxxxxxxxxxxx 0.0.0.31 access-list 1 permit xxxxxxxxxxxx 0.0.1.255 access-list 1 deny any access-list 100 permit udp host 192.168.38.137 eq isakmp host xxxxxxxxxxxx eq isakmp access-list 100 permit udp host 192.168.38.137 eq non500-isakmp host xxxxxxxxxxxx eq non500-isakmp access-list 100 permit esp host 192.168.38.137 host xxxxxxxxxxxx access-list 100 deny ip any any access-list 111 permit udp host 192.168.38.137 eq isakmp host xxxxxxxxxxxx eq isakmp access-list 111 permit udp host 192.168.38.137 eq non500-isakmp host xxxxxxxxxxxx eq non500-isakmp access-list 111 permit esp host 192.168.38.137 host xxxxxxxxxxxx access-list 112 permit ip 10.174.231.0 0.0.0.255 any access-list 112 permit ip 192.168.38.136 0.0.0.7 any access-list 112 permit ip 192.168.38.192 0.0.0.15 any no cdp run ! route-map NAT-VPN permit 10 match ip address 111 match interface Ethernet0/0 ! ! line con 0 line aux 0 line vty 0 4 access-class vty-access in login local ! ! end

1. CEF is enabled - this is a goood thing.

1. The percentage of packet processed switched does not seem to be a problem

What is the packet per second rate when CPU is at 90%. Sugges that you configure load-interval 30 on each Ethernet interface and then clear counters.

Post: show process cpu show interface when router CPU utilization is high

Also suggest you read the following Cisco doc on troubleshooting high CPU utilization:

Already read, but not deeply, so I will have another look.

Thanks,

Alex.

Post

show process cpu

show interface

when you get a chance

If most of the CPU is interrupt level (fast/CEF/whatever but NOT process) switching then the router is doing what it should and there may not be much you can do.

You have indicated that the CPU is not accounted for in the sh proc cpu list which suggests that you are indeed fast switching. This is confirmed by the stats that you have posted.

I suspec that NAT is fast switched but it will still be a lot of extra work.

Normal fast switching. Look up cache Get new MAC header from cache Decrement TTL Fix up IP checksum IIRC?

Oh my head hurts. Basically if nearly all of your CPU is Interrupt level

[e.g. CPU utilization for five seconds: 95%/92%; one minute: 90%; five minutes: 83% 95% of which 92% is Interrupt level]

then the box is full and you have to take some load off, or suffer, or get a bigger one.

End of story. Period.

Removing say NAT seems like it would help a lot but I have no direct experience.

Post "sh proc cpu" please to confirm.

You may of course be passing unwanted traffic e.g. some kind of DoS attack but that is another story altogether.

Good luck.

I had experience with such issues and there was always about NAT:) NAT is fast switching operation (using cpu interrupts), but I think latest IOS-es are using CEF for NAT since you have 'CEF translated packets" counter in the 'show ip nat stat' command output.

I had about 200 hosts connected to the Internet trough my 2811 box and after I blocked p2p clients CPU interrupt usage has been fallen on 40% (since I have another about 250 hosts connected to the Internet, but they are not NATed by this box). p2p clients are causing a lot of NAT translations by single host since they are connected with multiple peers - thus there are multiple NAT translations generated by a single host.

Post 'show ip nat stat' output...

B.R. Igor

Thanks Igor,

I thought about p2p and I did "show ip nat tra" and "show ip nat sta" but p2p is not my case as I had only static translations shown by the output.

I don't know what to think...

Alex.

I would be very surprised if the number of NAT entries affected the CPU significantly. It will use a bit of memory however Cisco do no memory management so extra memory use cannot affect the CPU other than the initial allocation process and a possible subsequent free.

Algorithms exist that can do such table lookups very efficiently and I am sure that cisco have heard of them.

Internal hosts doing port scans of course could generate a lot of NAT table activity.

NATted traffic on the other hand will I am sure use quite a bit more CPU than non NATted traffic, CEF or no CEF however this will I believe be independent of the number of NAT table entries.

One other thing that can affect the CPU of a smallish router very badly are broadcasts. I have in the past put

acess-list 100 deny ip any 255.255.255.255 255.255.255.255 acess-list 100 deny ip any local-ip-net local-ip-net-wildcard

on the interface of a small router.

This will of course break some things, e.g dhcp forwarding, RIP, but basic IP routing is OK. ARP is OK too since it does not use the IP protocol (0800?) but uses (0806?) instead.

Many arp requests?

These DoS like things will though show up in the sh proc cpu as non-interrupt level tasks.

Anybody,

I just know from my past experience that when I eliminated p2p that where NATed I significantly reduced my CPU interrupt load. Also, Cisco says in it's high cpu load troubleshooting guides that NAT is interrupt process that uses cache switching instead of CEF (I'm not sure that this considers latest IOS versions), so of course NAT is mostly memory issue, but could bring to the high cpu load due to interrupts...

B.R. Igor

I'm waiting for a Contract token and then ask Cisco expert. I will inform you about the solution.

Thanks to all, Alex