High cpu with Catalyst 6509 sup2/msfc2/pfc2 512Mb ram

Hello,

We're facing a high cpu problem in one of our catalysts. Unfortunately (and as usual!) there's nothing usefull in sh processes full. It says everything is "0%" and the total is 60%. Amazing. :-(

The strange thing is it happens in only one of our cats. The other one is running well, below 20% with similar number os vlans and traffic.

There are only 2 differences between they. First is that the "problematic" one has a PFC2 with hardware revision 3.2 and the "good" one is hw level 3.3. And the other difference is the "problematic" one has WS-X6248-RJ-45 modules with firmware 5.1(1)CSX and the "good" one has 5.4(2) (same hardware revision).

I've been through a lot of documents in Google but could not found any caveat related to 5.1(1)CSX revision to justify this behavior. Tried to debug lots of information (it could be some defective vlan "attacking" the cat), but no luck at all.

We're not running dcef since our modules doesn't support it.

Can anybody help pointing some debug procedure of pointing any information about 5.1(1)CSX caveats (if any) ?

Here is our sh process cpu after we transfered lots of vlans to the other cat. Thank you!

sh processes cpu

CPU utilization for five seconds: 27%/24%; one minute: 37%; five minutes: 39% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 4 10 400 0.00% 0.00% 0.00% 0 Chunk Manager 2 1756 29819 58 0.00% 0.00% 0.00% 0 Load Meter 3 152 517 294 0.00% 0.00% 0.00% 0 Constellation Ba 4 603236 39558 15249 0.00% 0.22% 0.22% 0 Check heaps 5 4 38 105 0.00% 0.00% 0.00% 0 Pool Manager 6 0 2 0 0.00% 0.00% 0.00% 0 Timers 7 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun 8 10724 77393 138 0.00% 0.00% 0.00% 0 EnvMon 9 0 16 0 0.00% 0.00% 0.00% 0 PF_Split Sync Pr 10 1480 148954 9 0.00% 0.00% 0.00% 0 IPC Periodic Tim 11 0 1 0 0.00% 0.00% 0.00% 0 IPC BackPressure 12 816 148954 5 0.00% 0.00% 0.00% 0 IPC Deferred Por 13 0 69 0 0.00% 0.00% 0.00% 0 IPC Seat Manager 14 1200164 2255539 532 1.55% 0.77% 0.75% 0 ARP Input 15 4 30 133 0.00% 0.00% 0.00% 0 Cat6k SNMP 16 0 453 0 0.00% 0.00% 0.00% 0 Cat6k SNMP Trap 17 172936 36962 4678 0.08% 0.06% 0.08% 0 HC Counter Timer 18 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers 19 0 2 0 0.00% 0.00% 0.00% 0 Dialer event 20 336 318 1056 0.00% 0.00% 0.00% 0 Entity MIB API 21 84 14914 5 0.00% 0.00% 0.00% 0 Compute SRP rate 22 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect 23 16 164 97 0.00% 0.00% 0.00% 0 rf proxy rp agen 24 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd 25 2216 33105 66 0.00% 0.00% 0.00% 0 Net Background 26 15928 2858 5573 0.00% 0.00% 0.00% 0 Logger 27 3756 148944 25 0.00% 0.00% 0.00% 0 TTY Background 28 148412 148963 996 0.00% 0.00% 0.00% 0 Per- Second Jobs 29 0 1 0 0.00% 0.00% 0.00% 0 EOBC Monitor Pro 30 0 6 0 0.00% 0.00% 0.00% 0 RPC rf_rfmib_not 31 0 3 0 0.00% 0.00% 0.00% 0 SCP Multicast 32 12 12 1000 0.00% 0.00% 0.00% 0 RPC Sync 33 4 3 1333 0.00% 0.00% 0.00% 0 RPC rpc- slave-21 34 844 1635 516 0.00% 0.00% 0.00% 0 RPC draco-oir 35 20 42 476 0.00% 0.01% 0.00% 1 Virtual Exec 36 832 30071 27 0.00% 0.00% 0.00% 0 Net Input 37 48 514 93 0.00% 0.00% 0.00% 0 Env action 38 0 1 0 0.00% 0.00% 0.00% 0 PF Sync process 39 76 6255 12 0.00% 0.00% 0.00% 0 pclc_g2_fw 40 0 2 0 0.00% 0.00% 0.00% 0 Flash ICC Server 41 4096 16958 241 0.00% 0.00% 0.00% 0 Const ICC reques 42 0 1 0 0.00% 0.00% 0.00% 0 Const ICC event 43 0 1 0 0.00% 0.00% 0.00% 0 Const ICC except 44 608 6208 97 0.00% 0.00% 0.00% 0 ICC Async mcast 45 5616 148666 37 0.00% 0.00% 0.00% 0 Heartbeat Proces 46 48 53 905 0.00% 0.00% 0.00% 0 RPC pf- split-rp 47 0 1 0 0.00% 0.00% 0.00% 0 Mistral Reset Pr 48 0 1 0 0.00% 0.00% 0.00% 0 ECC correctable 49 1968 148981 13 0.00% 0.00% 0.00% 0 PF_Init Process 50 0 1 0 0.00% 0.00% 0.00% 0 RPC_DEBUG 51 2300008 12423123 185 1.06% 1.09% 0.98% 0 Compute load avg 52 60532 2488 24329 0.32% 0.03% 0.00% 0 Per- minute Jobs 53 8 34 235 0.00% 0.00% 0.00% 0 RPC idprom-MP 54 0 1 0 0.00% 0.00% 0.00% 0 chkpt timer 55 4 8 500 0.00% 0.00% 0.00% 0 Remote Console P 56 0 3 0 0.00% 0.00% 0.00% 0 CWAN OIR Handler 57 0 1 0 0.00% 0.00% 0.00% 0 DSX3MIB ll handl 58 0 2 0 0.00% 0.00% 0.00% 0 Earl NDE Task 59 89488 482218 185 0.00% 0.04% 0.01% 0 esw_vlan_stat_pr 60 0 1 0 0.00% 0.00% 0.00% 0 CT3 Mailbox 61 0 1 0 0.00% 0.00% 0.00% 0 CE3 Mailbox 62 0 1 0 0.00% 0.00% 0.00% 0 MIP Mailbox 63 0 1 0 0.00% 0.00% 0.00% 0 vcq_proc 64 0 1 0 0.00% 0.00% 0.00% 0 HAL Mailbox 65 2392 148969 16 0.00% 0.00% 0.00% 0 CWAN CHDSX PROCE 66 1612 148969 10 0.00% 0.00% 0.00% 0 CWAN CHOCX PROCE 67 820 639 1283 0.00% 0.00% 0.00% 0 SW VLAN RP 68 0 2 0 0.00% 0.00% 0.00% 0 Spanning Tree 69 1132316 9595548 118 0.32% 0.58% 0.71% 0 IP Input 70 124756 255208 488 0.00% 0.06% 0.05% 0 CDP Protocol 71 9904 13222 749 0.00% 0.00% 0.00% 0 CEF Scanner 72 28588 1222 23394 0.00% 0.00% 0.00% 0 QoS Manager 73 116 950 122 0.00% 0.00% 0.00% 0 HWIF QoS Process 74 1377180 30083 45779 1.22% 0.58% 0.55% 0 QOS Stats Export 75 3384 14015 241 0.00% 0.00% 0.00% 0 MOP Protocols 76 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana 77 16 2488 6 0.00% 0.00% 0.00% 0 LDP Background 78 0 1 0 0.00% 0.00% 0.00% 0 frr_tunnel 79 22220 6768 3283 0.16% 0.01% 0.00% 0 IP Background 80 1056 5725 184 0.00% 0.00% 0.00% 0 IP RIB Update 81 0 7 0 0.00% 0.00% 0.00% 0 SNMP Timers 82 0 1 0 0.00% 0.00% 0.00% 0 PPP IP Add Route 83 352 2488 141 0.00% 0.00% 0.00% 0 IP Cache Ager 84 3780 153549 24 0.00% 0.00% 0.00% 0 L3 Manager 85 0 1 0 0.00% 0.00% 0.00% 0 HTTP Timer 86 0 1 0 0.00% 0.00% 0.00% 0 CHKPT EXAMPLE 87 0 1 0 0.00% 0.00% 0.00% 0 CONST_STILE_PROC

88 1060 59856 17 0.00% 0.00% 0.00% 0 TCP Timer 89 272 677 401 0.00% 0.00% 0.00% 0 TCP Protocols 90 0 1 0 0.00% 0.00% 0.00% 0 Probe Input 91 0 1 0 0.00% 0.00% 0.00% 0 RARP Input 92 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers 93 2512 10499 239 0.00% 0.00% 0.00% 0 DHCPD Receive 94 8092 153932 52 0.00% 0.00% 0.00% 0 Feature Manager 95 0 1 0 0.00% 0.00% 0.00% 0 COPS 96 16 2488 6 0.00% 0.00% 0.00% 0 VACL Logger 97 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall 98 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background 99 40 2488 16 0.00% 0.00% 0.00% 0 TCP Intercept Ti 100 0 2 0 0.00% 0.00% 0.00% 0 SPX Input 101 92476 29937 3089 0.16% 0.04% 0.02% 0 Adj Manager 102 359744 297890 1207 0.00% 0.05% 0.07% 0 CEF process 103 71172 28475 2499 0.00% 0.03% 0.01% 0 IPC LC Message H 104 6188 1156513 5 0.00% 0.00% 0.00% 0 CEF IPC Backgrou 105 22344 74523 299 0.00% 0.00% 0.00% 0 HIDDEN VLAN Proc 106 0 1 0 0.00% 0.00% 0.00% 0 CWAN-RP SCP Proc 107 0 7 0 0.00% 0.00% 0.00% 0 CWAN-RP SCP Inpu 108 0 2 0 0.00% 0.00% 0.00% 0 Tag Input 109 0 3 0 0.00% 0.00% 0.00% 0 Any Transport ov 110 0 2 0 0.00% 0.00% 0.00% 0 ETH/ MPLS Backgro 111 0 1 0 0.00% 0.00% 0.00% 0 TTFIB XDR msgQ T 112 0 1 0 0.00% 0.00% 0.00% 0 IP Flow Backgrou 113 0 1 0 0.00% 0.00% 0.00% 0 Laminar Icc Eve 114 0 2 0 0.00% 0.00% 0.00% 0 SLB Main CPU pro 115 0 2 0 0.00% 0.00% 0.00% 0 Rommon upgrade p 116 0 35 0 0.00% 0.00% 0.00% 0 AggMgr Process 117 204 1617 126 0.00% 0.00% 0.00% 0 RPC sw_vlan_rp 118 48 901 53 0.00% 0.00% 0.00% 0 RPC vtp_const_ap 119 116 1495 77 0.00% 0.00% 0.00% 0 RPC stp_sp2mp_ap 120 0 2 0 0.00% 0.00% 0.00% 0 RPC span_switch_ 121 0 2 0 0.00% 0.00% 0.00% 0 RPC dot1x_aaa_rp 122 0 27 0 0.00% 0.00% 0.00% 0 Logger ICC Proce 123 24 288 83 0.00% 0.00% 0.00% 0 RPC idbman_sp2mp 124 36 26 1384 0.00% 0.00% 0.00% 0 RPC pagp_switch_ 125 0 139 0 0.00% 0.00% 0.00% 0 RPC qnq_switch_s 126 0 4 0 0.00% 0.00% 0.00% 0 RPC pt_switch_sp 127 0 6 0 0.00% 0.00% 0.00% 0 RPC udld_rp 128 36 210 171 0.00% 0.00% 0.00% 0 RPC capi_rp 129 0 2 0 0.00% 0.00% 0.00% 0 RPC rf_cli_sp2rp 130 0 2 0 0.00% 0.00% 0.00% 0 RPC snmp_notify 131 0 2 0 0.00% 0.00% 0.00% 0 RPC pm_switch_sp 132 0 2 0 0.00% 0.00% 0.00% 0 RPC psecure_rp 133 437480 11593084 37 0.00% 0.07% 0.09% 0 Standby (HSRP) 134 360 2263 159 0.00% 0.00% 0.00% 0 RPC pm- mp 135 11096 2100678 5 0.00% 0.00% 0.00% 0 PM Callback 136 12 1245 9 0.00% 0.00% 0.00% 0 DHCPD Timer 137 0 7 0 0.00% 0.00% 0.00% 0 EC-Card- Cfg 138 0 7 0 0.00% 0.00% 0.00% 0 PAgP-MP Process 139 0 7 0 0.00% 0.00% 0.00% 0 LACP-MP Process 140 112 42281 2 0.00% 0.00% 0.00% 0 DHCPD Database 141 0 2 0 0.00% 0.00% 0.00% 0 Module enable-di 142 32 35 914 0.00% 0.00% 0.00% 0 RPC c6k_power_rp 143 0 10 0 0.00% 0.00% 0.00% 0 RPC c6k_rp_envir 144 12 20 600 0.00% 0.00% 0.00% 0 RPC lcc_sp 145 0 3 0 0.00% 0.00% 0.00% 0 RPC fabman_sp 146 1216 19779 61 0.00% 0.00% 0.00% 0 RPC online_diag_ 147 0 2 0 0.00% 0.00% 0.00% 0 RPC pikachu_rp 148 376 23650 15 0.00% 0.00% 0.00% 0 RP Error Detecti 149 416 262 1587 0.00% 0.00% 0.00% 0 RPC pclc_g2_fw_r 150 8 8 1000 0.00% 0.00% 0.00% 0 Const FIB MAC 151 0 2 0 0.00% 0.00% 0.00% 0 VTPMIB EDIT BUFF 152 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps 153 0 1 0 0.00% 0.00% 0.00% 0 SONET Traps 154 64 10972 5 0.00% 0.00% 0.00% 0 FCP Router 155 1220 144807 8 0.00% 0.00% 0.00% 0 NTP 156 0 1 0 0.00% 0.00% 0.00% 0 IPC IFCOM Messag 157 0 1 0 0.00% 0.00% 0.00% 0 CWAN OIR IPC Rea 158 6472 69847 92 0.00% 0.00% 0.00% 0 OSPF Hello 159 15668 153013 102 0.00% 0.00% 0.00% 0 OSPF Router 160 7468 3332 2241 0.00% 0.00% 0.00% 2 Virtual Exec 161 112 2122 52 0.00% 0.00% 0.00% 0 TACACS+ 163 4776 22934 208 0.00% 0.00% 0.00% 0 IP SNMP 164 1464 11506 127 0.00% 0.00% 0.00% 0 PDU DISPATCHER 165 6040 11973 504 0.00% 0.00% 0.00% 0 SNMP ENGINE 166 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro 167 0 1 0 0.00% 0.00% 0.00% 0 SNMP Traps

sh ver

Cisco Internetwork Operating System Software IOS (tm) c6sup2_rp Software (c6sup2_rp-JS-M), Version 12.1(20)E2, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) TAC Support:

(c) 1986-2004 by cisco Systems, Inc. Compiled Fri 30-Jan-04 15:09 by pwade Image text-base: 0x40008F90, data-base: 0x41B48000

ROM: System Bootstrap, Version 12.1(11r)E1, RELEASE SOFTWARE (fc1) BOOTLDR: c6sup2_rp Software (c6sup2_rp-JS-M), Version 12.1(20)E2, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)

RJ6509_1 uptime is 1 day, 17 hours, 29 minutes Time since RJ6509_1 switched to active is 1 day, 17 hours, 29 minutes System returned to ROM by power-on (SP by power-on) System restarted at 20:10:58 BRST Sun Jan 28 2007 System image file is "sup-bootflash:c6sup22-js-mz.121-20.E2"

cisco WS-C6509 (R7000) processor (revision 2.0) with 458752K/65536K bytes of memory. Processor board ID SCA043500PS R7000 CPU at 300Mhz, Implementation 39, Rev 3.3, 256KB L2, 1024KB L3 Cache Last reset from power-on Bridging software. X.25 software, Version 3.0.0. SuperLAT software (copyright 1990 by Meridian Technology Corp). TN3270 Emulation software.

147 Virtual Ethernet/IEEE 802.3 interface(s) 144 FastEthernet/IEEE 802.3 interface(s) 52 Gigabit Ethernet/IEEE 802.3 interface(s) 381K bytes of non-volatile configuration memory.

32768K bytes of Flash internal SIMM (Sector size 512K). Configuration register is 0x2142 (will be 0x2102 at next reload)

sh module

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------

----------- 1 2 Catalyst 6000 supervisor 2 (Active) WS-X6K-SUP2-2GE SAL063766YT 3 48 48 port 10/100 mb RJ-45 ethernet WS-X6248-RJ-45 SAD04280WNK 4 48 48 port 10/100 mb RJ-45 ethernet WS-X6248-RJ-45 SAD04290DCU 5 48 48 port 10/100 mb RJ-45 ethernet WS-X6248-RJ-45 SAD04260HEK 7 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX SAL09423BPR

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------

------------ ------- 1 0006.d65b.ce98 to 0006.d65b.ce99 3.10 7.1(1)

12.1(20)E2 Ok 3 0001.9751.b150 to 0001.9751.b17f 1.2 5.1(1)CSX 8.2(0.58)TFW Ok 4 0002.fcb8.1950 to 0002.fcb8.197f 1.2 5.1(1)CSX 8.2(0.58)TFW Ok 5 0001.9751.5030 to 0001.9751.505f 1.2 5.1(1)CSX 8.2(0.58)TFW Ok 7 0015.c69b.3798 to 0015.c69b.37c7 10.1 7.2(1) 8.2(0.58)TFW Ok

Mod Sub-Module Model Serial Hw Status

--- --------------------------- --------------- ---------------

------- ------- 1 Policy Feature Card 2 WS-F6K-PFC2 SAL063761PQ

3.2 Ok 1 Cat6k MSFC 2 daughterboard WS-F6K-MSFC2 SAD063702ZE 2.5 Ok

Mod Online Diag Status

--- ------------------- 1 Pass 3 Pass 4 Pass 5 Pass 7 Pass