1. Home
  2. Cisco Systems
  3. Virtual exec process ... > 60% CPU on 6513

Virtual exec process ... > 60% CPU on 6513

Hello all,

Does anyone know why the virtual exec process on our 6513 causes moments of > 60% cpu utilisation?

We've been having 'problems' on and off ie. the network goes down but all we ever see in logs are eigrp neighbour changes. 'sh proc cpu history' shows >

80% cpu utilisation during any 72 hour period but our box/es are not moving that much traffic.

We set up a notebook running 'expect' that polls the 6513's process cpu utilisation every minute and 95 % of the time high utilisation, as a combination of the 'virtual exec' and some 'snmp' process, coincides with a scheduled CiscoWorks poll at 2am but a few times the virtual exec process has been high during the day when all we've been doing was logged into the switch to make a switchport change and review the config.

I realise that having 'expect' log into the switch every minute and run a command adds to the utilisation but the norm is < 25% total. It's just a few times a day we see this 'spike' and it's always the virtual exec process. Apart from it and the snmp process, when CiscoWorks runs, no other process records greater than single digit utilization.

Any suggestions / advice appreciated.

BernieM

Reply to
BernieM
Loading thread data ...

Hi Bernie,

Cisco software-based routers use software in order to process and route packets. CPU utilization on a Cisco router tends to increase as the router performs more packet processing and routing. Therefore, the show processes cpu command can provide a fairly accurate indication of the traffic processing load on the router.

Catalyst 6500/6000 switches do not use the CPU in the same way. These switches make forwarding decisions in hardware, not in software. Therefore, when the switches make the forwarding or switching decision for most frames that pass through the switch, the process does not involve the Supervisor Engine CPU.

On Catalyst 6500/6000 switches, there are two CPUs. One CPU is the Supervisor Engine CPU, which is called the Network Management Processor (NMP) or the Switch Processor (SP). The other CPU is the Layer 3 (L3) routing engine CPU, which is called the MSFC or the Route Processor (RP).

You may wish to investigate Common Causes and Solutions for High CPU Utilization Issues:

formatting link
Found on the Cisco web page Catalyst 6500/6000 Switch High CPU Utilization:

formatting link
Hope this helps.

Brad Reese BradReese.Com - Cisco Network Engineer Directory

formatting link
Hendersonville Road, Suite 17 Asheville, North Carolina USA 28803 USA & Canada: 877-549-2680 International: 828-277-7272 Fax: 775-254-3558 AIM: R2MGrant Website:
formatting link

Reply to
www.BradReese.Com

formatting link

formatting link

Thanks for the reply Brad.

One of the main things I picked up from the document that appears to relate to our environment is the affect that ACL's can have. Specifically, as we typically have "ip unreachables" enabled on our vlan interfaces, some ACL-denied packets are leaked to the MSFC (default rate of 500 packets per second) as are ACE's that require logging. We have 'big' ACL's, a default 'any any log' at the end, and lots Windows hosts in test/development environments that try to do all sorts of weird things.

I also now understand the relationship between the two % values in the top row of "show processes cpu" output.

In the following example 57% is the total CPU utilisation while 48% is the interrupt CPU utilization. The individual process % values total the difference between the two.

CPU utilization for five seconds: 57%/48%; one minute: 56%; five minutes:

48% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 0 5 0 0.00% 0.00% 0.00% 0 Chunk Manager 2 12 18062 0 0.00% 0.00% 0.00% 0 Load Meter 4 164532 13717 11994 0.00% 0.21% 0.17% 0 Check heaps 5 0 1 0 0.00% 0.00% 0.00% 0 Pool Manager !--- Output suppressed. 172 0 9 0 0.00% 0.00% 0.00% 0 RPC aapi_rp 173 243912 2171455 112 9.25% 8.11% 7.39% 0 SNMP ENGINE 174 68 463 146 0.00% 0.00% 0.00% 0 RPC pm-mp

Tomorrow I'll do some "show inter | inc Switched" to determine the % of traffic getting punted to the CPU ie.

L2 Switched: ucast: 0 pkt, 184954624 bytes - mcast: 1 pkt, 500 bytes L3 in Switched: ucast: 2889916 pkt, 0 bytes - mcast: 0 pkt, 0 bytes mcast

I also have a permanent 'mon session' set up to pass WAN traffic to a Packeteer 'PacketSeeker' box for traffic profiling. I'll revise that approach.

Am I correct in saying .... It appears that it isn't so much what gets processed by software but the fact the CPU has to be 'interrupted'?

Rgds,

BernieM

Reply to
BernieM

In your first post, you have probably nailed the problem. Your network "problem" is probably related to EIGRP. You need to investigate why you have EIGRP neighbor loss. The CPU issues could be the why the neighbors are resetting. If you have big ACL's, this could be a source of the high CPU. There is a document on CCO that explains ACL's in details and how they affect each routing platform. ACL's that are too big (or not written correctly) can cause ALL packets to be 'punted' to the processor to be processed switched. Use of the "log" parameter on ACL's should only be used to debug ACL's, and should not used in production. Again, this could be the source of high CPU. Lastly, if you are using HSRP, "no ip unreachables" needs to be configured on any VLAN interfaces running.it. SPAN is done in hardware, so I doubt this is the source of your problem.

Scott

formatting link
>

formatting link
>

Reply to
thrill5

Cabling-Design.com Forums website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.