since the upgrade to 12.3(16a) on a Cisco 2503I router, I noticed that the free processor memory is shrinking. In the beginning there were
9773344 bytes free. After one hour and 20 minutes the free space shrank to
8602384 bytes and this continued until the router reported: %AAA-3-ACCT_LOW_MEM_UID_FAIL: AAA unable to create UID for incoming calls due to insufficient processor memory At this time it was even impossible to access the router via the console port :-( . The router has 16 MB RAM and 16 MB flash. What could cause this memory problem?
Regards, Christoph Gartmann
Didn't find your answer? Ask the community — no account required.
M
Merv
IOS bugs that cause memory leaks - memory is not released after a process is done with it
W
www.BradReese.Com
Hi Christoph,
You may wish to investigate Cisco's Troubleshooting Memory Problems:
formatting link
Sincerely,
Brad Reese BradReese.Com Cisco Repair Service Experts
formatting link
Hendersonville Road, Suite 17 Asheville, North Carolina USA 28803 USA & Canada: 877-549-2680 International: 828-277-7272
W
www.BradReese.Com
Christoph,
Cisco TAC contact information in Germany:
formatting link
Brad Reese Refurbished Cisco List Pricing & Current Inventory Availability
formatting link
C
Christoph Gartmann
No I am bit further: the problem remains under V 12.3(17). It turned out to be the "IP Input" process. What does this process do? Why does it require more an more memory?
Regards, Christoph Gartmann
M
Merv
The process "IP Input" takes care of process-switching IP packets.
router1#sho int stats BRI0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 18 74 18 74 Route cache 0 0 0 0 Total 18 74 18 74 BRI0:1 Switching path Pkts In Chars In Pkts Out Chars Out Processor 0 0 0 0 Route cache 0 0 0 0 Total 0 0 0 0 BRI0:2 Switching path Pkts In Chars In Pkts Out Chars Out Processor 0 0 0 0 Route cache 0 0 0 0 Total 0 0 0 0 Ethernet0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 236792 24179973 22348 2132481 Route cache 3594288 1382376689 3593592 1384114207 Total 3831080 1406556662 3615940 1386246688 Interface Serial0 is disabled
Interface Serial1 is disabled
router1#sho inter switching BRI0
Protocol Other Switching path Pkts In Chars In Pkts Out Chars Out Process 0 0 18 74 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload. BRI0:1
All statistics for this interface are zero. BRI0:2
All statistics for this interface are zero. Ethernet0 Lokales Ethernet Throttle count 2 Drops RP 2 SP 0 SPD Flushes Fast 1 SSE 0 SPD Aggress Fast 0 SPD Priority Inputs 16576 Drops 0
Protocol IP Switching path Pkts In Chars In Pkts Out Chars Out Process 97060 12240266 12308 1380023 Cache misses 97000 - - - Fast 3599742 1384468891 3599046 1386206409 Auton/SSE 0 0 0 0
Protocol ARP Switching path Pkts In Chars In Pkts Out Chars Out Process 16856 1011794 7123 427380 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0
Protocol CDP Switching path Pkts In Chars In Pkts Out Chars Out Process 0 0 437 180044 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0
Protocol Other Switching path Pkts In Chars In Pkts Out Chars Out Process 124386 11038880 2627 157620 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Interface Serial0 is disabled
Interface Serial1 is disabled
M
Merv
The router has CEF enabled which is good
The CEF process accounts for 30% of total CPU usage.
There is a lot of non-IP traffic being received on the Ethernet port
What release of IOS was used on this router previosuly and what was the motivation to upgrade the IOS?
It might be worthwhile to revert and baseline the CPU usage and the memoru usage ( show memory free )
H
Hansang Bae
You should tell people to use "sho proc cpu | excl 0.00"
C
Christoph Gartmann
We have AppleTalk running.
Before it was 12.3(9). We have a problem with VPN-Tunnels over TCP. This router seems to drop the first fragment of an encrypted tunnel packet. So we thought going from 12.3(9) to 12.3(16a) would be worth a try.
This is what we will do next.
Regards, Christoph Gartmann
A
anybody43
There are a couple of things that it might be worth looking at. As stated this is probably a software bug, but just may be legitimate use of memory. This seems unlikely from looking at the process list, if it is complete.
Use sh proc mem at intervals to see if you can locate the process that is using up the memory.
The AAA process mentioned in the log is an obvious candidate to examine closely, however it may be that it was a different process that ate all the memory first.
If you can find it you may be able to turn it off :-))) It may be feature that you don't need.
You can use sh mem (just the top 3 lines) to check that you do not have memory fragmentation. That is - Largest much less than Free. What to do about it is another matter but read on.
It may be worth reducing buffer allocation/free activity. If (by chance) the bug is there you MAY be able to eliminate the effect by statically allocating the required buffers.
Here is for example a router that is suffering some buffer stress.
Small buffers, 104 bytes (total 151, permanent 50, peak 186 @ 6d03h): 149 in free list (20 min, 150 max allowed) 3258440 hits, 15510 misses, 3849 trims, 3950 created 8536 failures (0 no memory)
A failure can resut in a dropped packet.
I would just set 190 permanent small buffers (after checking that I had enough memory).
104 bytes each gives us 104 * 190 = 20,000.
There will probably be an allocation overhead per buffer so don't squeeze the memory too tightly. It would be easy to check using sh mem.
I feel that the Cisco buffer allocation strategy is now too conservative.
By the way:- I had NO idea that this router was suffereing from this issue, thanks for pointing me in that direction.
It is underspecified for the job but we don't want to change it since it will be obsolete soon.
If you like, post the shows and we can have a look.
i.e. first few lines of sh mem when the memory is almost gone, sh buff.
You could always just try various releases. Good luck.
C
Christoph Gartmann
It is "IP input".
[...]
Similar situation here.
Currently I am back to 12.3(9) and here things look all right.
Regards, Christoph Gartmann
M
Merv
If you have a support contract open a case with Cisco regarding your tunnel issue and then the memory leak issue.
C
Christoph Gartmann
Unfortunately I have no contract for this old router :-(
Regards, Christoph Gartmann
Join the Discussion
Have something to add? Share your thoughts — no account required.
Didn't find your answer?
Ask the community — no account required
Report Content
You are reporting this content to the moderators. They will look at it
ASAP.