since the upgrade to 12.3(16a) on a Cisco 2503I router, I noticed that the free processor memory is shrinking. In the beginning there were
9773344 bytes free. After one hour and 20 minutes the free space shrank to
8602384 bytes and this continued until the router reported: %AAA-3-ACCT_LOW_MEM_UID_FAIL: AAA unable to create UID for incoming calls due to insufficient processor memory At this time it was even impossible to access the router via the console port :-( . The router has 16 MB RAM and 16 MB flash. What could cause this memory problem?
No I am bit further: the problem remains under V 12.3(17). It turned out to be the "IP Input" process. What does this process do? Why does it require more an more memory?
router1#sho int stats BRI0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 18 74 18 74 Route cache 0 0 0 0 Total 18 74 18 74 BRI0:1 Switching path Pkts In Chars In Pkts Out Chars Out Processor 0 0 0 0 Route cache 0 0 0 0 Total 0 0 0 0 BRI0:2 Switching path Pkts In Chars In Pkts Out Chars Out Processor 0 0 0 0 Route cache 0 0 0 0 Total 0 0 0 0 Ethernet0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 236792 24179973 22348 2132481 Route cache 3594288 1382376689 3593592 1384114207 Total 3831080 1406556662 3615940 1386246688 Interface Serial0 is disabled
Interface Serial1 is disabled
router1#sho inter switching BRI0
Protocol Other Switching path Pkts In Chars In Pkts Out Chars Out Process 0 0 18 74 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload. BRI0:1
All statistics for this interface are zero. BRI0:2
All statistics for this interface are zero. Ethernet0 Lokales Ethernet Throttle count 2 Drops RP 2 SP 0 SPD Flushes Fast 1 SSE 0 SPD Aggress Fast 0 SPD Priority Inputs 16576 Drops 0
Protocol IP Switching path Pkts In Chars In Pkts Out Chars Out Process 97060 12240266 12308 1380023 Cache misses 97000 - - - Fast 3599742 1384468891 3599046 1386206409 Auton/SSE 0 0 0 0
Protocol ARP Switching path Pkts In Chars In Pkts Out Chars Out Process 16856 1011794 7123 427380 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0
Protocol CDP Switching path Pkts In Chars In Pkts Out Chars Out Process 0 0 437 180044 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0
Protocol Other Switching path Pkts In Chars In Pkts Out Chars Out Process 124386 11038880 2627 157620 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Before it was 12.3(9). We have a problem with VPN-Tunnels over TCP. This router seems to drop the first fragment of an encrypted tunnel packet. So we thought going from 12.3(9) to 12.3(16a) would be worth a try.
There are a couple of things that it might be worth looking at. As stated this is probably a software bug, but just may be legitimate use of memory. This seems unlikely from looking at the process list, if it is complete.
Use sh proc mem at intervals to see if you can locate the process that is using up the memory.
The AAA process mentioned in the log is an obvious candidate to examine closely, however it may be that it was a different process that ate all the memory first.
If you can find it you may be able to turn it off :-))) It may be feature that you don't need.
You can use sh mem (just the top 3 lines) to check that you do not have memory fragmentation. That is - Largest much less than Free. What to do about it is another matter but read on.
It may be worth reducing buffer allocation/free activity. If (by chance) the bug is there you MAY be able to eliminate the effect by statically allocating the required buffers.
Here is for example a router that is suffering some buffer stress.
Small buffers, 104 bytes (total 151, permanent 50, peak 186 @ 6d03h): 149 in free list (20 min, 150 max allowed) 3258440 hits, 15510 misses, 3849 trims, 3950 created 8536 failures (0 no memory)
A failure can resut in a dropped packet.
I would just set 190 permanent small buffers (after checking that I had enough memory).
104 bytes each gives us 104 * 190 = 20,000.
There will probably be an allocation overhead per buffer so don't squeeze the memory too tightly. It would be easy to check using sh mem.
I feel that the Cisco buffer allocation strategy is now too conservative.
By the way:- I had NO idea that this router was suffereing from this issue, thanks for pointing me in that direction.
It is underspecified for the job but we don't want to change it since it will be obsolete soon.
If you like, post the shows and we can have a look.
i.e. first few lines of sh mem when the memory is almost gone, sh buff.
You could always just try various releases. Good luck.
Cabling-Design.com Forums website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.