since the upgrade to 12.3(16a) on a Cisco 2503I router, I noticed that the free processor memory is shrinking. In the beginning there were
9773344 bytes free. After one hour and 20 minutes the free space shrank to
8602384 bytes and this continued until the router reported: %AAA-3-ACCT_LOW_MEM_UID_FAIL: AAA unable to create UID for incoming calls due to insufficient processor memory At this time it was even impossible to access the router via the console port :-( . The router has 16 MB RAM and 16 MB flash. What could cause this memory problem?
Before it was 12.3(9). We have a problem with VPN-Tunnels over TCP. This router seems to drop the first fragment of an encrypted tunnel packet. So we thought going from 12.3(9) to 12.3(16a) would be worth a try.
There are a couple of things that it might be worth looking at. As stated this is probably a software bug, but just may be legitimate use of memory. This seems unlikely from looking at the process list, if it is complete.
Use sh proc mem at intervals to see if you can locate the process that is using up the memory.
The AAA process mentioned in the log is an obvious candidate to examine closely, however it may be that it was a different process that ate all the memory first.
If you can find it you may be able to turn it off :-))) It may be feature that you don't need.
You can use sh mem (just the top 3 lines) to check that you do not have memory fragmentation. That is - Largest much less than Free. What to do about it is another matter but read on.
It may be worth reducing buffer allocation/free activity. If (by chance) the bug is there you MAY be able to eliminate the effect by statically allocating the required buffers.
Here is for example a router that is suffering some buffer stress.
Small buffers, 104 bytes (total 151, permanent 50, peak 186 @ 6d03h): 149 in free list (20 min, 150 max allowed) 3258440 hits, 15510 misses, 3849 trims, 3950 created 8536 failures (0 no memory)
A failure can resut in a dropped packet.
I would just set 190 permanent small buffers (after checking that I had enough memory).
104 bytes each gives us 104 * 190 = 20,000.
There will probably be an allocation overhead per buffer so don't squeeze the memory too tightly. It would be easy to check using sh mem.
I feel that the Cisco buffer allocation strategy is now too conservative.
By the way:- I had NO idea that this router was suffereing from this issue, thanks for pointing me in that direction.
It is underspecified for the job but we don't want to change it since it will be obsolete soon.
If you like, post the shows and we can have a look.
i.e. first few lines of sh mem when the memory is almost gone, sh buff.
You could always just try various releases. Good luck.