Picking an IOS for a 2500

James,

Are you receiving:

%SYS-2-MALLOCFAIL

messages?

Sincerely,

Brad Reese BradReese.Com Cisco Repair Service Experts

Hendersonville Road, Suite 17 Asheville, North Carolina USA 28803 USA/Canada Toll Free: 877-549-2680 International: 828-277-7272 United Kingdom: 44-20-70784294

I would have thought so, but no - late collision messages (I'm using the aui ports, so their 10/half, perfectly normal) and rarely (once a week) an AAA error about memory allocating, but I recall it seemed reevant for accounting logs and not packet processing, et al.

If I see another one, I'll post it in, but I'm seeing fairly heavy traffic continuously, and I would have thought memory problems would hit the logs as frequently as the memory dropped, which is continuously.

5 minute input rate 26000 bits/sec, 4 packets/sec 5 minute output rate 20000 bits/sec, 4 packets/sec

Nimiety>sh mem Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor 1BD228 12852696 10257100 2595596 2595596 2595596 I/O E00000 2097152 392580 1704572 1704572 1704212

James,

You may wish to quickly "scan over" this Cisco Memory TAC Case Solution:

http://129.41.16.73/core/showcase?case=K74529211 Sincerely,

Brad Reese Cisco TAC Contacts Worldwide

That looks helpful - I will certainly review it closely. I've also just enabled CEF to see if that helps slow the leak down.

I have a tiny routing table (nothing dynamic, only a few statics), and the router itself has 16M/16M, which is the max for the 2500 chassis, if I'm not mistaken.

I find it interesting that there's no fragmentation ever to the memory (free, lowest and largest are always equal). There does seem to be a fair amount of dead memory:

Nimiety#sh proc mem Total: 14949848, Used: 11611408, Free: 3338440 PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 106608 0 3497036 0 0 *Init* 0 0 272 0 272 0 0 *Sched* 0 0 5308612 0 5548 378300 0 *Dead*

but those figures don't increment the way show mem (or show mem sum) decrease.

Regardless, any additional info on suggested IOSs, or an explanation why a command which should work and doesn't in practice, would be very appreciated! James...

True enough. This is one of those with 14 extra megs to the 2 onboard.

Nimiety>sh ver Cisco Internetwork Operating System Software IOS (tm) 2500 Software (C2500-IS-L), Version 12.3(15a), RELEASE SOFTWARE (fc2) Technical Support:

(c) 1986-2005 by cisco Systems, Inc. Compiled Fri 22-Jul-05 02:59 by dchih Image text-base: 0x0307D684, data-base: 0x00001000

ROM: System Bootstrap, Version 5.2(8a), RELEASE SOFTWARE BOOTLDR: 3000 Bootstrap Software (IGS-RXBOOT), Version 10.2(8a), RELEASE SOFTWARE (fc1)

Nimiety uptime is 1 day, 11 hours, 47 minutes System returned to ROM by reload System image file is "flash:c2500-is-l.123-15a.bin"

cisco 2500 (68030) processor (revision L) with 14336K/2048K bytes of memory. Processor board ID 01771407, with hardware revision 00000000 Bridging software. X.25 software, Version 3.0.0.

2 Ethernet/IEEE 802.3 interface(s) 2 Serial network interface(s) 32K bytes of non-volatile configuration memory. 16384K bytes of processor board System flash (Read ONLY)

BTW, I have to reboot the router again... the memory has dwindled away, but still no memory errors in the logs. I'm sure if I let the memory run out completely, they would come, obviously. James...

Well, for more recent 2500s, yes. As it happens, the older 2500 motherboard had 2M RAM soldered on in addition to the SIMM socket, so they would actually support 18M RAM.

I have no idea why Cisco later removed it.

Regards,

Marco.

So I've just upgraded to 12.3.16 and the leak is still there.

Has anyone had this problem before?

~ So I've just upgraded to 12.3.16 and the leak is still there. ~ ~ Has anyone had this problem before? Troubleshooting Memory Problems Document ID: 6507

OK - point taken - I need to do some homework and come up with some facts and details.

So here's what I have found. I am not getting any MALLOCFAIL errors. After memory has dwindled away, I get these on occasion:

1d08h: %AAA-3-ACCT_LOW_MEM_UID_FAIL: AAA unable to create UID for incoming calls due to insufficient processor memory

After comparing several sh processes memory commands, I found one process which seems to increment at roughly the same rate at which the memory drops, while the other processes remain roughly the same:

PID TTY Allocated Freed Holding Getbufs Retbufs Process 31 0 5030248 0 71112 36360 0 IP Input

In fact comparing about an hour later show the change already:

PID TTY Allocated Freed Holding Getbufs Retbufs Process 31 0 5166172 0 71476 36360 0 IP Input

Now, the troubleshooting guide suggests that this is usually found in fragmentation cases (not mine, as free, lowest and largest are identical in size), and to resolve it by either adding memory (not an option on my 16m/16m 2514) or to fix the problem. Hmmm. There's also a known issues note about the CiscoWorks app, which I don't use, which is a shame, because that's exactly what happens.

So, back to IP Input - the known issues note mentions that the cache entry table grows filling up the memory, and I realised that I'm using several NAT entries and extending them all:

ip nat inside source static tcp a.b.c.4 21 w.x.y.z 21 extendable ip nat inside source static tcp a.b.c.10 80 w.x.y.z 8001 extendable ip nat inside source static tcp a.b.c.4 53 w.x.y.z 53 extendable ip nat inside source static udp a.b.c.4 53 w.x.y.z 53 extendable ip nat inside source static tcp a.b.c.4 80 w.x.y.z 80 extendable ip nat inside source static tcp a.b.c.4 25 w.x.y.z 25 extendable ip nat inside source static tcp a.b.c.3 22 w.x.y.z 22 extendable

Also, my nat translations table is always large, several screenfuls - I think this may be it, so here are the stats despite clearing them yesterday afternoon:

Nimiety#sh ip nat st Total active translations: 101 (7 static, 94 dynamic; 101 extended) Outside interfaces: Dialer1, Ethernet1, Virtual-Access1 Inside interfaces: Ethernet0 Hits: 357371 Misses: 14397 Expired translations: 21626 Dynamic mappings:

-- Inside Source [Id: 1] access-list 1 interface Dialer1 refcount 5

So my questions are: can I unextend the nat translations above without affecting traffic? I have an ADSL link with several servers behind this router as you can see. If not, can I either limit the size of the IP Table process or forceably clear that process when it grows too large without reloading the router? I can go barely more than a day before running out of memory and being forced to reload or reboot, so I appreciate any input or suggestions you may have!

IOS (tm) 2500 Software (C2500-IS-L), Version 12.3(16), RELEASE SOFTWARE (fc4) Nimiety uptime is 1 day, 20 minutes

N....