Dynamic NAT Failure

It sounds like your pool of port numbers is emptied. Nothing changed to the router, but has the number of LAN user increased compared to two years ago?

How are the ip nat translation time-out values set?

FW

The number of LAN users has probably increased from 30 to 45, at most, over the past 2 years. The timeout values are at their defaults, which I can lookup if needed.

I have read about cases where the pool of port numbers is emptied or the NAT table is filled up when the timeout values are too great, but I would presume there should be at least a few dynamic NAT entries when doing a "show ip nat trans" if that were the case. Instead, I'm not seeing anything at all...

Thanks,

Anyone other ideas on this? Thanks.

Well it looks like a bug or just maybe you are legitimately out of memory. i.e. by design.

If you are lucky you may be able to do something about it.

Please post sh ver sh mem ! First few lines. sh proc mem ! ? I forget exactly, the one that lists memory stats by process sh buff

when you have a failure and after a reboot.

I'll make a note of these commands to run them the next time the failure happens. Given the rare occurence of this problem, this may not be for another month or so...

Thanks for the help.

The failure occurred again this past Sunday. Exactly the same failure... Below is the output from the above commands. Let me know if this sheds any light on this problem.

Router#sh ver Cisco Internetwork Operating System Software IOS (tm) 2500 Software (C2500-I-L), Version 12.0(26), RELEASE SOFTWARE (fc1) Copyright (c) 1986-2003 by cisco Systems, Inc. Compiled Mon 31-Mar-03 18:33 by srani Image text-base: 0x0302F634, data-base: 0x00001000

ROM: System Bootstrap, Version 5.2(5), RELEASE SOFTWARE BOOTFLASH: 3000 Bootstrap Software (IGS-RXBOOT), Version 10.2(5), RELEASE SOFTWA RE (fc1)

Router uptime is 2 weeks, 6 days, 9 hours, 20 minutes System restarted by reload at 09:17:42 EDT Mon Aug 28 2006 System image file is "flash:c2500-i-l.120-26.bin"

cisco 2500 (68030) processor (revision D) with 4096K/2048K bytes of memory. Processor board ID 01716848, with hardware revision 00000000 Bridging software. X.25 software, Version 3.0.0.

2 Ethernet/IEEE 802.3 interface(s) 2 Serial network interface(s) 32K bytes of non-volatile configuration memory. 8192K bytes of processor board System flash (Read ONLY)

Configuration register is 0x2102

Router#sh mem Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor 7C4C0 3681088 1161484 2519604 2319620 2456252 I/O 400000 2097152 392024 1705128 1632252 1699580

Router#sh proc mem Total: 5778240, Used: 1553584, Free: 4224656 PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 36308 1852 1292872 0 0 *Init* 0 0 460 52380 460 0 0 *Sched* 0 0 2443584 972616 12812 377700 0 *Dead* 1 0 268 268 1748 0 0 Load Meter 2 2 1268 0 6016 0 0 Virtual Exec 3 0 0 0 4748 0 0 Check heaps 4 0 829620 0 4844 621600 0 Pool Manager 5 0 268 268 4748 0 0 Timers 6 0 268 268 4748 0 0 Serial Backgroun 7 0 16468 412612 7796 0 0 ARP Input 8 0 268 268 4748 0 0 DDR Timers 9 0 4700 1076 8372 0 0 Entity MIB API 10 0 96 0 4844 0 0 SERIAL A'detect 11 0 35311556 253432 25244 72720 0 IP Input 13 0 244 0 4992 0 0 PPP IP Add Route 14 0 272 0 5020 0 0 X.25 Encaps Mana 15 0 0 7588 6748 0 0 TCP Timer 16 0 49976 0 8256 0 0 TCP Protocols 17 0 340 0 5088 0 0 Probe Input 18 0 96 0 4844 0 0 RARP Input 19 0 1943216 1941180 5984 0 0 BOOTP Server 20 0 228 21988 5976 0 0 IP Background 21 0 0 27862472 4748 0 0 IP Cache Ager 22 0 244 0 4992 0 0 PAD InCall 23 0 364 268 6844 0 0 X.25 Background 24 0 0 0 4748 0 0 Socket Timers 25 0 0 0 4748 0 0 ISDN Timer 27 0 0 0 4748 0 0 CallMIB Backgrou 28 0 0 0 4748 0 0 ISDNMIB Backgrou 29 0 96 0 6844 0 0 SNMP ConfCopyPro 30 0 96 0 4844 0 0 Critical Bkgnd 31 0 37312 22708 7108 0 0 Net Background 32 0 448 268 6928 0 0 Logger 33 0 268 420 4748 0 0 TTY Background 34 0 0 172 5748 0 0 Per-Second Jobs 35 0 192 0 4940 0 0 Net Input 36 0 268 268 4748 0 0 Compute load avg 37 0 6720 195256 4748 5040 715948 Per-minute Jobs 38 0 1948 6674660 6344 0 0 IP NAT Ager 39 0 0 0 4748 0 0 IP RACL Ager 40 0 0 0 4748 0 0 SNMP Timers 41 0 1368 268 7848 0 0 IP SNMP 42 0 96 0 4844 0 0 SNMP Traps 43 0 1512 268 5992 0 0 NTP 1551664 Total

Router#sh buff Buffer elements: 499 in free list (500 max allowed) 61398697 hits, 0 misses, 0 created

Public buffer pools: Small buffers, 104 bytes (total 50, permanent 50): 49 in free list (20 min, 150 max allowed) 565056 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) Middle buffers, 600 bytes (total 25, permanent 25): 23 in free list (10 min, 150 max allowed) 75370 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) Big buffers, 1524 bytes (total 50, permanent 50): 50 in free list (5 min, 150 max allowed) 845807 hits, 1868 misses, 373 trims, 373 created 584 failures (0 no memory) VeryBig buffers, 4520 bytes (total 10, permanent 10): 10 in free list (0 min, 100 max allowed) 0 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) Large buffers, 5024 bytes (total 0, permanent 0): 0 in free list (0 min, 10 max allowed) 0 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) Huge buffers, 18024 bytes (total 0, permanent 0): 0 in free list (0 min, 4 max allowed) 2 hits, 2 misses, 4 trims, 4 created 0 failures (0 no memory)

Interface buffer pools: Ethernet0 buffers, 1524 bytes (total 32, permanent 32): 8 in free list (0 min, 32 max allowed) 593197 hits, 825857 fallbacks 8 max cache size, 5 in cache Ethernet1 buffers, 1524 bytes (total 32, permanent 32): 8 in free list (0 min, 32 max allowed) 1324 hits, 376 fallbacks 8 max cache size, 8 in cache Serial0 buffers, 1524 bytes (total 32, permanent 32): 7 in free list (0 min, 32 max allowed) 24659 hits, 17223 fallbacks 8 max cache size, 8 in cache Serial1 buffers, 1524 bytes (total 32, permanent 32): 7 in free list (0 min, 32 max allowed) 25 hits, 0 fallbacks 8 max cache size, 8 in cache

Thanks!

Comments inline.

Warning - All of this is clutching at straws really but you may just fix it you never know.

Memory OK.

Largest ~= Free ~= Lowest (more or less) Memory not fragmented and you have never run out.

Zero misses is unusual but good!

This is really, really clutching at straws but you may be lucky. The idea of the following is to try to give the router the best opportunity to cope with what may be an overloaded condition.

Failures we don't want. Lets try to get rid of them. There are several options here you could try for example.

conf t buffers big min-free 20 ! 20 * 1524 = 30000 ish

Above will use a bit more than 30k of RAM and you have enough.

You will have to balance the memory that you have with the number of buffers that you allocate.

Quite a lot of fallbacks, I think that these occur when the interface queues are full and the router allocates main memory for more queued packets. Something may be a bit busy.

How is the CPU?

I suggest that you could monitor these buffer failures to see if they occur regularly or maybe in a burst that could be swamping the router and resulting in the failure.

A 2500 is a pretty marginal device in a modern LAN. All it would need is a few broadcasts and it would be filled up for a while.

About the smallest routers that you can get today from Cisco that have not had end-of-life announced are the 850/870. they do 25000/10000 packets per second. A 2500 does 4400 pps.

I have in the past applied access lists to try to protect routers from Windows broadcasts. Search the group for the thread "too many input drops in a 1721 router" "Queue Drops" "Input Drops With An Empty Input Queue"

Did you have to reboot or did "clear ip nat tr *" fix it?

Please also post sh int after failure. If you have another grab a show tech, sorry should have suggested that before.

Sorry that I can't be of more precise assistance.