NM-1E1R2W problem in 3620

Does the router's receiving interface show any input drops ?

post show interface show buffers

In article , snipped-for-privacy@rogers.com (Merv) writes: | | > I know that it is a received (ACK) packet that is being dropped | each time | | Does the router's receiving interface show any input drops ? | | post | show interface | show buffers

Below, along with a few others. This is after two copies of a ~16M image, stalling a total of 35 times (must be a bad day). Oh, I should also mention that I've tried several versions of IOS from 11.3 to 12.3 with no change.

Router#sho int e0/0 Ethernet0/0 is up, line protocol is up Hardware is AmdP2, address is 0010.7b68.65a1 (bia 0010.7b68.65a1) Internet address is 10.0.0.220/24 MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, reliability 255/255, txload 8/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:05, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 38000 bits/sec, 79 packets/sec 5 minute output rate 333000 bits/sec, 78 packets/sec 64602 packets input, 3882560 bytes, 0 no buffer Received 1546 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 63199 packets output, 35139231 bytes, 0 underruns 0 output errors, 18 collisions, 2 interface resets 0 babbles, 0 late collision, 94 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out Router#sho buffer Buffer elements: 500 in free list (500 max allowed) 126418 hits, 0 misses, 0 created

Public buffer pools: Small buffers, 104 bytes (total 50, permanent 50): 49 in free list (20 min, 150 max allowed) 65061 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) Middle buffers, 600 bytes (total 31, permanent 25, peak 31 @ 00:18:40): 29 in free list (10 min, 150 max allowed) 63057 hits, 2 misses, 0 trims, 6 created 0 failures (0 no memory) Big buffers, 1536 bytes (total 50, permanent 50): 50 in free list (5 min, 150 max allowed) 38 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) VeryBig buffers, 4520 bytes (total 10, permanent 10): 10 in free list (0 min, 20 max allowed) 0 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) Large buffers, 5024 bytes (total 0, permanent 0): 0 in free list (0 min, 10 max allowed) 0 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) Huge buffers, 18024 bytes (total 0, permanent 0): 0 in free list (0 min, 4 max allowed) 0 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory)

Interface buffer pools: CD2430 I/O buffers, 1536 bytes (total 0, permanent 0): 0 in free list (0 min, 0 max allowed) 0 hits, 0 fallbacks

Header pools: Header buffers, 0 bytes (total 265, permanent 256, peak 265 @ 00:19:17): 9 in free list (10 min, 512 max allowed) 253 hits, 3 misses, 0 trims, 9 created 0 failures (0 no memory) 256 max cache size, 256 in cache 0 hits in cache, 0 misses in cache

Particle Clones: 1024 clones, 0 hits, 0 misses

Public particle pools: F/S buffers, 256 bytes (total 384, permanent 384): 128 in free list (128 min, 1024 max allowed) 256 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 256 max cache size, 256 in cache 0 hits in cache, 0 misses in cache Normal buffers, 1548 bytes (total 512, permanent 512): 384 in free list (128 min, 1024 max allowed) 128 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 128 max cache size, 128 in cache 0 hits in cache, 0 misses in cache

Private particle pools: Ethernet0/0 buffers, 1536 bytes (total 96, permanent 96): 0 in free list (0 min, 96 max allowed) 96 hits, 0 fallbacks 96 max cache size, 64 in cache 64644 hits in cache, 0 misses in cache Serial0/0 buffers, 1548 bytes (total 14, permanent 14): 0 in free list (0 min, 14 max allowed) 14 hits, 0 fallbacks 14 max cache size, 7 in cache 7 hits in cache, 0 misses in cache TokenRing0/0 buffers, 1548 bytes (total 64, permanent 64): 0 in free list (0 min, 64 max allowed) 64 hits, 0 fallbacks 64 max cache size, 64 in cache 0 hits in cache, 0 misses in cache 4 buffer threshold, 0 threshold transitions

Router#sho diag Slot 0: Combo 1E, 1R, 2W Port adapter, 4 ports Port adapter is analyzed Port adapter insertion time unknown EEPROM contents at hardware discovery: Hardware revision 1.0 Board revision G0 Serial number 25155766 Part number 800-01223-03 FRU Part Number: NM-1E1R2W=

Test history 0x0 RMA number 00-00-00 EEPROM format version 1 EEPROM contents (hex): 0x00: 01 1F 01 00 01 7F D8 B6 50 04 C7 03 00 00 00 00 0x10: 80 00 00 00 01 02 05 17 FF FF FF FF FF FF FF FF

WIC Slot 0: Serial 1T WAN daughter card Hardware revision 1.0 Board revision H0 Serial number 14964054 Part number 800-01514-01 FRU Part Number WIC-1T=

Test history 0x0 RMA number 00-00-00 Connector type Wan Module EEPROM format version 1 EEPROM contents (hex): 0x20: 01 02 01 00 00 E4 55 56 50 05 EA 01 00 00 00 00 0x30: 88 00 00 00 99 07 07 01 FF FF FF FF FF FF FF FF

Router#sho control e0/0 Interface Ethernet0/0 Hardware is AMD Presidio ADDR: 62BDE700, FASTSEND: 60030A58, MCI_INDEX: 0 DIST ROUTE ENABLED: 0 Route Cache Flag: 11 LADRF=0x0000 0x0100 0x0000 0x0000 CSR0 =0x00000072, CSR3 =0x00001044, CSR4 =0x0000491D, CSR15 =0x00000000 CSR80 =0x0000D900, CSR114=0x00000000, CRDA =0x03D1E230, CXDA =0x03D1E5A0 BCR9 =0x00000000 (half-duplex) HW filtering information: Promiscuous Mode Disabled, PHY Addr Enabled, Broadcast Addr Enabled PHY Addr=0010.7B68.65A1, Multicast Filter=0x0000 0x0100 0x0000 0x0000 amdp2_instance=0x62BDFEA8, registers=0x3C100000, ib=0x3D1E0C0 rx ring entries=32, tx ring entries=64 rxring=0x3D1E120, rxr shadow=0x62BE011C, rx_head=17, rx_tail=0 txring=0x3D1E360, txr shadow=0x62BE01C8, tx_head=36, tx_tail=36, tx_count=0 Software MAC address filter(hash:length/addr/mask/hits): 0xC0: 0 0100.0ccc.cccc 0000.0000.0000 0 spurious_idon=0, filtered_pak=0, throttled=0, enabled=0, disabled=0 rx_framing_err=0, rx_overflow_err=0, rx_buffer_err=0 rx_bpe_err=0, rx_soft_overflow_err=0, rx_no_enp=0, rx_discard=0 tx_one_col_err=8, tx_more_col_err=10, tx_no_enp=0, tx_deferred_err=94 tx_underrun_err=0, tx_late_collision_err=0, tx_loss_carrier_err=0 tx_exc_collision_err=0, tx_buff_err=0, fatal_tx_err=0 hsrp_conf=0, need_af_check=0 tx_limited=0(64)

There are no dropped packets or buffer issues showing in the posted output.

Interesting enough the board went thru 6 design revisions (Board revision G0) - that is a bit unusual.

In article , snipped-for-privacy@rogers.com (Merv) writes: | | > | Does the router's receiving interface show any input drops ? | > | | > | post | > | show interface | > | show buffers | >

| > Below, along with a few others. This is after two copies of a ~16M image, | > stalling a total of 35 times (must be a bad day). Oh, I should also mention | > that I've tried several versions of IOS from 11.3 to 12.3 with no change. | | | There are no dropped packets or buffer issues showing in the posted | output.

Indeed. But I can infer with some certainty that a packet is being lost at some level. I'm thinking media-level corruption (though I would expect some CRC errors) or a lost interrupt. Is there any debugging I can enable that might pin it down (without swamping the machine with its display)?

| Interesting enough the board went thru 6 design revisions (Board | revision G0) - that is a bit unusual.

Well, the WIC is H0 so it has had even more revisions. :)

Dan Lanciani ddl@danlan.*com

the machine with its display)?

depending on your IOS version : debug tftp debug tftp events debug tftp packets

Unfortunately it will be a lot of output

I fyou tftp the same file several times, do the "stalls" occur at the same point each time or is it totally random ?

In article , snipped-for-privacy@rogers.com (Merv) writes: | | > Is there any debugging I can enable that might pin it down (without swamping the machine with its display)? | | depending on your IOS version : | debug tftp | debug tftp events | debug tftp packets | | Unfortunately it will be a lot of output

The "events" didn't give me anything and the "packets" gave me a lot of output, but nothing useful. The transfer runs until a stall, but then it fails completely (with a series of retransmits and no ACKs). So I have to assume that all the console output is breaking something and invalidating the test. In any case, the type of information from high- level tftp debugging could at best confirm my original inference. I was looking for something that might display lower-level errors like missed interrupts (if they are in fact detectable).

| I fyou tftp the same file several times, do the "stalls" occur at the | same point each time or is it totally random ?

Neither the position nor the total number of stalls is consistent. Once I got through a ~16M copy with only a single stall, but never with none. Lately there have been around 20 stalls per 16M. For completeness I tried different flash and DRAM modules with no change in behavior. I won't swear that the stalls are genuinely "random" but there is no obvious pattern. I suspect it may relate to other net traffic. Soon I will try it on an isolated network segment, but I don't think that will really tell me anything useful.

Dan Lanciani ddl@danlan.*com