Input Drops With An Empty Input Queue

My guess would be a burst of traffic hitting some interface.

You will get drops if you are getting buffer failures and you may get drops with buffer misses (IIRC).

If you have free memory you could increase the buffers until you stop getting any drops or misses.

Either increase the permanent (which I prefer) or the min free. Do be careful that you have sufficient memory. Look at the "lowest" and the "largest".

This router (all may do this) helpfully shows the peak number of actual buffers so increasing the permanent buffers to more this value will reduce the amount of allocation and free activity.

sh buff buffers small permanent 100 buffers small min-free 50 buffers middle permanent 75 buffers middle min-free 30

sh ver uptime is 15 weeks, 1 day, 7 hours, 31 minutes

sh buff Public buffer pools: Small buffers, 104 bytes (total 100, permanent 100, peak 136 @ 3w6d): 100 in free list (50 min, 150 max allowed) 48808491 hits, 1523 misses, 463 trims, 463 created 0 failures (0 no memory) Middle buffers, 600 bytes (total 75, permanent 75, peak 105 @ 7w0d): 73 in free list (30 min, 150 max allowed) 16234811 hits, 700 misses, 160 trims, 160 created 0 failures (0 no memory) Big buffers, 1536 bytes (total 50, permanent 50, peak 62 @ 2w0d): 49 in free list (5 min, 150 max allowed) 38306730 hits, 304 misses, 64 trims, 64 created 49 failures (0 no memory) VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 46 @ 7w0d): 10 in free list (0 min, 100 max allowed) 67340 hits, 40 misses, 80 trims, 80 created 0 failures (0 no memory)

#sh mem Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor 816A6990 24064624 10776824 13287800 13020740

11456432 I/O 2D99C00 2515968 780784 1735184 1565264 1735020

sh int | inc drop Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops:

662 Input queue: 1/75/1621/0 (size/max/drops/flushes); Total output drops: 0 Output queue: 0/1000/64/0 (size/max total/threshold/drops) Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Input queue: 0/75/0/1057 (size/max/drops/flushes); Total output drops: 0 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops: 0 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

ATM0 is up, line protocol is up Dialer1 is up, line protocol is up (spoofing) Virtual-Access2 is up, line protocol is up Ethernet0 is up, line protocol is up FastEthernet1 is up, line protocol is up FastEthernet2 is down, line protocol is down FastEthernet3 is down, line protocol is down FastEthernet4 is down, line protocol is down Virtual-Access1 is up, line protocol is up Virtual-Access2 is up, line protocol is up

######## As you can see I have done some work but there is still some to go. This is a router with a single Ethernet and an ADSL.

I may rack the buffers up some more today.

I've messed with the buffers with mixed results. I didn't track very well so I'm starting from square one again. I've set my buffers to the following:

buffers small permanent 310 buffers small max-free 400 buffers small min-free 90 buffers middle permanent 230 buffers middle max-free 300 buffers middle min-free 70 buffers big permanent 100 buffers big max-free 140 buffers big min-free 30 buffers verybig permanent 25 buffers verybig max-free 40 buffers verybig min-free 10 buffers large permanent 6 buffers large max-free 10 buffers large min-free 2 buffers huge permanent 6 buffers huge max-free 10 buffers huge min-free 2

This seemed to work OK. I followed Cisco guidelines and eventually reached those numbers after a few days of adjustments. I didn't see any failures or misses with these settings but if I remember correctly, I still got slight increments in the ignored counter on the serial interface. I'm going to monitor the router with these buffer settings and the current input queue of 150 (probably going to end up bumping this into the 250 range like before if drops continue) and I'll post my results.

Thanks for the response, much appreciated.

Spencer

I'm seeing the ignored counter increase with these buffer settings. The "show buffers" output does not show incrementing failures/misses for the public buffer pools either. I was still seeing input drops with an input queue of 175 so I've bumped it to 200.

I'm not sure what is going on here...ungh.

Read the definitions of throttles and ignored.

I can't recall right now.

There at least three different ways that input packets can be lost

Throttles (Interrupt throttles) [I guess a CPU limitation] Ignores ??? Input Drops ???

Throttles may not always be listed in the sh int. Have a look at sh controllers if it's not.

If this is a real user affecting issue you may need to get your wallet out and get a beefier router. Get your suplier to supply against your requirement so that they have to fix it if it doesn't work.

Consider looking at scheduller allocate command. Also "sheduler interval" ? The CPU may be too busy with something to deal with the packets. Aim to free up the router to do a variety of tasks as opposed to keeping it's head down finishing one this before going on to another. Think of Windows 'Server mode' vs 'workstation mode' optimisations. MyComputer/Properties/Advanced/PerformanceSettings/ Advanced/ProcessorScheduling/ programs vs Background services.

Go for programs.

Reducing for example the min timeslice that non interrupt tasks get may free time for packet processing. You may of course stop the router from working at all:)

sh proc cpu CPU utilization for five seconds: 12%/6%; one minute: 11%; five minutes: 9% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

130 16719024 80474 207762 0.00% 0.17% 0.19% 0 crypto sw pk pro

The "207762" above is the average time in microseconds that this process runs for on each invocation. I suspect that this is the root cause of my packet loss on this router since I suspect that it will not be able to do anything else during this period.

It is however not affecting the users significantly so we are living with it. (bit of web browsing).

TCP retransmissions are wonderful.

I'm not seeing any throttles on the serial interface, just the ignored and occassional drop at this point. I'm still tuning the buffers, I have things working better but I'm still seeing an occassional small burst of ignored and drop errors, sometimes on average only one burst per 24 hours. I'm hoping this is just a buffer tuning/input queue tuning issue.

A little background info on the interface that has me wonder sometimes if it has something to do with cabling. When I first took over for this router, the serial interface was spewing tons of input errors and crc errors. I ended up swapping out the DS3 cable and the errors went away but then I noticed input queue flushes. I started increasing the hold-queue in value for the serial interface and the flushes went away but then the input drops started coming through with the ignored errors.

I'm going to continue tweakin the buffers and hold-queue and see where I can get. I don't think this problem is having really any impact on our operations but it would be nice to eliminate this annoyance.

Fun fun fun,

Spencer

There are few options to work around problem of CPU overload and thereby resulting in input drops:

1. Using CEF.
2. Using SPD headroom tuning.

Would be worth checking these options too. HTH

Guru Prashant

snipped-for-privacy@hotmail.com wrote:

Process

One thing I forgot.

You could consider applying access lists to block unwanted traffic. This is a good _first_step. Sorry I forgot it.

I am not sure what you might want on your serial interface however block directed IP broadcasts at least.

Says: "As mentioned above, the only packets that are sent to the input queue are the ones that are destined to the router itself."

It does not seem to say this above, it does say:

"When a packet enters the router, the router attempts to forward it at interrupt level. If a match cannot be found in an appropriate cache table, the packet is queued in the incoming interface's input queue for processing."

Very confusing.

You might consider the following which may identify the nature of the problem traffic.

Create an access list to deny known traffic other than valid inbound traffic on the serial interface. Disable - console, terminal, trap logging. Enable - buffered logging Create a large logging buffer. Set up some monitoring of the input drops so that the logging buffer can be captured when the event occurs before the buffer wraps.

Enable "deb ip pac xxx"

Clearly this may well crash the router or maybe it won't. You will have to investigate this yourself however the steps above will I believe minimise the impact on the router.

Initially leave fast switching enabled. This should mean that _only_ process switched traffic will be logged but this might do the trick.

If you don't get to see the nature of the traffic you could then try disabling fast switching.

Depends on how much you want to worry about it.

It does not sound as if you are experiencing user visible problems. Interesting though.

Thanks for the input, I'm going to follow up on those suggestions. In the meantime, since I left Friday, here is what I saw on my serial T3 interface in question:

Serial4/0 is up, line protocol is up Hardware is M1T-T3 pa MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec, reliability 255/255, txload 7/255, rxload 7/255 Encapsulation FRAME-RELAY IETF, crc 16, loopback not set Keepalive set (8 sec) Restart-Delay is 0 secs LMI enq sent 28894, LMI stat recvd 28894, LMI upd recvd 0, DTE LMI up LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0 LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE Broadcast queue 0/256, broadcasts sent/dropped 3852/0, interface broadcasts 0 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 2d16h Input queue: 0/210/49/0 (size/max/drops/flushes); Total output drops:

0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 1223000 bits/sec, 252 packets/sec 5 minute output rate 1319000 bits/sec, 258 packets/sec 114387931 packets input, 1071347993 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 0 parity 0 input errors, 0 CRC, 0 frame, 0 overrun, 201 ignored, 0 abort 119572070 packets output, 583407540 bytes, 0 underruns 0 output errors, 0 applique, 0 interface resets 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions rxLOS inactive, rxLOF inactive, rxAIS inactive txAIS inactive, rxRAI inactive, txRAI inactive

I'm no longer seeing the no buffer errors but still a slight increment in drops and ignored. I compared output from the "show buffers" command from Friday and today and there were no failures/misses/trims for the public buffer pools.

I log CPU utilization using MRTG/SiteScope and I haven't seen any utilization above 35% for the week. I'm using CEF. I think I've disabled fast switching before and it didn't work out very well, if I recall I saw tons of errors.

You mentioned the ACL and that made me think of something. I took over this router from the previous admin and I have always questioned the ACL configuration. Basically everything is pretty much allowed through the serial T3 interface THEN everything is blocked on the interface servicing our network that is accessible by the public. We only allow web/ftp type traffic to certain servers on that public network. Would it be better to have this on the serial T3 interface?

Spencer

I moved the ACL from the inside interface servicing our public network to the outside, serial T3 interface. I didn't see any drops or ignored errors since making the change. I noticed that the previous admin also allowed ICMP traffic to hit all our public servers. I switched that off and only allowed echo-replies back in so people inside our network can ping out. Hopefully the interface will continue to show the same stats.

Spencer

Well done.

Prevention is better than cure.

As you have found out it is usually best to deny traffic as early as possible.

I am not much in favour of denying all diagnostics, but it clearly depends on the circumstances. e.g. Cisco let you ping them.

I bumped into this today when looking for something else.

It has some interesting material that I have not seen before.

e.g.

show buffers input-interface serial 0/0

show buffers input-interface [interface type] [interface number] header

router#show interfaces ethernet 0/0 mac-accounting Ethernet0/0 Input(494 free) 0000.0c5d.92f9(58 ): 1 packets, 106 bytes, last: 4038ms ago 0004.c059.c060(61 ): 0 packets, 0 bytes, last: 2493135ms ago 00b0.64bc.4860(64 ): 1 packets, 106 bytes, last: 20165ms ago 0090.f2c9.cc00(103): 12 packets, 720 bytes, last: 3117ms ago Total: 14 packets, 932 bytes Output (511 free) 0090.f2c9.cc00(103): 8 packets, 504 bytes, last: 4311ms ago Total: 8 packets, 504 bytes

Could be handy one day.

Good document.

We found input/output queue drops in our Internet routers. It was because the LAN Interface was receiving too much traffic to be handled for the wan Interface (FE vs E3). We usually saw that when we had infected PC in our network trying to spread the virus.

If after permit the echo in your net you saw a dicrease of the drops, I could be that you have some infected machines sending echos to the Internet. I would check which ip address are sending pings (or echos) and how frequently.

-as

snipped-for-privacy@hotmail.com wrote:

I can say with confidence that this problem is resolved. I haven't seen any drops or ignored errors since making the changes. Thanks for the info anybod. Aservin, fortunately we don't have any rogue machines within the network. I'm logging echo-replies coming back in as I allow pings (only out from within our network) and I don't see anything strange. We were gettin hammered by all the people portscanning and what not our class-c range, when echos to our public network servers were allowed before.

Serial4/0 is up, line protocol is up Hardware is M1T-T3 pa MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec, reliability 255/255, txload 6/255, rxload 5/255 Encapsulation FRAME-RELAY IETF, crc 16, loopback not set Keepalive set (8 sec) Restart-Delay is 0 secs LMI enq sent 19909, LMI stat recvd 19909, LMI upd recvd 0, DTE LMI up LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0 LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE Broadcast queue 0/256, broadcasts sent/dropped 2654/0, interface broadcasts 0 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 1d20h Input queue: 0/210/0/0 (size/max/drops/flushes); Total output drops:

0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 928000 bits/sec, 197 packets/sec 5 minute output rate 1113000 bits/sec, 199 packets/sec 79925516 packets input, 2547962089 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 0 parity 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 85124305 packets output, 2471220331 bytes, 0 underruns 0 output errors, 0 applique, 0 interface resets 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions rxLOS inactive, rxLOF inactive, rxAIS inactive txAIS inactive, rxRAI inactive, txRAI inactive

Aaahhhh...nice and clean.

Spencer