Queue Drops

-------- Original Message --------

If you are using WFQ do fair-queue 256 256 256 if that doesn't help you might want to switch to FIFO and do: hold-queue 375 in, hold-queue 375 out. Let me know if that does not help.

This may well reduce or eliminate the queue drops or it may not. You should however understand their root cause first.

For example TCP is _DESIGNED_ (I can't emphaise it enough:-) to create _output_ queue drops. If you are getting _legitimate_ TCP created queue drops then if you whack up the buffers the only result is that potentially _all_ TCP traffic will be delayed by the length (in time) of the queue.

Lets see 375 * 1500 * 8 / 1,500,000 = 3 seconds (by my guestimate)

This may well have a detrimental effect on the performance of your systems.

"In both sites I am seeing a lot of input and output queue drops on the

router interfaces connected to the ISP"

Output queue drops are usualy a consequence of a full link, Input queue drops are rather a different matter.

1. Check the routers buffers

sh buff

If you have memory (and who doesn't now?) increast the permanent buffers until there are few misses.

Don't go mad, just gently ease them up until most misses go away.

1. Worry about the CPU utilisation.

1. Worry about the internal traffic hitting the router and causing it to be too busy. I have used for example IP broadcast filters on the internal interface of 'weak' routers.

access-list 100 deny any host 255.255.255.255 access-list 100 deny any my.subnet.0 0.0.0.255 ! e.g. access-list 100 permit any any

This can clearly break some stuff but is likely to be OK on an internet router.

Finally, queues are bad, very bad, sometimes.

-------- Original Message --------

That's true, however going to tweak the buffers before knowing or not if does misses are because of the interface is going to far and can have harmful effects into the router itself. For instance (although is required some times and i believe almost everybody has done it) self buffer and memory adjustment is TAC unsupported. For you to know if those misses are or not because of the queue, you can find that out by using show interface -interface- switching and show interface stats. Then you can elaborate more and have an idea whether that traffic is being lost because is being forwarded and why etc. This document is based for the 10k however is still applicable for smaller platforms.

Oh well, another fine argument with TAC, one day.

Buffer adjustments are in my experience rarely needed however I have seen a few occassions when they have been necessary.

IIRC this has been when a router has been inadequate really for the job in hand and (again IIRC) reducing buffer thrashing (allocate/free cycles) has seemed to make a difference.

I wote about this in another thread in this group recently.

The issue is:- - Buffers are needed to handle traffic - Traffic is handled at interrupt level - If a new buffer is needed at Interrupt level and there is not one available then you get a Failure and the packet is dropped. - If bursts of traffic arrive after the buffers have been freed then the router has to both handle the traffic and do buffer management.

I see no reason to burden an already busy router with a load of unnecesary buffer allocate/free cycles.

I see _absolurely no harm at all_ in setting the buffers as big as you like as long as there is enough memory for whatever else is being done. I think that the cisco buffer management software is very likely more aggressive than needed in freeing buffers given the costs and availability of memory today.

This particular issue is almost certainly in some sense due to high router CPU. I still feel that changing the queue sizes is unlikely to help. Since the OP has not responded maybe they have gone away.

Thanks for the nice link. It is also here:-

wrote in message news: snipped-for-privacy@f14g2000cwb.googlegroups.com...

I do not see the links being maxed out when the output drops happen.

Here is the output from the show buffers command on the 2811 with the multilink Frame Relay interface:

Buffer elements: 1118 in free list (500 max allowed) 1490609 hits, 0 misses, 1119 created

Public buffer pools: Small buffers, 104 bytes (total 57, permanent 50, peak 106 @ 1d11h): 52 in free list (20 min, 150 max allowed) 1240146 hits, 1669 misses, 354 trims, 361 created 497 failures (0 no memory) Middle buffers, 600 bytes (total 53, permanent 25, peak 99 @ 1d20h): 21 in free list (10 min, 150 max allowed) 247771 hits, 321 misses, 330 trims, 358 created 49 failures (0 no memory) Big buffers, 1536 bytes (total 50, permanent 50, peak 56 @ 1d20h): 50 in free list (5 min, 150 max allowed) 118601 hits, 2 misses, 6 trims, 6 created 0 failures (0 no memory) VeryBig buffers, 4520 bytes (total 10, permanent 10): 9 in free list (0 min, 100 max allowed) 9625 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) Large buffers, 5024 bytes (total 0, permanent 0): 0 in free list (0 min, 10 max allowed) 0 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) Huge buffers, 18024 bytes (total 0, permanent 0): 0 in free list (0 min, 4 max allowed) 0 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory)

Interface buffer pools: CD2430 I/O buffers, 1536 bytes (total 0, permanent 0): 0 in free list (0 min, 0 max allowed) 0 hits, 0 fallbacks IPC buffers, 4096 bytes (total 2, permanent 2): 2 in free list (1 min, 8 max allowed) 0 hits, 0 fallbacks, 0 trims, 0 created 0 failures (0 no memory)

Header pools: Header buffers, 0 bytes (total 768, permanent 768): 256 in free list (128 min, 1024 max allowed) 512 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 512 max cache size, 512 in cache 38405 hits in cache, 0 misses in cache

Particle Clones: 1024 clones, 0 hits, 0 misses

Public particle pools: F/S buffers, 256 bytes (total 768, permanent 768): 256 in free list (128 min, 1024 max allowed) 512 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 512 max cache size, 512 in cache 0 hits in cache, 0 misses in cache Normal buffers, 1548 bytes (total 768, permanent 768): 768 in free list (128 min, 1024 max allowed) 73 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory)

Private particle pools: IDS SM buffers, 240 bytes (total 128, permanent 128): 0 in free list (0 min, 128 max allowed) 128 hits, 0 fallbacks 128 max cache size, 128 in cache 0 hits in cache, 0 misses in cache FastEthernet0/0 buffers, 1536 bytes (total 192, permanent 192): 0 in free list (0 min, 192 max allowed) 192 hits, 0 fallbacks 192 max cache size, 128 in cache 1031571 hits in cache, 0 misses in cache FastEthernet0/1 buffers, 1536 bytes (total 192, permanent 192): 0 in free list (0 min, 192 max allowed) 192 hits, 0 fallbacks 192 max cache size, 128 in cache 64 hits in cache, 0 misses in cache Serial0/1/0 buffers, 1536 bytes (total 64, permanent 64): 0 in free list (0 min, 64 max allowed) 64 hits, 0 fallbacks 64 max cache size, 32 in cache 254803 hits in cache, 0 misses in cache Serial0/2/0 buffers, 1536 bytes (total 64, permanent 64): 0 in free list (0 min, 64 max allowed) 64 hits, 0 fallbacks 64 max cache size, 32 in cache 254809 hits in cache, 0 misses in cache Serial0/3/0 buffers, 1536 bytes (total 64, permanent 64): 0 in free list (0 min, 64 max allowed) 64 hits, 0 fallbacks 64 max cache size, 32 in cache 254816 hits in cache, 9 misses in cache FastEthernet0/0/0 buffers, 1548 bytes (total 128, permanent 128): 0 in free list (0 min, 128 max allowed) 128 hits, 0 fallbacks 128 max cache size, 64 in cache 117472 hits in cache, 0 misses in cache NETGX_CACHE_BUFFERS buffers, 1700 bytes (total 256, permanent 256): 0 in free list (0 min, 256 max allowed) 256 hits, 0 fallbacks 256 max cache size, 256 in cache 0 hits in cache, 0 misses in cache

Are the misses in the Public buffer pools for small, middle, and big buffers a problem?

This router has plenty of memory:

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor 455F2140 434167488 167078100 267089388 266466672

I am monitoring the router CPU utilization with MRTG and I never see the utilization go above 20%.

Since this is an Internet router in front of our firewall the only traffic hitting the router is from it's redundant pair and the outside interface of the firewall. I wouldn't think broadcasts should be an issue here.

I am monitoring the router using MRTG and Nagios via SNMP. Could that be causing an issue?

Yes, but what is strange, though is that this happens in two different sites using two different hardware platforms. The one site is running a 2811 and the other is using a 3825. In both sites the connections to other ISPs on the redundant router do not have any queue drops. The drops only seem to happen on the frame relay interfaces connected to this ISP. The routers in question are running the latest version of the 12.3T train, 12.3(14)T5.

Thanks for you help on this.

V/

As a follow up to this message it turns out the queue drops on the multilink Frame Relay interface were due to a bug in the IOS. The bug ID is CSCeg88174.

Upgrading to IOS version 12.4(3b) fixed the problem.

V/