6509 looks bad in traceroute

Have you tried changing the "scheduler allocate" default values? (maybe this will cause other problems though)

you may have a look also in configuration guide -> Security ->

Configuring Denial of Service Protection

John

We don't see the same problem. Here are a couple of traceroutes from Edinburgh to

in (approximately) London, the first with UDP the second with ICMP doens't respond to UDP). The first two hops are 6509/Sup720Bs running 12.2(33)SXH and 12.2(18)SXF4 respectively with no explicit control plane commands; the next three at least are Junipers and the rest are unknown. Hop 1 is busier than hop

1. Hop 3 is about 200 miles away, hop 4 about 400 miles away.

Sam

traceroute to

(212.58.253.70), 64 hops max, 40 byte packets 1 x.x.x.x 0.696 ms 0.924 ms 1.236 ms 2 x.x.x.x 0.332 ms 0.323 ms 0.401 ms 3 x.x.x.x 6.046 ms 5.912 ms 5.899 ms 4 x.x.x.x 11.058 ms 10.678 ms 10.738 ms 5 x.x.x.x 10.849 ms 10.676 ms 10.681 ms 6 x.x.x.x 10.843 ms 10.848 ms 11.134 ms 7 x.x.x.x 11.604 ms 11.547 ms 11.570 ms 8 x.x.x.x 12.446 ms 12.454 ms 12.462 ms 9 * *^C

traceroute to

(212.58.253.72), 64 hops max, 60 byte packets 1 x.x.x.x 0.671 ms 0.968 ms 1.002 ms 2 x.x.x.x 0.294 ms 0.267 ms 0.280 ms 3 x.x.x.x 6.073 ms 5.983 ms 5.957 ms 4 x.x.x.x 10.686 ms 10.673 ms 10.662 ms 5 x.x.x.x 10.693 ms 10.720 ms 10.688 ms 6 x.x.x.x 11.428 ms 10.928 ms 10.849 ms 7 x.x.x.x 11.609 ms 11.643 ms 11.573 ms 8 x.x.x.x 12.458 ms 12.469 ms 12.451 ms 9 x.x.x.x 13.740 ms 12.921 ms 12.889 ms

1.1 =A0 1.4 =A0 1.0 =A011.1 =A0 1.5 0 =A013.2 =A012.2 =A032.0 =A0 2.8 6 =A017.7 =A012.7 207.9 =A027.5 4 =A022.8 =A012.8 204.9 =A037.8 8 =A017.3 =A015.3 =A031.8 =A0 3.0 2 =A013.8 =A012.7 =A021.4 =A0 1.3 2 =A026.6 =A013.0 177.8 =A038.8 4 =A014.7 =A012.9 =A042.0 =A0 4.4

Clearly I have no idea what the platforms are however the observed behaviour of slow node ping response but OK throughput is common on the internet. I agree that the only one that matters is the throughput.

Regarding your issue - what is the CPU like on the box. I see that most traceroutes are responded to in a timely manner so if the cpu is under stress it may only be for short times.

There are various tools for troubleshooting the CPU. TAC helped us a lot with an on board packet capture that is not as far as I can see documented.

You can span the CPU traffic to a port too. This is documented.

You will have to try to convince the customers at the end of the day. I just say that the box has more important things to do, such as forwarding traffic, to bother with timely responses to pings.

This is an important but secondary function.

Keep in mind that I used mtr, not plain traceroute. In a nutshell, mtr is one traceroute after another, in waves until stopped. So the problem I'm seeing is not all the time. I have to "catch" it, and mtr does this.

If you have time and are interested, I'd love to see 50 or 100 mtr cycles through your boxes. Thanks for your time and reply.

Agreed. BGP scanner etc have been past culprits. Don't know how to make my bgp processing any 'better' though, I have to have full table on these guys :)

I think our Cisco rep also just emailed me this procedure. I might try it, although I'd rather do it on a box that doesnt have live customers (don't have that yet :( )

Agreed. I have some customers who don't believe (understand) that distinction of TO vs. THROUGH.

From mtr does not appear that any ICMP packets are being dropped on the OPS 6500

It would be good if the OP could run ping in verbose mode with say 100 pings so we can see if there is any pattern to which ping packets are delayed.

I seem to recall seeing another posting regarding ICMP latency and jitter on SUP720

Correct. I don't see drops, only delays. The drops in that output were a coincidence. I have many mtr runs I've done that have 0 drops, but delays on

6500/7600 hops.

To the 6509 or through it. I.e. are you proposing that I still ping the end host ? I dont think I'll see it on that so I think I answered my own question. A ping to the sup is the same, for all intent and purpose, as an mtr probe (ttl). I'll give this a shot and post results.

Yeah, I searched a bit and didnt come up with anything (which surprised me).

Alas mtr doesn't configure and make on Mac OS X (well, it configures but doesn't make) and I don't have the time or porting expertise to debug that. Sorry.

Sam

mtr can be compiled on OSX per the web page here

:: I built it on the Mac by: :: * Renaming occurrences of dns_open and dns_locate to mtr_dns_open and :: mtr_dns_locate to avoid clashes with existing declarations in :: libresolv. :: :: * Using :: make CC='gcc -arch ppc -arch i386' LD='gcc -arch ppc -arch ppc64 :: -arch i386' :: to build a universal binary. :: :: Many thanks to Matt Kimball, Roger Wolff and everyone else for a :: useful tool! :: :: Quentin Stafford-Fraser

Its also in darwinports.

Short answer, it's ~ every 60 seconds. So, per-minute tasks (BGP scanner etc). Don't know what I can do to make this 'better' though. :(

This is from a host directly attached (i.e. 6509 is my def gw.) I didn't -v on this because when I tried it I didnt see anything of value.

$ ping x.x.x.x | egrep -v 'time=0\\.' PING x.x.x.x (x.x.x.x) 56(84) bytes of data.

64 bytes from x.x.x.x: icmp_seq=7 ttl=255 time=171 ms 64 bytes from x.x.x.x: icmp_seq=8 ttl=255 time=14.9 ms 64 bytes from x.x.x.x: icmp_seq=9 ttl=255 time=173 ms 64 bytes from x.x.x.x: icmp_seq=10 ttl=255 time=159 ms 64 bytes from x.x.x.x: icmp_seq=71 ttl=255 time=11.7 ms 64 bytes from x.x.x.x: icmp_seq=72 ttl=255 time=29.5 ms 64 bytes from x.x.x.x: icmp_seq=73 ttl=255 time=32.8 ms 64 bytes from x.x.x.x: icmp_seq=74 ttl=255 time=134 ms 64 bytes from x.x.x.x: icmp_seq=135 ttl=255 time=223 ms 64 bytes from x.x.x.x: icmp_seq=136 ttl=255 time=32.2 ms 64 bytes from x.x.x.x: icmp_seq=137 ttl=255 time=95.0 ms 64 bytes from x.x.x.x: icmp_seq=198 ttl=255 time=106 ms 64 bytes from x.x.x.x: icmp_seq=199 ttl=255 time=6.31 ms 64 bytes from x.x.x.x: icmp_seq=200 ttl=255 time=39.0 ms 64 bytes from x.x.x.x: icmp_seq=201 ttl=255 time=132 ms 64 bytes from x.x.x.x: icmp_seq=215 ttl=255 time=15.7 ms

ping to the 6509 over the same path used for the mtr using so that the latency for each packet is display

with the details for the latency for each ping packet a pattern may be more

so for example one might see if every nth packet is delayed more significantly than others, etc

Ehh, I think I found it.

Back to this old trusty doc:

BGP Scanner

Walks the BGP table and confirms reachability of the next hops. BGP scanner also checks conditional-advertisement to determine whether or not BGP should advertise condition prefixes, performs route dampening. In an MPLS VPN environment, BGP scanner imports and exports routes into a particular VPN routing and forwarding instance (VRF).

Once a minute.

While BGP scanner runs, low priority processes need to wait a longer time to access the CPU. One low priority process controls Internet Control Message Protocol (ICMP) packets such as pings. Packets destined to or originated from the router may experience higher than expected latency since the ICMP process must wait behind BGP scanner.

Here it is during high ping times:

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 283 3728699656 22744504 163941 78.71% 9.53% 6.69% 0 BGP Scanner

I just thought they would have figured out a way to get this under control with sup720 and all the bling nowadays :( I understand there are more important things than responding to ping (like walking the BGP table), but I'd love to have customers actually admit to issues inside their own networks aside from catching our 6509s during BGP Scanner :(

The reason is reasonably simple, on the 6500/7600 as on most Cisco platforms, generating ICMP messages like TTL expired or Echo Reply are done at process level. At process level each process has to be scheduled by the IOS scheduler, so in order to generate the ICMP message the relevant process first has to be scheduled, typically there will be some latency before that happens, which is what is then seen in the time taken to generate the ICMP message.

This may be confusing to some people, but as noted elsewhere it has no real importance to the operation of the router or the network.

Shouldn't "BGP next-hop address tracking" obsolete the BGP scanner (or at least most of its work)?

This is supposed to be in 12.2(33)SXH and presumably 12.2(33)SRA and up, and enabled by default.

(Personally I never really noticed because we're not doing full routing.)

I hope so. We are unfortunately not running that new of code. Working on picking IOS that has NHT fixes, upgrading and doing lots more mtr runs.

cf,

If your ICMP generation latency spikes really do coincide with BGP Scanner runs (sounds like they do), *and* if BGP Scanner is well behaved (no idea), then configuring "process-max-time 20" may help. You should still see ICMP latency spikes, but they might be lower (like 40ms rather than 200ms.)

Aaron

There's a compiled binary there too, which I grabbed.

$ sudo mtr -r -c 50

HOST: Loss% Snt Last Avg Best Wrst StDev 1. x.x.x.x 0.0% 50 0.4 1.4 0.4 29.9 4.3 2. x.x.x.x 0.0% 50 0.5 0.8 0.3 12.0 1.7 3. x.x.x.x 0.0% 50 6.0 9.3 5.9 92.7 13.8 4. x.x.x.x 0.0% 50 11.7 12.0 10.7 42.3 4.8 5. x.x.x.x 0.0% 50 10.9 15.4 10.7 108.9 19.4 6. x.x.x.x 0.0% 50 11.3 18.3 10.8 183.5 27.0 7. x.x.x.x 0.0% 50 11.6 11.6 11.5 12.0 0.1 8. x.x.x.x 0.0% 50 12.5 12.6 12.4 16.5 0.6 9. x.x.x.x 0.0% 50 12.9 19.3 12.8 329.2 44.7 $ sudo mtr -r -c 100 HOST: Loss% Snt Last Avg Best Wrst StDev 1. x.x.x.x 0.0% 100 0.3 1.1 0.3 14.4 2.1 2. x.x.x.x 0.0% 100 2.9 1.0 0.3 22.5 2.8 3. x.x.x.x 0.0% 100 5.9 8.1 5.9 86.3 11.2 4. x.x.x.x 0.0% 100 10.7 12.8 10.7 92.5 10.9 5. x.x.x.x 0.0% 100 14.6 16.6 10.6 186.5 28.9 6. x.x.x.x 0.0% 100 10.9 16.1 10.8 161.4 18.9 7. x.x.x.x 0.0% 100 11.6 14.6 11.5 207.2 22.1 8. x.x.x.x 0.0% 100 12.6 12.6 12.4 20.4 0.8 9. x.x.x.x 0.0% 100 13.8 13.0 12.7 15.1 0.3

As before hops 1 and 2 are 6509/Sup720Bs, hops 3-5 are Junipers, the rest are unknown to me. Hop 3 is about 200 miles away, hops 4 and onwards are about 400 miles.

Long series of pings (300 seconds = 5 minutes) to the 6500s reveal hop 1 having a irregular glitches in delay. There my be something once a minute but it's not consistent and not pronounced. That box runs OSPF, RIP, BGP and PIM for IPv4 unicast and multicast. The 5-minute CPU is reported at about 25%. 'show ip route summ' shows a total of 139 networks and 330 subnets (whatever that's supposed to mean) so the routing overhead is pretty minimal.

Hop 2 has occasional higher delays but they're even less consistent than hop 1. It runs a similar mix of protocols (no RIP), ships more traffic but has fewer routes (73+98). 5 minute CPU is about 18%.

After all that, I'm with the people who say you need to educate your users... :-)

Sam