forwarding latency

I guess they don't publish such things for marketing reasons. They think that it is best not to.

I don't know if you will find these precise numbers however there are a number of organisations that do performance tests on equipment. These are often "competitive" tests where say A Vendor pays for a test and the Tester compares The Vendors kit with other kit against a test sctipt that The Vendor has generated.

The Vendor of course knows what the result is going to be, but they get they believe a marketing advantage from having the tests executed by an independent body.

Scott Bradner of Harvard university used to run such tests and I recall that someone who used to (or does?) post here does too. Scott founded the Harvard Network Device Test Lab.

Serialisation delay:- 64 1518

10M 51us 1200us 100M 5us 120us 1,000M 0.5us 12us

If you are able to get pre-sales support from Cisco and you are willing to sign a non-disclosure agreement you may get a verbal briefing on such numbers from a Cisco Systems Engineer.

"[why the change in forwarding latency depending on packet size?" Well the packet has to get transmited around inside the box and it may be subject to serialisation delay there too. e.g. IIRC the original 6500 with 32G bus had a bus 256 bits wide.

Your latency numbers look OK but they are only a small part of the picture when you have multiple hops. The serialisation delay will nearly always dominate. In a normal data network even if you are sending a small frame, if a full size frame hits the output buffer just before yours arrives that your small frame will be delayed by the serialisation delay of the full size frame.

With TCP, you can get multiple back to back frames sent by a single sender. TCP fairly aggressively probes the network to fill it up.

Good luck.

you need to think about what matters for the applications you use - they have to put up with serialisation delay, possibly at several points in the path between 2 end points.

so a 1500 byte packet is 12000 bits, or 12 uSec more delay on a 1 Gbps port.

Cat 4500s have an internal architecture with blade ASICs, link to backplane, central switch then back out.

the links from a blade to the fabric depend on the Sup type - a Sup5 has around a 6Gb/s link, so store and forward across that will add delay which is packet size dependent.

Same as for 4500 - you need to know which Sup / line card type to understand the underlying traffic path.

bandwidths are higher, so effect on delay for bigger packets is proportionally less.

1 thing to notice is that multiple hops is such that delays thru a single switch do not dominate - so a few nSec in the switch are probably irrelevant.

or - the packet serialisation delay dominates?

not seen much on latency in switches - the main reason is that the numbers should be low enough that store and forward dominates.

try

i did some tests on Cat6k - although we were mainly worrying about overall thruput and what happens under congestion conditions.

thanx for the heads-up on why such test results may be misleading

as far as i can tell, NDTL is defunct ... if you know differently, do tell

ahh, of course ... i had forgotten about this :( hmmm, given this ... i'm curious to understand how the Catalyst 4003 achieves its invariant forwarding latency, i.e. unaffected by packet size

ahh, yes, good point thanx for the input

--sk

hi stephen,

ok, i buy that -- i need to understand the underlying hardware architecture in order to spin a coherent story behind the latency i'm observing

got it, serialization delay dominates on multihop paths

thanx for the pointer ... i see that eantc reports a 12us forwarding delay in the C6K for 64 byte packets, under some sort of load ... quite a bit more than the 5us i'm seeing ... but then ... i'm not testing under load. still, same ball park, and i find that reassuring

--sk

[...]

on a vaguely related note, i'm listening to vendors (they sell Fibre Channel switches) tell me that their gear reduces network latency by employing cut-thru switching ... this, combined with low forwarding latencies (2us) means, according to their view, that networks constructed from their gear contribute little to the total latency calculation ... serialization delay is incurred roughly once, no matter how many hops

[i think they are making assumptions here ... like ... minimal load on the network ... once multiple frames are competing for an egress port, all but one of those frames gets buffered and will incur serialization delay on egress. this view also assumes that all ports are transmitting at the same speed ... i.e. any 2Gb 4Gb transitions immediately reverts the traffic to store-and-forward]

as far as i can tell, cut-thru was a flash-in-the-pan in the Ethernet world ... i recall it as Kalpana's special sauce in 1994 ... phased out by the end of the 90s

i don't have a Fibre Channel analyzer, so i'm having trouble validating these claims myself

-can anyone confirm, through personal observation, that cut-thru happens in Fibre Channel gear?

-can anyone confirm 2us forwarding delays?

-other comments?

i'm not particularly concerned about this ... seems to me that disks and OSes and applications are still delivering latencies on the orders of milliseconds ... so shaving microseconds off network latency sounds inconsequential to me. however, i'm curious as to how Fibre Channel switch manufacturers could be delivering forwarding latencies which are so much lower than those of the Ethernet switch manufacturers ... i mean ... silicon is silicon, isn't it? ;) ... and i'm also curious as to how cut-thru has survived in Fibre Channel land, whereas it hasn't in Ethernet land. [of course, these claims may be sales fluff, i.e. technically not accurate]

--sk

Just a quick comment for now.

The real issue with Cut through as I understand it is that it cannot be made to work across ports of differing speed. Well slow-to-fast is the problem. This is I believe why it was abanoned by pretty much all of the mainstream network kit makers, well Cisco anyway:-).

It may be that with fiber channel this issue is not present or is worked around. With Ethernet and arpa/Ethernet II encapsulation it is not known until the end of the frame arrives how long the frame is. FC may be different.

Without measuring the fc kit is is clearly hard to be sure what is going on. I don't know about FC at all really but remember that the 6500 for example can do a lot more than L2 forwarding of a frame. It has huge buffers to manage, does L3 forwarding, QoS, all kinds of packet re-writing, ACLs. All in hardware at line rate (lets say).

Oh lets not forget multicast and broadcast. Seems OK to me that it takes a few us to figure out what to do.

It seems to me that you need to get you Cisco Partner to wheel in a nice SE from Cisco. That used to be the only way to get a handle on the various architectures.

Some of Cisco's on-line presentations are quite nice but may be available only to partners.

I was up to speed on the 6500 32G bus but have no real idea about the 4500 or sup720.

If you are not able to convince Cisco to get a pre-sales resource in then you may be in some difficulty.

There was years ago an academic paper on the internet that seemed to describe the 8500 architecture. I suspect that the current cross-bars are not the same but it would give you an insight into the issues that need to be considered.

I will have a look for it.