Efficiency of link aggregation

As the differently hashed data flows probably won't have common congestion control, I don't see why you wouldn't get 2x/3x the throughput - with packet loss for 1/2 or 1/3 of the individual flows - when you have enough flows flowing.

I've seen 1.8gbit/s on two gbit links with pure IP traffic. Unknown packet loss, but nobody complained. Unscientific, of course, just a memory of having seen RRD graphs some time ago.

I'd be interested in studies about TCP behaviour when link aggregation is run without the hashing, with roundrobin or better scheduling, without maintaining ordering. Given the same number of probing TCP flows, will throughput be better or worse than with order maintainance, provided the links do _not_ saturate.

best regards Patrick

First, there is no requirement that a hash be used to select a link for any particular flow. More important, the performance is more dependent on applications, and not the aggregation method. If two flows are assigned to separate links, and they can saturate their respective links, you will get double the throughput of a single link.

How you choose the links is up to the designer, and not mandated by the standard (other than the fact that a given conversation must map to a single link).

-- Rich Seifert Networks and Communications Consulting 21885 Bear Creek Way (408) 395-5700 Los Gatos, CA 95033 (408) 228-0803 FAX

Send replies to: usenet at richseifert dot com

There is no question that misordering will kill performance in most TCP implementations. Buffers cannot be delivered to the TCP client until later (out of order) segments are delivered; thus there may be problems with buffer overruns, and *possibly* additional copy operations. In any case, re-ordering is out of the "fast path" of the code; I suspect that any hardware TCP implementations take misordering and relegate it to software exception handling, which will kill performance.

-- Rich Seifert Networks and Communications Consulting 21885 Bear Creek Way (408) 395-5700 Los Gatos, CA 95033 (408) 228-0803 FAX

Send replies to: usenet at richseifert dot com

the hash algorithm vaires by device - here is a cisco ref from 1 of their docs, showing several different switches do it differently.

note that some platforms support different hash schemes, and it is down to config which is used. Also, hashing is independent at each end of a link set, so you might not even see the same results in both directions....

Note the cisco hash at least on Catalyst 3000 and 6000 seems to work best with binary size sets of links - using 3 or 5 links is going to cause major imbalances in the load.

i did some lab tests with 2 links a Cisco Cat 6509 and foundry MLX, presenting traffic from a load generator with around 100 streams.

the 6509 split traffic so the load on each link was balanced within 10 to 20%. The MLX was pretty close to ideal split, with less than 1% out of balance....

That's correct. However the most common, cost effective implementation is that of using a hash...other mechanisms would need flow tables and possibly need to maintain state, which make things more expensive.

Anoop

Thanks for the info.

Do you know if there was any packet loss or if the links were saturated when doing these tests?

Anoop

AFAIR we set up to run at 85 or 90% of the aggregate and we had any losses at all it was down near the measurement sample accuracy of the Agilent.

Note - we didnt try running at the 99% load we would use to test for close to wire speed - this was about whether aggregating links would use a useful improvement in thruput.

We were not looking for perfect balancing - on the assumption that was impractical :) Given the results that was what we got - better than 1 link, but not a full 2 Gbps.

ie - we run the tester to take lots of short samples, and the code we use just uses each "tick" to count recieved packets, so you get a bit of jitter.

Thanks again.

What I'm wondering is how close can you get to 2G without loss. My intuition says that as one increases the number of members, the maximum possible utilization of the whole trunk goes down. In other words, with 1 link, you can get 100% of that, with 2 links you get 95% of 2G, with 3 links you get 90% of 3G, etc. Of course, I'm just using numbers off the top of my head, but I was wondering if there are any studies that actually show that, or that at least discuss this issue.

Anoop

On 03.08.2008 23:17 anoop wrote

imho you can get close to 100% with n links in a LAG. I don' see why not. E.g. let flow1 take link1, flow2 link2 etc.

Arnold

i think this will depend on the load profile, and the balancing scheme.

in the worst case, you get no balancing at all (imagine using dest MAC as the hash, and sending to a router interface).

any hash scheme is intended to send a specific flow consistently down the same link, so if you need FIFO (and using Ethernet sort of implies you get that), and there is no packet re-order hardware, then AFAICT any 1 conversation can get at most the bandwidth of an individual link.

finally all the hash based schemes use statistical muxing of the flows

- so the assumption is the combination of flows and the hash gives you smoothed traffic on each link, and a balance across different links.

i have vague memories of an AT+T reseach paper a decade or more ago that showed measured flow profiles in both Internet and corporate LANs were fractal - which means none of those assumptions are all that accurate....

The golden rule is if it matters that much, you dont have enough bandwidth :)

Probably depends as much as anything else on what other bottlenecks appear when. If the aggregated links aren't sharing an I/O slot they are probably better-off. If the host software doing the aggregation isn't overly single-threaded (and that is a relative term based on CPU and link speeds involved) then it should do pretty well indeed.

rick jones