Re: Using Two ADSL Internet Connections Simultaneously

In article , James Carlson wrote:

snipped-for-privacy@host122.r-bonomni.com (Robert Bonomi) writes: >> "Incoming" traffic is an entirely different problem. And >> load-balancing _that_ traffic cannot be done in anything approaching a >> satisfactory manner without 'help' from the 'upstream' end. > Indeed. >> And it requires that both DSL circuits terminate at the same >> 'upstream' provider. > Not necessarily. There are at least two other possibilities here, > both of which allow for connections to multiple providers: > - NAT in use, and load balancing on a per-connection basis. This > automatically balances the return traffic as well, as everyone on > the net thinks you're actually two separate independent IP nodes.

NO, it does _Not_. You cannot change the NAT translation _during_ a 'session' (a single TCP connection). And if the 'incoming' data characteristics change radically _during_ that session, the 'balance' goes out the window.

Consider a scenario where there is -one- durable connection presently in progress, which is, say, 'streaming audio' to laptop #1, and coming in over circuit #1.

Now, the over the space of a minute, other 19 laptops each initiate a web request to a trivial text-only web-page with the Windows XP SP2 update _information_ on it, including a link to a copy of the actual service pack which resides on that same server. Oh, yeah, those requests have the HTTP 'keepalive' protocol flag set. Circuit #2 is 'unused' at the moment, so -- based on traffic levels -- _all_ these HTTP sessions are going to go on circuit #2, using 'source' addresses that will cause return data to come in over that selfsame circuit #2. Which _is_ reasonable at this point, the overall traffic from retrieving the 19 copies of that text web-page is likely less than the the one-minute block of streaming-audio.

*BUT*, now each laptop decides to download the actual service pack. BAM! usage on circuit #2 goes through the roof. And circuit #1 is still loafing along at a small fraction of capacity. Yes, this is an extreme case, but it illustrates the point that there is _no_PRACTICAL_way_ to balance the incoming load without active co-operation from the 'upstream' end(s) of the circuits.

You (on the receiving end) *cannot* unilaterally (meaning "without active cooperation from the remote end[s]) change which circuit those packets are coming in over. You cannot suddenly switch the IP address the laptops are using; "keepalive" is in effect, the request goes as part of the _same_ connection, and of course, once the download request was sent, the laptop is only doing 'listen and ack'.

- You're a big company and you can afford to arrange BGP peering > with the ISPs and inject routes into the backbone.

THAT doesn't solve the "problem", either. Not even 'mostly'.

Again, the original scenario was a 'site' with a maximum of 20 machines (all laptops) at the location. In _that_ situation, "good luck" in getting a BGP announcement for a /27 (or smaller) propagated past an immediate upstream. IF _they_ will agree to accept it. To get a moderately-reliably 'forwardable' announcement, you're going to have to use "gross overkill" sized blocks. you'll have a _terrible_ time providing adequate 'justification' to get PA space for that application from an upstream. And you're way too small to qualify for your own PI space, for _this_ application alone. Now, if you're big enough that you've got a PI /16, say. *and* can afford to dedicate a couple of /24s to this inefficient (at best, approximately a _four_percent_ utilization of the address-space), you _can_ 'play games' to influence the inbound traffic. *BUT*, to re-route individual hosts, *without* changing their IP address, you are likely to have to BGP 'announce' routes for a /32. Over time, this _will_ lead to fracturing of the space, and you _will_ be announcing separate routes for most of those hosts, *individually*. *OR* you get _really_ wasteful of address-space, and _use_ only one address in each /24 -- thus needing about a /19 to support 20 laptops. With an address-space utilization of approximately 0.25%. Yeah, this would work, but I can't imagine that anyone would classify it as a "practical" solution.

We won't even go into 'what happens' to a pre-existing connection with an _active_ stream of traffic when a route 'withdrawal' (for an address presently routed via circuit #1) arrives at an intermediate router before the 'announcement' for routing via circuit @2 arrives.

Note: if you already announce an 'inferior' grade of route through circuit #2 for that address, you cannot ensure that inbound traffic will come through circuit #1 -- it _is_ "guaranteed" that "somebody" will be using a 'policy' bias in their routing that causes them to select the carrier supplying circuit #2 over that or circuit #1.

*sigh*

There are others as well that involve just living with the fact that
> you'll appear to be separate nodes on the net, and remaining > multihomed -- this is what you'd probably do if you were doing this > for (say) a web server with multiple A records.

Review the original context -- site supporting "a maximum of 20 _laptops_", wanting to load-balance the two circuits.

How many folks run servers (web, or otherwise) _with_multiple_A_records_ on laptops?

Absent any specifications of the data flow, a reasonable "first guess" is that the traffic will be mostly: web-page retrieval, e-mail reading/sending, possibly some RSS feeds, along with some other 'streaming' incoming data.

Not guaranteed, of course, but absent better data for that scenario, it is 'betting odds' that at least 95% of the total traffic is 'incoming'. Which means that the 'easy' approaches -- which are for balancing _outgoing_ traffic -- aren't of much use. Especially since -- for the aforementioned kinds of traffic -- there is _very_little_ correlation between incoming and outgoing traffic on a machine-by-machine, or even connection-by- connection basis.

Obviously, the "more you know" about the actual traffic generated, the better your chances of designing a policy that works effectively for _that_ traffic mix. Caveat: even a minor change in traffic characteristics can utterly invalidate a 'carefully tuned/optimized for one particular scenario' balancing policy.

As soon as you have any form of 'durable' connection involved, any attempt to balance things based solely on conditions at the time of connection _initiation_ is doomed to to lead to 'far less than optimal' balancing at a point later in time. 'Things change', and the circuit assignment for that already established connection cannot be modified to adapt to the fact that "the world has changed out from under it".

> *BUT* the 'standard' routing code _in_the_kernel_ of most operating >> systems does =not= support multiple equal-priority routes to the same >> destination, *with* rotating use of those routes on a per-packet >> basis. > Doing it on a per-packet basis ("round robin") is a mistake. It > causes poor performance by reordering packets and often causes trouble > with various middleboxes. Instead, you want to hash based on flow > identification, which some systems can do.

THIS depends on what your objectives are. :)

If you're interested in maximizing your link utilization, _without_ regard to impact on QOS (as it were) to the users, the performance hits due to out-of-order packet reception are "not my problem".

Packet re-ordering _may_ occur in some instances, *BUT* it is not a guaranteed problem. Furthermore, if the pair of lines are 'bonded' into a single logical circuit -- which requires cooperation from the remote end -- then this "possible" issue effectively disappears.

If the circuits go to different end-points, you do get a whole raft of other possible issues -- including, but not limited to, remote servers that are multi-homed _on_ both of the networks you are connected to. They see packets that are part of the same 'connection' arriving on different interfaces. This _can_ confuse some kinds of systems, notably 'load balancers' that exist in front of a 'farm' of "identical" servers.

Also, if the circuits go to different end-points, then you _are_ likely to have issues with 'larger than single packet' communications to "anycast" servers. They may well be routed to _different_ servers. Available evidence suggests that this would be a 'vanishingly small' issue for 'typical' _laptop_-origin traffic.