Ethernet and Flow Control

It depends on what else it is connected to.

-- glen

In general, I would use the default for that NIC. The feature is pretty much useless .In much the same way as an ICMP source quench, a PAUSE frame really does not give enough information to do anything good with in a working network, and can easily go to the wrong device and cause more problems.

Class of Service and Quality of Service, e.g. Diff-Serv and 802.3p are really the way to go.

If push came to shove, I guess I would set generate and respond. But that is with great hesitation, since the better solution in a network with so much traffic and so little bandwidth might be to rely on proven methods like TCP window size and other congestion control methods.

If you need 802.3x PAUSE frames switch to switch, then your network probably already collapsed. Switch to switch (to me) is a don't care.

Wrolf

The general thinking is that it makes sense to enable 802.3x flow control only on "edge" ports; i.e. those ports that are connected directly to end-stations. Enabling 802.3x on an inter-switch link is usually not a good idea because of the following.

- It causes head-of-line blocking (slows traffic that isn't necessarily

causing congestion).

- It causes higher burstiness which is not desireable for certain applications such as VoIP.

- It is not sensitive to the priority of frames. The transmission of all frames is stalled for the duration of the pause on the link.

This last point is particularly significant for control protocols such as LACP (used for link aggregation), STP/RSTP/MSTP, and routing protocol packets. Very long pause durations could result in instability in the network because the control protocols may assume (incorrectly) that they have lost the adjacency.

Anoop

I think you mean 802.1p.

Anoop

(snip)

How about for a link into a multispeed switch which could get saturated with data it couldn't get out the other side?

-- glen

One can always use 802.3x but one would have to live with all of its drawbacks -- and all of them will be apparent even in this case. You will have head-of-line blocking and you will see an increase burstiness/jitter. Only the network administrator can decide if this is acceptable for their network based on the applications that they are running.

The efficient way to solve things like link-speed mismatches in the middle of the network is end-to-end flow control, but unfortunately not all protocols implement it and this is especially true for multicast applications.

Anoop

I thought I should add a few more points to this discussion.

The side effect of 802.3x slowing down control traffic (which is usually link by link for the layer 2 controls protocols) is considered a bad thing. This could be solved by doing something like per-priority pause since hopefully the network is dimensioned such that the higher priority classes never see congestion in the network.

Folks have talked about per-priority pause in IEEE 802 but it hasn't really gathered momentum because:

- The Ethernet MAC doesn't really know about priorities. Priorities are used by the higher layers so this isn't a MAC issue.

- Per-priority pause could be specified by the bridging folks but they are reluctant to do so because they don't believe that link-by-link flow control is the correct way to solve this problem (it still suffers from head-of-line blocking).

- They would rather depend on end-to-end flow control and network engineering/dimensioning with appropriate traffic prioritization.

Anoop

All of the above is correct, but perhaps the overriding factor is the belief (in the LAN community) that it is much easier and cheaper to solve problems with *bandwidth* than with network complexity. All of these flow control and QoS schemes become irrelevant if there is significant overcapacity in the network. As long as capacity is cheap, there is no need to add complex mechanisms in the network to try to squeeze out a little more performance.

-- Rich Seifert Networks and Communications Consulting 21885 Bear Creek Way (408) 395-5700 Los Gatos, CA 95033 (408) 228-0803 FAX

Send replies to: usenet at richseifert dot com

(snip)

With gigabit prices fairly low, it is hard to disagree, except for the installed base which is too much work to upgrade.

But there is always the LAN/WAN interface, and always will be. As WANs get faster LANs will still stay ahead.

-- glen

Indeed. However, as Ethernet "grows up" to serving more diverse applications, it's being found that "throwing bandwidth at the problem" can only take us so far. There is now work in IEEE 802.1 to enhance infrastructure based on Ethernet to support delivery of isochronous traffic such as real-time audio and video (IEEE P802.1AS) and for supporting storage and high-performance computing applications (still in very early stages, no project number yet).

Anoop

Well, it's taken us a lot farther than anyone every thought it would.

1000 Mb/s LANs are now cheap, commodity items; the same will be true of 10G (and perhaps higher) in the relatively near future.

Every time someone comes up with some "performance intensive" application, they claim that they need to add new mechanisms into the network to ensure the level of desired performance. Historically, by the time they design this complexity, the availability of cheap bandwidth has made the problem moot. Hence, the demise of isoEthernet, FDDI-II, ATM QoS, and a slew of similar schemes.

I used to have a bumper sticker on my car that said: "Nobody asks for QoS when they have enough bandwidth." Here in Silly-con Valley, I would get lots of horn-honks and "thumbs up" greetings for that one. (The message would would get blank stares in most other regions.)

-- Rich Seifert Networks and Communications Consulting 21885 Bear Creek Way (408) 395-5700 Los Gatos, CA 95033 (408) 228-0803 FAX

Send replies to: usenet at richseifert dot com

While I tend to agree with this observation, I also feel compelled to point out that most people have found non-standard ways to deal with the problems that arise due to lack of necessary traffic controls. I'm referring to solutions offered by companies such as Packeteer and F5 Networks which are used at the LAN-WAN interface to address the problem of speed mismatch pointed out by Glen.

Anoop

Rich Seifert wrote: (snip)

I remember driving through San Jose seeing billboards for EPROMS.

I wouldn't expect them in other reasons.

-- glen

Uh, what difficulties do you encounter in delivering such service? Real time audio and video have been successfully delivered on a 128kb channel and multiple high definition streams with Dolby Digital 5.1 audio are delivered on a 6 MHz channel every day. There isn't any "trick" required to deliver this on even 100 Mb/sec Ethernet unless there is a large volume of unrelated traffic, and 10 gig will carry a full cable TV system with a fraction of its bandwidth.

These just plain aren't "hard" problems that require special techniques.

Uh, 10 gig has about 4 times the bandwidth of the fastest SCSI channel and can carry it over a much greater distance. It's already able to support storage. As for high performance computing, if you're talking about clustering then filling a 10 gig pipe is difficult even for supercomputers.

You might be right, but for now there's a lot of people spending time and money on these problems. I don't have enough expertise in this space to argue about the problems but you might want to check out some of the work on those standards and see whether the problem they are trying to address is real.

Anoop

Anoop, I too have seen, over the past 15 years at least, this continuing interest in creating QoS mechanisms for networks. First, that was the raison d'etre of several non-Ethernet layer 2 schemes, to varying degrees. (This includes of course ATM, but also FDDI and 100VG-anylan, not to mention add-ons to Ethernet itself.)

And later, as of the mid 1990s, it became an on-going goal to introduce these knobs at either Layer 2 or Layer 3 of IP-based networks. E.g. RSVP, Intserv, Diffserv, MPLS.

But when push comes to shove, when some measure of delay or jitter control is necessary, almost always the practical solution chosen is to throw excess bandwidth at the problem. And excess bandwidth always seems to become available when it is needed.

This has been my observation. No one thought 1G Ethernet was designed to end systems. Right? Well, it's becoming rather ho-hum already.

Bert

This is what Rich alluded to as well and I really cannot argue against it. Perhaps one area is the addition of priority in 802.1p. That is used quite widely in enterprise networks to priortize VoIP traffic from the phone over data traffic from computing systems.

And that does work for most of the applications today. Apps that need anything better won't have any buyers until the network is actually capable of supporting their requirements or someone figures out a way of working around that need.

I agree that throwing bandwidth at the problems that QoS mechanisms are trying to address (especially within a LAN) is a valid solution. But some QoS mechanisms do have merit - basic prioritization in all kinds of networks, and shaping at the LAN-WAN boundary are a couple that come to mind. Anoop

Which doesn't mean that they are problems for which a solution is needed, only that people who want to work on them have gotten funding from somewhere.

Sorry, too busy watching realtime streaming video .