120 Hosts Running GigE at Wire Speed Minimum Cost

On 27.02.2005 07:43 Walter Roberson wrote

Aren't you mixing "low cost" with "minimum cost"? Every set of solutions has one with minimum cost which is not necessarily low.

Arnold

In article , Arnold Nipper wrote: :Aren't you mixing "low cost" with "minimum cost"? Every set of solutions :has one with minimum cost which is not necessarily low.

When someone asks for "minumum cost" then I take it to -mean-

*minimum* cost, as in the lowest -possible- cost that meets the criteria.

Unfortunately, the criteria so far has not been entirely clear, and where it has been clear it has not been entirely believable. :(

There are interpretations of the criteria would could be satisfied for < $US3000. Of course such a solution would not allow the switches involved to be monitored, but the OP has asked for "minumum cost", not for advice on what we think would be most reasonable solution for the task. If the "minimum cost" is one that has an 85% chance of going down any given hour, but gives the required bandwdith when it is up, then that is the solution that the OP would want us to specify. The OP poster made it clear (to me, anyhow) that he has no interest in the -quality- of the network, just in the price.

Two NICs each and configure each as a bridge .

>

I can vouch for GSM7312's (and gsm7324's) doing this. Their layer 2 stuff as well - gsm712, fsm726s, etc. Do NOT attempt to push out a GHOST image over multicast from a gig host if you use these - the switch WILL melt down.

Personally, I think you are right about that! :-)

What's the symptom ? Lots of dropped packets ? Lockup ?

(I'm assuming the references to smoke in this thread are metaphorical.)

Some have Gig down, with it wired in as PCI-X. What percentage of desktops have good gigE implementations I can't answer.

I've seen in the neighorhood of 1800 Mbit/s (FDX) on a variety of PCI-X gigE implementations. The trick is to open multiple connections and use overlapped/ threading techniques to keep the pipe full.

When you do this on a gig switch with a series of "pairs", or with a fanout test with multiple clients all going into one fast server on one of the ports, each beaming data back and forth wide open, you can watch the switch melt down in a lot of cases.

This is of course the fallacy of believing snake oil like Tolly reports. I've seen 8-port gig-E switches that have passed Tolly testing start dropping link randomly in minutes under this type of test. I've also seen cheap $89 5-port gig switches run the same test, at slightly better throughput for a week solid without hiccup.

Most of the low-cost high port-count switches (24,

48) will not take kindly to you trying to run all the ports wide open simultaneously. This has nothing to do with the presence or absence of a published claim to be a non-blocking switch.

Further, those that are managed switches will have their management interfaces cease being responsive at all under this type of load.

You can achieve the same thing with multicast load on IGMP switches, they'll work for a brief period sending the stream only to subscribed ports, then suddenly start flooding the traffic to all the ports.

Apparently there isn't any money in vendors publishing REAL stress tests on switches, because far too many of them would fail.

If you define "do something useful with it" as processing it all and the writing it all to disk, then if it won't work right now, it's not far off. There are several varieties of storage controller/drive combos that can achieve r/w throughput in excess of 125MB/s. They're not cheap, and they're not on desktops typically, but it can be done.

Depending upon CPU horsepower and the quality of the network driver, it can be done. There are some systems that can handle "wire speed" both directions, I.e. FDX (2Gbps).

Odds are the link is faster than any app likely to be used already. There are some specific gig lan drivers which generate insanely high CPU loads for a given throughput, so it's not generically answerable.

IOW, he's serious about it.

Unlikely. When you start daisy chaining switches, the numbers don't stand up.

How do you "just cable 120 machines together" without switches ???

Right, start with dedicated storage controllers and lots and lots of spindles. HP SmartArray hardware is a good place to start looking.

Alacritech seems to have the best CPU load per (whatever unit of transfer you like) of the current Gigabit ethernet adapters. The last I looked, they only supported Windows platforms though, which may or may not be an issue for you. That will help keep the network I/O from getting in the way of system work being done.

You are probably correct that a large, high-end non-blocking switch is what you need *IF* everybody is sending and receiving in parallel all the time. If the workstations are randomly hitting the server, at intervals, it might not be such a problem. Odds are in such a scenario that the network will be far less than fully utilized while the disk controllers (on both the server and workstation sides) will be firewalled fairly often.

Why not build a small 30-node test bed, using a couple of 16-port Netgear gig-E switches (about $800 in hardware) and running some simulated load testing to see where the bottleneck is before buying an expensive switch only to find out that you should be spending money on storage hardware instead?

Ugh.

I would be annoyed if a Nortel Passport 8600 did this.

Thank you for your excellent post, which included enough detail on the Cisco product to tell me that (as usual) the truth is complex and we will probably need to do the grunt work to detail out the requirements. While I appreciate the need to do that, you will have to trust me that there are many organizations who think they can just spend their way out of any design problem, and they end up just not doing design.

I remember one situation where a company bought a mainframe upgrade for $1M to speed up a key application. The application continued to be slow. I did performance tuning to discover the bottleneck. What I found was that their database vendor had a blocking queue on reports against the database. Only one user at a time was allowed to run reports!! None of this was documented. It was something they only admitted when we confronted them with the data. No amount of additional CPU would have changed the processing time.

It's difficult to believe that managers who control money in large companies would spend $1M (or more) rather than spend $10K to just have someone think and research to define what problem they really need to solve. But I see exactly that all of the time. And I have given up on changing the world. The world is largely run by people who act on gut instinct, and many of those people get extremely offended when you point out that someone needs to write requirements. They usually say something dismissive like "Well that's what we have done!" or "We have excellent people and we know our problem, now are you going to help me solve it?" It does no good to point out that the chicken scratches on some chalkboard aren't requirements. Some organizations fundamentally don't understand how to write requirements, or how to analyze requirements.

If we could get a very large switch that really could do wire speed on 120 hosts at wire speed simultaneously cheaply, then you don't need to work out the details of what the actual throughput would be on individual workstations for purposes of the network design. To the extent that the network scales to X usage on all ports simultaneously, you are covered by network capacity if you actually use 25% of X. Intentional overdesign is not a bad thing if it does not substantially change cost. When I see a 10 slot Extreme BlackDiamond switch (which claims 384 Gbps backplane speed) selling for next to nothing, you at least wonder if overdesign might not cost much. After reading your post, the first thought I have is that the Extreme product probably has a dozen board-level bottlenecks that they do not disclose. I would be interested in any feedback from anyone who has tried to stress that product, particularly using its layer 3 capabilities.

As far as file servers go, the thought was that they would end up using either PCI Express bus servers with quad gigE cards, or just increase the number of servers using multiple single port cards. The number of server ports they end up needing obviously does in turn affect the number of network ports you need. And if you need to now cross connect two large switches, how you do that without buying expensive new 10 Gbps technology becomes a problem.

MOST IGMP switches will do this. It's very hard to find one that will not.

The symptom is mostly described above. Flooding all ports, combined with management ports or interfaces becoming unusable during this activity.

Usually. Although, I have seen one switch literally cooked in the sense that the normal green power LED to indicate happiness switched to red and the switch shut down. Subsequent attempts to revive it were useless. It literally died from overload.

Machine 1 to machine 2 with a crossover cable(*) [...] machine 119 to machine 120 with a crossover cable.

Agree ahead of time that it's not a useful scenario, though they will get "wire-speed". 8*)

(*) IIRC, a 1000BaseT crossover is a regular straight-thru cable, but that's a detail.

[give a switch a decent load and...]

Ok, now I'm intrigued. Which ones are known to _not_ do this? I can make a couple of guesses (cisco, extreme, hp, alphabetical order), but I haven't actually tried or anything. Thoughts? Experiences?

What do you call "low-cost" or better yet what's the cost floor at which you'd not expect to melt down?

It's mostly the "low cost" ones that smaller companies buy because they afraid of all the zeros in the price of the serious vendors.

By and large, you can tell just by looking, because they all have the same chassis, just different colored paint and logos attached. The bulk of these switches (and increasingly some of the name brand switches) are all implemented and built by a a few Taiwan companies.

Once you've found one with the problem, you can find several more just by cracking the cases open and looking at the internals when the chassis looks the same.

In general, anything made by Accton (and OEM'd to a bunch of vendors) is not worth the time it takes to plug in the power cord. You just never know what will happen with their stuff. If you've ever had access to their stuff before it is gone RTS, you'll understand, otherwise you wouldn't believe it possible. Even the stuff that does get shipped is often riddled with bugs, but in areas that they hope the majority of their customers will never see in practice.

Some of the stuff made by Delta is good, some is not. In general, they're a notch above Accton most of the time.

Don't be fooled into thinking that the name brands you know of build their own gear. Almost none of them do. Many don't even stick with a consistent vendor from model to the next.

I've even seen some of the supposedly "good" gear, like HP Procurve switches do flaky stuff, like not even handling WOL packets correctly. In general, every switch (not just vendor) is different and you have to look at each one individually to be sure.

The other aspect is that for the small businesses just using them as modern "paper cup and string" links between PCs and to browse the web, they usually will never notice the issues we're talking about, so there is no real reason for them to buy the higher end geear.

I'm not sure you can simplify it that far. I've seen the same EXACT switch sold from 2 or 3 different vendors, with price differences greater than the sales price of the lowest one.

You can't go by just brand name, as the "brands" are really nothing more than "chassis colors" in a lot of cases, with the guts all coming from the same place.

The good news is if you buy one of the big, serious switches, then you know you can at least get your salesman's attention if/when you run into trouble. Unfortunately, there is often a HUGE price difference between their stuff and 10 or 15 competing products at a much lower price point. Some of those lower cost products are decent, some are not. You have to look at each one.

So, what to do? If you plan on buying a dozen 24 port switches, pick one you think does what you need at a decent price, buy ONE of them from someplace with a reasonable return policy, and put it through a trial by fire for a few weeks with everything you can think to throw at it. If you do multicasting, then send as much of that through it as you can. If you think you'll be running the switch wide open, then do that. I've seen switches reboot due to firmware bugs when performance counters overflow internally. So send it long streams of data, for days or weeks and make sure it doesn't roll over. If it makes it, order 11 more. If not, return it and start over.

You cannot rely on third party testing and "certifications" to guarantee they work. Many of those are technically only as challenging as the vendor writing the check to the test house so they can be listed on a website with some gold star next to the model number. It's a joke, but not a funny one.