120 Hosts Running GigE at Wire Speed Minimum Cost

Just a comment but Consumer Reports as a matter of policy buys everything they test through the normal purchasing channels, they don't rely on the manufacturers sending them samples that they believe might be altered to improve the test results.

There are several reasons for this, IMHO. First of all, it is not easy to conduct such a review. To /thoroughly/ exercise even a fraction of the capabilities of a managed switch (and make sure that they work for each and every port as expected) can take weeks to complete, plus a lot of extra system hardware. Running wide-open gige (and 10gige is even worse) on all ports requires not only systems, but high-powered systems.

It also requires dedicated software tools to stress it (ttcp and the like are a joke for type of work). All the good tools are a) proprietary, b) expensive, c) both.

The typical trade rag magazine gets all their content from contacts in the industry, they don't have the time, money, or capability to perform this type of testing.

So, there are companies that do this type of work, but they charge a lot of money for it. Some do it really well, some just take the money and put out a pretty glossy full-color "book report" telling the world that the product will power up and the manufacturer can afford to hire them to publish it. Apart from that, it's basically worthless to the consumer.

Correct.

About 6 months ago, I was involved with several others in some fairly serious discussions about trying to start some sort of *legitimate* version of what Consumer Reports ought to be. We were talking about doing it all without accepting advertising, trying to kill every product we got ahold of, and publishing solid, repeatable and comparative performance data for switches, NICs, storage subsystems, and high-end data center/enterprise class servers and workstation equipment. Everyone would probably like to have access to that type of information, and a library of results that could be directly compared when making decisions, etc. The problem is, nobody wants to pay for anything. Guess what? That type of hardware costs a lot of money, even if you try and sell off a lot of it when you're done with evaluations. We got hung up on trying to build a business case that made sense without having to sell out, or having watered down, useless information like Consumer Reports.

We figured that there was no way to get the vendors to send us the hardware for eval, because as soon as we started publishing the truth about their products, AND the performance differences between them, nobody would send us anything anymore. Truth is to be avoided at all costs. :-)

Which creates a separate problem in that they really don't get a representative sample of the goods they buy. I'm sure they've discussed this and decided this is best for them but if they get the 1 in 1000 that's way worse or way better than average, the review is useless.

And if they buy rev 2 because where they bought is the last part of the country to get the rev 3 units, it's worse than useless.

CU does a good job if you know how to use the results. But they are far from perfect.

HP is having a 30 day return policy for up to 2 of some models of their switches just now. I need 4 for one office but am buying 2 just to run them through their paces before I jump in with both feet.

I'm looking at about $10k total. I can just imagine the sale FUD if I was looking to spend $250K or $2.5M. Almost like the old mainframe days when IBM and others (some still do) made you sign licensing agreements where you would never make public any benchmarks you might do.

I don't know that I have time (or the client's dimes) for all you suggest. To some degree we're future buying. These things will be 1/2 the size of the HP 4000s they'll replace and due to an office rearrangement we now have 4 clusters of "stuff" vs 3 in the past. If nothing else I plan to do a server to server duplication of the 40,000 word, excel, cad, photoshop, etc... files, run a backup to tape, and have 1/2 the remaining watching videos from the internet with the rest watching movies off another server. Just to see if the switches will crash. This should deal with our "normal" load for a while. Each of the things I mentioned will move data across the network and I'll do it so that one run pounds the switch to switch setup while the other stays in the switch. Then I'll move on to some of the other setups you mentioned.

It's hard to find the time to test VLANs past what we plan to do. As you noted.

:)

But their dependence on single point per model or even manufacturer means a good shopper treats CU the same. As a single point of data in the buying experience.

And let's not even get into the "political bias" they bring into their "unbiased" testing. What you test FOR and how you weight the tests is a major bias they don't discuss.

For those old folks around here, remember when the weekly rag, (ComputerWolrd?), had an ad every week showing SyncSort wiping out IBM's sort. After a few years, IBM posted some ads showing how they had bested SyncSort. Then it came out they had rigged the results. Not just a tad, but by a huge disparity in how the tests were run. Several folks at IBM got spanked over that one.

For those who don't know these references, it's from the late 70s, early

In article , David Ross wrote: :> Just a comment but Consumer Reports as a matter of policy buys everything :> they test through the normal purchasing channels,

:And if they buy rev 2 because where they bought is the last part of the :country to get the rev 3 units, it's worse than useless.

CU deliberately spreads their buying across the country.

In article , David Ross wrote: :Almost like the old mainframe days :when IBM and others (some still do) made you sign licensing agreements :where you would never make public any benchmarks you might do.

Microsoft's .NET EULA has a clause to that effect.

An excellent plan. Be sure to run all ports in parallel, not just test a few of them at a time. Obviously, for this to work best, you need stress and performance measurement software than can achieve wirespeed (and hardware fast enough to get there). To verify this, take two fast systems. Put them on any pair of ports by themselves. If you cannot achieve 125MBytes/s one way (ram to ram) between the machines, or something in the neighborhood of 200-220 MBytes/s FDX (ram to ram) between those two systems, then you don't have the right test software, the right hardware, or both. (BTW, you should be able to get the same exact numbers just using a direct connection between the two systems if you suspect the switch is a problem).

If you get that working, then try these types of tests: [ I am going to pretend like they are only 4 port switches just to make this easier to explain. Extending it to 16, 24, 48, etc. is trivial ]

Switch P1 P2 P3 P4

S C C C (S=server, C= client)

BTW, when I say client, I mean in the TCP sense, not in the "slow desktop" sense.

Put a server on P1, and have p2, p3 and p4 all send and receive (in parallel if possible) simultaneously to P1 and make sure it doesn't start clamping down as clients are added, all being funnelled into 1 port. For larger switches, they are often split logically internally, with two "half switches" that communicate through a higher speed interconnect. Example, a 24-port switch may be to 12- port switches ganged together. Usually this is done logically, so that 1-12 are on one "side", and 13-24 are on the other. Make sure that you measure throughput with all clients on the same "half" of the switch, and on the opposite "half" and you get equivalent results. You also may want to have multiple servers, say one or two on each half, with clients funneling into them from the same, or opposite sides of the switch and verify that performance doesn't drop off.

Next, set up your switch like this:

Switch P1 P2 P3 P4 Ca Cb Cc Cd

Run traffic (FDX if possible) between pairs of ports. Ca Cb, Cc Cd, etc. By doing this, if the switch is truly non-blocking, you should not see the individual pairs slow down, even with all of them running wide open.

If you care about multicast, then you want to try this as well, were you generate on one port, and subscribe from some small subset of the other ports. Verify (if it support IGMP) that *only* the ports that are subscribed are getting the traffic. Do this by having the non-subscribed ports sit idle so you can easily tell if the switch starts flooding by looking at the traffic lights.

Also, during all of the above scenarios, when the switch is under high load, attempt to manage the switch (assuming it is a managed switch) via an ethernet connection, or via the serial port. If the management interface doesn't essentially cease to respond during wide open tests, particularly multicast tests, that is rare.

If it is a stackable switch, you want to extend the above to put as much traffic across the stack links as possible.

If it supports VLANs, and all the other goodies, there are endless variations and additional items to test. The ones above are important, I have seen some less than wonderful switches actually start randomly dropping links when the traffic levels on all the ports are near wirespeed.

Also, make sure you throw in some 100mbit and 10mbit (if that's a possibility) into the mix to make sure it handles that properly.

This is by NO means an exhaustive test as described above, but it does represent a very important (and often omitted) set of test cases for a much larger full test of a switch. Many of the switches on the market today would never have made it out the door in their present form if these tests had been run on them before their RTS date.

Because they didn't want to take the risk that their next customer might find out that their hardware was slower then the next guy's.

They seldom provide a "single point per manufacturer". In most tests there are multiple models by a given manufacturer and they also have owner-survey results.

Their methodology is another story. Sometimes it's good, sometimes it's not.

What "legal loss in court to Bose is this"? If you are referring to the

Now why would they want to give Bose good reviews after they spent all that money securing the right to give them bad ones?

The interesting thing about the SyncSort episode was that ComputerWorld (?) was read by nearly EVERYONE at the time and there seemed to be a different ad each week showing a different trouncing of IBM sort by sync sort. And this went on for YEARS. Literally years. That ad series ran longer than the life span of most software these days. :)

One only has to look as far as their legal loss in court to Bose over them actually having the temerity tp post a legitimate evaluation of the horrific frequency response characteristics of a particular model of Bose speakers, and then the subsequent 100% laudatory remarks upon Bose products after that legal disaster to realize they are not worth reading.

"If you sue us, we'll give you a good review, even though your product is abysmal."

You don't have to go back that far. Remember Apple cooking the results for G5 performance by using different compiler optimization settings and hoping that nobody would verify their results? That was just a couple years ago, IIRC.

Or the unamed video card vendor who put the PC Magazine benchmark into hardware. That was pretty funny. "WOW..this card is out of this world!" "Wait...why, then, does it blow chunks in real world apps.."

That also used to be a favourite trick with mother boards. Another was "Winmarks". Someone would compare their 386 system to a 6 MHz IBM AT with a 286 CPU and find it ran (for instance) 20 times faster. They'd then claim they had a 120 MHz CPU!