Getting inconsistent results with TTCP

Hi,

I forgot to mention that when I run similar tests at home between two machines with 10/100 NICs, I get very consistent results, regardless how long I wait between tests. This problem with inconsistency only seem to happen with these servers with GigE interfaces. Also BTW, these servers each have 4 Xeon 3.0GHz CPUs.

Jim

You could also use another OS that might possibly do a better job.

NetBSD currently have the speed record for long distance gigabit network.

Try a larger number of packets -n 2048 is the default and only transmits 16 MB. This only takes a second or three and may be within the timing accuracy of the calls ttcpw uses

-- Robert

Robert,

Thanks! Using a larger number of packets (100,000), I get much more consistent results, but at the same time, the results seem relatively low.

Between two servers, both with GigE interfaces, and connected using a fiber cross-over cable, I am seeing only about 38-40 Mbytes/sec. Both servers are identical, with 4 x Xeon 3.0 Ghz CPUs.

I have a much smaller test configuration at home, with two PCs (AMD Athlon 1.2GHz) with 10/100 NICs, going through an SMC 10/100 switch, and on this setup, I am seeing ~11 Mbytes/sec consistently

Any ideas as to why I'm not seeing much higher Mbytes/sec than the

40Mbytes/sec with the GigE setup?

Jim

Robert,

I'm pretty sure that the GigE interfaces on these servers are effectively either 64bit/66MHz PCI or PCI-X. Their GigE interfaces are "embedded", i.e., they're part of the server system board. Plus, when I was using the TTCP default of 2048 packets, I was sometimes seeing ~114MBytes/sec results.

Any ideas on this?

Thanks, Jim

Sounds just about right for PCI cards.

You can easily get wirespeed (11+MByte/s) on 10/100, but it's very hard to get more than 30 across a plain

32bit/33 MHz PCI bus. Yes, the theoretical bandwidth _is_ 133 but the burst are short and setup is long.

To get closer to wirespeed, you need 64 bit or 66 MHz cards and slots. I would think a server board should have one or two.

-- Robert

Hi,

I've been continuing to do searching, and I found another tool called "iperf":

I've tried it on my 10/100 network setup, and it works ok, giving me consistent results in the 11 Mbytes/sec or 93 Mbits/sec range. I'll have to try this on the GigE network and see how it does...

Jim

Robert,

My comments/resp>

These servers are IBM server, so I think they use the "Champion" (or something like that) chipset.

The board has 2 PCI-X expansion slots.

Ok, thanks re. the UDP comment. I'll give that a shot. As I also mentioned in my previous post, I'm going to try 'iperf'.

Re. "jumbo frames": The embedded GigE interfaces are Intel 1000 interfaces. I think, but am not sure, that I can enable jumbo frames under Network->Properties. For Windows 2003, is that all I need to do to enable jumble frames, or do I also have to do any registry editting for MTU, etc.?

Re. the OS, unfortunately a different OS is not an option, as part of what I'm trying to determine is what kind of performance we can get over the GigE under Windows 2003.

That (switch vs. no-switch performance) is one of the other "parameters" that I'm testing, i.e., I will eventually want to get two sets of performance numbers:

- Going through a GigE switch that we have, and

- Going point-to-point over a fiber cross-over cable.

Jim

Well, solder-on isn't magic. They have to be soldered onto the right bus! What is the Northbridge on those boards? Does the board have any higher speed PCI slots?

Probably timing granularity. Try UDP in case MS-Windows is still infected with defective TCP recv windows. See if there is something like jumbo frames you can enable. Or as other poster suggested, try another OS that might be able to handle the hardware better.

I presume you're not going through any switch.

-- Robert

Robert,

Yes, I'm assuming that will be the case, but with my testing, I am hoping to get some idea how much slowere the switch will be vs. crossover.

Thanks, Jim

P.S. If anyone else knows if I have to do anything besides set the "Jumbo Frame" setting in the Network properties under Windows 2003, please post! Thanks...

Then it should have the GigE wired in for full capability.

I try not to know anything about MS-Windows. I would hope that seeting jumbo frames would set the MTU, but who knows. UDP '-u' gets around the TCP issues.

Then perhaps you should look for specific tuning tips for MS-Win2003.

It will be very odd if the switch beats crossover.

-- Robert

Well, even if you cannot use another OS "in vitrio" you should exmine where the limits are, in order to put some pressure on your lucky vendor.

Brand of swith _is_ importent, so is some tunebals in them. Or is switch brand also predefined ?

Peter,

I'm not sure who the actual manufacturer of the switch is, it's an "IBM" switch, but I believe the for their servers, it's re-branded by IBM (i.e., not made by them). Regardless, that (the IBM) switch will be the only one we can use in this case (as you surmised).

Jim

Why do we even bother since everything already is decided ??

Try the UDP. It will tell you if you need to fix the TCP windows sizes.

-- Robert

Peter,

What got me started on this (wanting to measure bandwidth on the GigE connections) is that we are considering booting the servers via GigE vs. some other options. As I'm looking into this, I found somewhat low rates when doing the boot (~16Mbytes/sec), so I started looking into the underlying performance of the GigE connection itself.

Jim

Boot protocols are (often) request/response - for example TFTP. As such, they can be very latency sensitive and 16 MB/s on GigE may not be all that bad. What protocol is used to load the boot image(s)?

If you want to "simulate" TFTP or a request/response sort of situation, you might consider something like a netperf UDP_RR test with "apropriate" request and response sizes. Or TCP_RR. It will report transactions per second, but you can multiply that by the request or response size to get a bandwidth.

rick jones

Hi Robert,

I just tried UDP instead of TCP/IP, with both Iperf and Netperf, and still got ~300 Mbits/sec. Seems like that (the 300 Mbits/sec) must be about right, at least without any tweaking (which is my next step).

Jim