Network Performance 1GB ports

The wise Personne enlightened me with:

What performance do you get when you connect the two PC's directly with a cross cable? What network cards do you have? Did you try with jumbo frames?

The performance limit doesn't have to be in the Cisco, and theoretical speeds are sometimes just that: theoretical. I would expect that you need decent PC's and decent network cards to fill 1Gb. And for real usage (where the PC has to use disk etc) I wonder if you can get anywhere near that.

Mark

Mark,

I did already this test before your post, and you are correct it doesn't come from the CIsco Switch. With a crosscable (5e) between my

2 computers (not servers) I get the same speed rate (around 50 and 60 MBytes/s) I know TCP/IP has some overhead but 52% is really a lot.

- Could be a driver issue

- I haven't tried yet Jumbo Frame (I'm not even sure I have the option in my Windows drivers)

- Netperf doesn't use disk, it is a RAM-RAM test, so no disk overhead.

The wise Personne enlightened me with:

I know it's a RAM to RAM test, that's why I said that a 'real life case' probably would be even worse. TCP/IP doesn't have to be a problem, and you can account for that in your test. Not sure what you are testing precisely with netperf, but the parameters you are using can have enough impact on your measurements. I normally use iperf, I don't know netperf, but it might have the same options. You'll see different measurements for e.g. udp vs tcp, since udp is connectionless so no problems with window sizes. If you are using tcp, play around with the window size/buffer size, and see the difference. You might play with smaller/bigger sizes, you'll see a big difference.

And I'm not entirely sure where PC hardware is nowadays. Can't tell you the max bandwidth of memory, PCI busses, etc. And you will see a big difference between using an el cheapo NIC and a topline NIC that does all kinds of offloading, because it will do stuff in hardware that your CPU doesn't have to worry about anymore.

And to stress that again: don't expect to get anywhere near Gbit speed in real life usage with standard pc hardware. It's just not realistic.

Mark

Mark, "And to stress that again: don't expect to get anywhere near Gbit speed in real life usage with standard pc hardware. It's just not realistic."

I'm not expecting to get a real 1GB connection, but I'm surprise to get only 50% of the theoretical number on a 1GB connection. When I use a 100M connection (with cross cable or with a switch) I get

88% of the 100MB bandwidth.

Thanks, Pers> The wise Personne enlightened me with:

That is because the limit is not some percentage of the full bandwidth, but an absolute number. You are not hitting the limit of bus speed, cpu speed, memory speed etc. with your 100 Mbit network.

I have done similar tests between Linux systems and got a little higher rates (600 Mbit if I remember correctly) but also not 1 Gbit. But I wasn't as surprised as you seem to be.

(I was testing the troughput of a 3com Layer-3 switch when compared to a cisco router)

Getting 500Mb/s+ throughput on a standard PC is pretty good. Now you should know why servers cost so much, all parts of the architecture are optimized for throughput.

The reason you only get 88% on a 100MB/s connection is because you can't actually use 100% of the bandwidth. The counters don't count the ethernet frame, plus there is a a required inter-frame gap between ethernet frames.

I'm not expecting to get a real 1GB connection, but I'm surprise to get only 50% of the theoretical number on a 1GB connection. When I use a 100M connection (with cross cable or with a switch) I get

88% of the 100MB bandwidth.

Thanks, Pers> The wise Personne enlightened me with:

Hi Personne,

If you have a 1Gbit interface and it negotiates a 1 Gbit link speed, then you really do have 1 Gbit CAPABILITY on that connection. HOWEVER... I work in Networking with lots of Cisco H/W and 100's of top end Servers (Dell, IBM, HP, Sun) using many different OS's and ALL machines have 1GB interfaces, and NONE of them are able to REGULARLY push more than around 800Mbit through that interface without issues. In fact most Servers rarely exceed around 700Mbit though a 1Gbit interface. This is a limitation with the design of the H/W and S/W being used in the Server, not with the Cisco H/W, as we really do see aggregate Link speeds of 9.98Mbit throughput on EACH 1Gbit interface of a MULTIPLE Gbit interfaces between Cisco Switches. The easiest way to see this is to create Channel groups that consist of Multiple Gbit interfaces.

I hope this helps............pk.

All the times I did the same test I saw one of the server's cpu hitting 100% usage, that was the limit. Providing that you have multiple cpus on both servers, maybe if you run iperf twice you can reach the adapter speed limit with the aggregate throughput. Please let me know. Bye, Tosh.

For those of you looking to push the limits of your Ethernet connections you will need to turn on a few tcp options in windows. Check out RFC 1323 and Microsoft TcpOpts1323 It is recommended to turn these features on with any link generating more than 16 mb/s operations. Cisco Routers enable this option by default. Routers only need tuned outside of the default range when using jumbo frames across them. Cisco gig links natively support up to 9k frames where fast Ethernet typically does not. Side note... Vista/ Windows 7 and 2008 Server all come with RFC1323 enabled. Linux and Unix have had these options set natively for years. Have fun pegging your links.

Thanks for all your replies

Meanwhile, at the comp.dcom.sys.cisco Job Justification Hearings, Tosh chose the tried and tested strategy of:

Try 'iperf -c ... -P n', where 'n' is the number of threads to run. The man page does say "The threading implementation is rather heinous", so who knows. A quick loopback test [ie client and server] on this quad core shows -P 3 to give the highest 'throughput' [19.2Gbps], which I suppose makes sense.

Did you try that in a real envoironment, I mean from host to host? There is something that I don't understand in that, in all my previous tests i've seen well clocked cpus jumping at 100% utilization with approx 400mbps ( megabit x sec) of unidirectional iperf traffic, 20 gig is too far from those results to make sense to me, don't you agree? Bye, Tosh.

It says he's using client and server on the same PC, which means that probably the data isn't leaving the CPU, certainly not RAM, and should be limited by the throughput of memory bus or better.

In host-to-host tests, it's usually the running of the actual network card that costs the cpu utilization, not the generating of the data or (apparently, according to this test) even the running of the IP stack.

Jasper

Meanwhile, at the comp.dcom.sys.cisco Job Justification Hearings, Tosh chose the tried and tested strategy of:

Depends what is using the CPU. Could be iperf, could be the NIC driver, could be the IP stack on the OS.

Iperf is always there, the ip stack should be involved in a loopback operation, maybe the nic drivers are the cpu eaters? But thats not the point, I'm curious to know if in a real environment using more iperf threads can help to saturate the adapter bandwidth. One day I'll give it a try. Bye, Tosh.