Ethernet Performance Testing and throughput

Well, you have one limitation right there - netperf at least does not include filesystem interaction which one would see with FTP. Personally I think that is a good thing, but then I have a built-in bias :)

FTP is still "bulk transfer" like a netperf TCP_STREAM test, but it has the filesystem interactions and so may be held-back by that. It may also be making different socket buffer size selections via setsockopt() than you used/got with your netperf testing.

IIRC a windows copy would be SMB/CIFS or the like. That is more a request/response protocol (as is NFS) and so its "window" will be how many requests can be oustanding at one time. You could in theory mimic that with netperf by configuring with ./configure --enable-burst and then using a TCP_RR test with request/response and burst sizes set per the settings/limits of your SMB/CIFS stuff. IIRC I have some stuff about that in the netperf manual you can find at:

Netperf (or even ttcp) can be used, sure.

Acceptable is in the eye of the beholder.

One of the limits to the perf of a TCP connection is:

Throughput

Unless you have vista, Microsoft version of CIFS have a 64 Kbyte limit per round trip transfer. AFAICT the transfer is request / response, so CIFS will not stream continuously.

So even with a large TCP window, there is an upper limit on CIFS file transfer performance given by the network path latency.

it is worth remembering that some apps do not use the full TCP window they have available - like CIFS. Here the upper layer protocol is providing a worse limit due to handshaking and protocol design.

Hi,

As Rick Jones has already discussed there are a number of individual limitations of such a circuit. The limitation of any particular application on any particular circuit cannot be guessed at in advance careful analysis is required.

The BIG surprise (well it was to me) is the effect that the end to end propagation delay has on the performance. For many situations it is the *dominating* factor.

I now use iperf in preference to ttcp since it is more flexible.

In brief the maximum achievable performance for a single TCP stream is the maximum data that can be sent before the sender stops and waits for an acknowledgement (lets call this the block size) times the time for one round trip. i.e. twice the propagation delay.

Clearly there can be other limitations in a system such as disk performance, cpu performance, memory swapping...

The block size can be constrained in different places in the system. Applications often have a block size limit, and TCP also has block size limits.

In the cases you mention

TCP - Max Receive Window size - is often a limit read about Window Scaling too.

ftp - quite likely has its own internal limits ttcp - has default block size settings that can be changed

window file copy - prior to Vista is limited ABSOLUTELY to 64k blocks but seems invariably to average 32k blocks. AND has other limitations too which may reduce the perfoirmance further. These are related to the way that each connection with a server (NOT each share - across all shares on a server) uses a SINGLE TCP connection. This is used to update the current Explorer display as well as to transfer files. Vista is supposed to be able to use larger blocks but I have not checked if it is really true. This will only be available if you use Vista for your server (or Longhorn if you have some pre-release version).

The windows limitation results in the MAX bandwitch say between NY and San Francisco being about 32k / 0.1 bytes per second or about 4M bps. in the case of a file copy. This is one reason that CIFS is rather uncommonly used on the Internet.

When I want to test if a link is able to support some particular bandwidth I use multiple iperf instances each running in a different window. Oh, I recall now, iperf has a switch to use some larger number of sessions automagically. I did not at first want to worry about automagic so I did it manually. Add sessions until the bandwidth stops increasing as sessions are added. Then you have the limit. - of something.

this is complex and a lot of networking people dont understand it.

To look for further information you can read about Bandwidth Delay Product.

TCP is also by the way VERY complex internally and its behaviour is not at all easy (for me) to understand. Briefly, one key is that there is in addition to the Rx window a Tx Congestion window. This is not explained anywhere that I am aware of. Key words to look for a "Reno" and "New Reno".

Good luck.

Gaak, iperf :) Seriously, offline feel free to let me know what might need to happen in netperf to make it more palatable to you.

Let's not call it the block size. That may start to convey message boundaries which do not exist in TCP. In TCPspeak that is the window size. One might refer to an "effective window size" which would be the minimum of the receiver's advertised window, the sender's computed congestion window, the sender's SO_SNDBUF size and how much the application is willing to put out onto the connection at one time.

Is that the feature of iperf you like - the internal-to-iperf multiple thread/connection stuff?

"Networking people" should have a basic grasp of bandwidth delay product :)

I suspect there are some words about it in one of the editions of Stevens' et al's TCP/IP Illustrated or Unix Network programming.

There are a few other congestion control algorithms out there as well. I believe that each have papers, if not actual RFC's discussing them.

Then there is the seminal (?) TCP congestion control and avoidance paper by Van Jacobson.

rick jones

In summary - the key is that netperf seems to me to be unix centric and I am not.

In rambling :-)

Well it's not that netperf is unpalatable as such. I have not really used it significantly due to the lack of availability of MS Windows binaries that look to be from a trustworthy source.

I am pretty sure that I had a look at it quite a few years ago, maybe on solaris.

I have always worked in MS Windows environments and I first started to use ttcp that Microsoft shipped as part of their IPV6 protocol stack. I was confident that this would be free of malicious code. I dont think that it is available now on the MS web site.

I then needed to test using multicast and I found that I could do this with iperf. I found some executable code that I was reasonably happy with.

If there is a reasonably safe looking windows binary for netperf then I will for sure have another look at it.

I really, really don't want to get into installing cygwin and compiling it. Maybe I can do it and maybe I can't get it to go.

I am far from sure but I suspect that I have considered netpref a few times but difficulties in locating a trustworthy looking executable for Windows has swayed me away from it in each case.

If you could point me to a Windows binary then I will for sure have a look. I would MUCH prefer a native build that did not need cygwin.

Regarding the multiple stream thing. It was not a key driver in selecting iperf. I did not even realise that it did it until recently. It is though very handy as I think that multiple streams are a very reliable way of working towards determining the true 'network' performance. i.e. it removes the influence of a number of potential bandwidth delay product limitations of the system.

In the absence of a native multiple stream in windows you can do something like

for /L %i in (1, 1, 9) do start cmd /c iperf -p200%i ...

at a guess from my dodgy memory

or "cmd /k" of you need the results to remain on the screen

As usual a bit of a ramble - hope it makes some sense.

Indeed, while netperf does compile and run under Windows (without need of cygwin - it will compile under the DDK but that too is a download), I do not distribute netperf in binary form and I can see where this makes its use under Windows more difficult.

rick jones

Personally, I use ttcp (there are small MS-Win32 compiled versions available, look for ttcpw.exe)

I can usually staturate 100 in both directions. Gig is much harder because the PCI bus isn't fast enough. When I can't get simultaneous saturations, then there's a problem to chase:

1) If one way is well above the other, it might be homemade cabling. 2) If UDP runs fine, but TCP does not, then the protocol stacks probably do no have correct TCPWindow sizes (a past weakness of MS-Windows).

-- Robert