Hi, can someone please give some background on ethernet performance
testing and throughput measurement applications like netperf, ttcp and
if there are any limitations/disadvantages associated with testing
using these applications. I recall using netperf to test my LAN and I
had pretty good performance but haven't been able to achieve the same
using FTP or windows copy.
Secondly are these applications good for measuring throughput on a
ethernet WAN circuit.
What is the acceptable throughput on a 100Mb WAN circuit and what
would be the limitation if any factors on such a circuit? (For eg: is
Propogation delay, distance etc a consideration and are there any
Well, you have one limitation right there - netperf at least does not
include filesystem interaction which one would see with FTP.
Personally I think that is a good thing, but then I have a built-in
FTP is still "bulk transfer" like a netperf TCP_STREAM test, but it
has the filesystem interactions and so may be held-back by that. It
may also be making different socket buffer size selections via
setsockopt() than you used/got with your netperf testing.
IIRC a windows copy would be SMB/CIFS or the like. That is more a
request/response protocol (as is NFS) and so its "window" will be how
many requests can be oustanding at one time. You could in theory
mimic that with netperf by configuring with ./configure --enable-burst
and then using a TCP_RR test with request/response and burst sizes set
per the settings/limits of your SMB/CIFS stuff. IIRC I have some
stuff about that in the netperf manual you can find at:
Netperf (or even ttcp) can be used, sure.
Acceptable is in the eye of the beholder.
One of the limits to the perf of a TCP connection is:
Unless you have vista, Microsoft version of CIFS have a 64 Kbyte limit
per round trip transfer. AFAICT the transfer is request / response, so
CIFS will not stream continuously.
So even with a large TCP window, there is an upper limit on CIFS file
transfer performance given by the network path latency.
it is worth remembering that some apps do not use the full TCP window
they have available - like CIFS. Here the upper layer protocol is
providing a worse limit due to handshaking and protocol design.
>there are some ancient slides about some of the above in an "SE"
>course section I used to give:
>ftp://ftp.cup.hp.com/dist/networking/briefs/ I believe.
As Rick Jones has already discussed there are a number of
individual limitations of such a circuit. The limitation
of any particular application on any particular circuit
cannot be guessed at in advance careful analysis
The BIG surprise (well it was to me) is the effect
that the end to end propagation delay
has on the performance. For many situations it is
the *dominating* factor.
I now use iperf in preference to ttcp since it is more flexible.
In brief the maximum achievable performance for a
single TCP stream is the maximum data that
can be sent before the sender stops and waits
for an acknowledgement (lets call this the block size)
times the time for one round trip. i.e. twice the
Clearly there can be other limitations in a system such as
disk performance, cpu performance, memory swapping...
The block size can be constrained in different places
in the system. Applications often have a block size limit,
and TCP also has block size limits.
In the cases you mention
TCP - Max Receive Window size - is often a limit
read about Window Scaling too.
ftp - quite likely has its own internal limits
ttcp - has default block size settings that can be changed
window file copy - prior to Vista
is limited ABSOLUTELY to 64k blocks but seems invariably
to average 32k blocks.
AND has other limitations too which may reduce
the perfoirmance further. These are related to the way that
each connection with a server (NOT each share - across
all shares on a server) uses a SINGLE TCP connection.
This is used to update the current Explorer display as
well as to transfer files.
Vista is supposed to be able to use larger blocks
but I have not checked if it is really true. This will only
be available if you use Vista for your server (or Longhorn
if you have some pre-release version).
The windows limitation results in the MAX bandwitch say
between NY and San Francisco being about 32k / 0.1
bytes per second or about 4M bps. in the case of a file copy.
This is one reason that CIFS is rather uncommonly
used on the Internet.
When I want to test if a link is able to support
some particular bandwidth I use multiple iperf
instances each running in a different window.
Oh, I recall now, iperf has a switch to use
some larger number of sessions automagically.
I did not at first want to worry about automagic
so I did it manually.
Add sessions until the bandwidth stops
increasing as sessions are added. Then you
have the limit. - of something.
this is complex and a lot of networking people dont
To look for further information you can read about
Bandwidth Delay Product.
TCP is also by the way VERY complex internally
and its behaviour is not at all easy (for me) to understand.
Briefly, one key is that there is in addition to
the Rx window a Tx Congestion window.
This is not explained anywhere that I am aware of.
Key words to look for a "Reno" and "New Reno".
Gaak, iperf :) Seriously, offline feel free to let me know what might
need to happen in netperf to make it more palatable to you.
Let's not call it the block size. That may start to convey message
boundaries which do not exist in TCP. In TCPspeak that is the window
size. One might refer to an "effective window size" which would be
the minimum of the receiver's advertised window, the sender's computed
congestion window, the sender's SO_SNDBUF size and how much the
application is willing to put out onto the connection at one time.
Is that the feature of iperf you like - the internal-to-iperf multiple
"Networking people" should have a basic grasp of bandwidth delay product :)
I suspect there are some words about it in one of the editions of
Stevens' et al's TCP/IP Illustrated or Unix Network programming.
There are a few other congestion control algorithms out there as well.
I believe that each have papers, if not actual RFC's discussing them.
Then there is the seminal (?) TCP congestion control and avoidance
paper by Van Jacobson.
In summary - the key is that netperf seems to me to be
unix centric and I am not.
In rambling :-)
Well it's not that netperf is unpalatable as such.
I have not really used it significantly due to the lack of
availability of MS Windows binaries that look to be from
a trustworthy source.
I am pretty sure that I had a look at it quite a few years
ago, maybe on solaris.
I have always worked in MS Windows environments
and I first started to use ttcp that Microsoft shipped
as part of their IPV6 protocol stack. I was confident
that this would be free of malicious code. I dont think
that it is available now on the MS web site.
I then needed to test using multicast and I found that
I could do this with iperf. I found some executable
code that I was reasonably happy with.
If there is a reasonably safe looking windows binary for
netperf then I will for sure have another look at it.
I really, really don't want to get into installing cygwin
and compiling it. Maybe I can do it and maybe I can't
get it to go.
I am far from sure but I suspect that I have considered netpref
a few times but difficulties in locating a trustworthy looking
executable for Windows has swayed me away from it in
If you could point me to a Windows binary then I will
for sure have a look. I would MUCH prefer a native
build that did not need cygwin.
Regarding the multiple stream thing. It was not a key
driver in selecting iperf. I did not even realise that it did it
until recently. It is though very handy as I think that
multiple streams are a very reliable way of working
towards determining the true 'network' performance.
i.e. it removes the influence of a number
of potential bandwidth delay product limitations
of the system.
In the absence of a native multiple stream in windows
you can do something like
for /L %i in (1, 1, 9) do start cmd /c iperf -p200%i ...
at a guess from my dodgy memory
or "cmd /k" of you need the results to remain on
As usual a bit of a ramble - hope it makes some sense.
Indeed, while netperf does compile and run under Windows (without need
of cygwin - it will compile under the DDK but that too is a download),
I do not distribute netperf in binary form and I can see where this
makes its use under Windows more difficult.
Personally, I use ttcp (there are small MS-Win32 compiled
versions available, look for ttcpw.exe)
I can usually staturate 100 in both directions. Gig is much
harder because the PCI bus isn't fast enough. When I can't get
simultaneous saturations, then there's a problem to chase:
1) If one way is well above the other, it might be homemade cabling.
2) If UDP runs fine, but TCP does not, then the protocol
stacks probably do no have correct TCPWindow sizes
(a past weakness of MS-Windows).