Ethernet Performance Testing and throughput

Have a question or want to start a discussion? Post it! No Registration Necessary.  Now with pictures!

Threaded View


Hi, can someone please give some background on ethernet performance
testing and throughput measurement applications like netperf, ttcp and
if there are any limitations/disadvantages associated with testing
using these applications. I recall using netperf to test my LAN and I
had pretty good performance but haven't been able to achieve the same
using FTP or windows copy.

Secondly are these applications good for measuring throughput on a
ethernet WAN circuit.

What is the acceptable throughput on a 100Mb WAN circuit and what
would be the limitation if any factors on such a circuit? (For eg: is
Propogation delay, distance etc a consideration and are there any
others)

Thanks


Re: Ethernet Performance Testing and throughput


Quoted text here. Click to load it

Well, you have one limitation right there - netperf at least does not
include filesystem interaction which one would see with FTP.
Personally I think that is a good thing, but then I have a built-in
bias :)

FTP is still "bulk transfer" like a netperf TCP_STREAM test, but it
has the filesystem interactions and so may be held-back by that.  It
may also be making different socket buffer size selections via
setsockopt() than you used/got with your netperf testing.

IIRC a windows copy would be SMB/CIFS or the like.  That is more a
request/response protocol (as is NFS) and so its "window" will be how
many requests can be oustanding at one time.  You could in theory
mimic that with netperf by configuring with ./configure --enable-burst
and then using a TCP_RR test with request/response and burst sizes set
per the settings/limits of your SMB/CIFS stuff.  IIRC I have some
stuff about that in the netperf manual you can find at:

http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html

Quoted text here. Click to load it

Netperf (or even ttcp) can be used, sure.

Quoted text here. Click to load it

Acceptable is in the eye of the beholder.

One of the limits to the perf of a TCP connection is:

Throughput <= Window/RTT

As your RTT increases, you need more window if you want a single
stream (TCP connection) to take full advantage of the link bandwidth.
Depending on how "clean" your WAN circuit happens to be you may have
issues with packet losses and their effect on the "effective" window
size.

rick jones

there are some ancient slides about some of the above in an "SE"
course section I used to give:
ftp://ftp.cup.hp.com/dist/networking/briefs/ I believe.

--
The computing industry isn't as much a game of "Follow The Leader" as
it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
                                                    - Rick Jones
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Re: Ethernet Performance Testing and throughput


On Fri, 3 Oct 2008 21:28:59 +0000 (UTC), Rick Jones

Quoted text here. Click to load it

Unless you have vista, Microsoft version of CIFS have a 64 Kbyte limit
per round trip transfer. AFAICT the transfer is request / response, so
CIFS will not stream continuously.

So even with a large TCP window, there is an upper limit on CIFS file
transfer performance given by the network path latency.
Quoted text here. Click to load it

it is worth remembering that some apps do not use the full TCP window
they have available - like CIFS. Here the upper layer protocol is
providing a worse limit due to handshaking and protocol design.
Quoted text here. Click to load it
--
Regards

stephen_hope@xyzworld.com - replace xyz with ntl

Re: Ethernet Performance Testing and throughput


Quoted text here. Click to load it

Hi,

As Rick Jones has already discussed there are a number of
individual limitations of such a circuit. The limitation
of any particular application on any particular circuit
cannot be guessed at in advance careful analysis
is required.

The BIG surprise (well it was to me) is the effect
that the end to end propagation delay
has on the performance. For many situations it is
the *dominating* factor.

I now use iperf in preference to ttcp since it is more flexible.

In brief the maximum achievable performance for a
single TCP stream is the maximum data that
can be sent before the sender stops and waits
for an acknowledgement (lets call this the block size)
times the time for one round trip. i.e. twice the
propagation delay.

Clearly there can be other limitations in a system such as
disk performance, cpu performance, memory swapping...

The block size can be constrained in different places
in the system. Applications often have a block size limit,
and TCP also has block size limits.

In the cases you mention

TCP  -  Max Receive Window size  - is often a limit
  read about Window Scaling too.

ftp - quite likely has its own internal limits
ttcp - has default block size settings that can be changed

window file copy - prior to Vista
  is limited ABSOLUTELY to 64k blocks but seems invariably
  to average 32k blocks.
  AND has other limitations too which may reduce
  the perfoirmance further. These are related to the way that
  each connection with a server (NOT each share - across
  all shares on a server) uses a SINGLE TCP connection.
  This is used to update the current Explorer display as
  well as to transfer files.
  Vista is supposed to be able to use larger blocks
  but I have not checked if it is really true. This will only
  be available if you use Vista for your server (or Longhorn
  if you have some pre-release version).

The windows limitation results in the MAX bandwitch say
between NY and San Francisco being about 32k / 0.1
bytes per second or about 4M bps. in the case of a file copy.
This is one reason that CIFS is rather uncommonly
used on the Internet.


When I want to test if a link is able to support
some particular bandwidth I use multiple iperf
instances each running in a different window.
Oh, I recall now, iperf has a switch to use
some larger number of sessions automagically.
I did not at first want to worry about automagic
so I did it manually.
Add sessions until the bandwidth stops
increasing as sessions are added. Then you
have the limit. - of something.

this is complex and a lot of networking people dont
understand it.

To look for further information you can read about
Bandwidth Delay Product.

TCP is also by the way VERY complex internally
and its behaviour is not at all easy (for me) to understand.
Briefly, one key is that there is in addition to
the Rx window a Tx Congestion window.
This is not explained anywhere that I am aware of.
Key words to look for a "Reno" and "New Reno".

Good luck.

Re: Ethernet Performance Testing and throughput


Bod43@hotmail.co.uk wrote:
Quoted text here. Click to load it



Gaak, iperf :) Seriously, offline feel free to let me know what might
need to happen in netperf to make it more palatable to you.

Quoted text here. Click to load it

Let's not call it the block size.  That may start to convey message
boundaries which do not exist in TCP. In TCPspeak that is the window
size.  One might refer to an "effective window size" which would be
the minimum of the receiver's advertised window, the sender's computed
congestion window, the sender's SO_SNDBUF size and how much the
application is willing to put out onto the connection at one time.


Quoted text here. Click to load it

Is that the feature of iperf you like - the internal-to-iperf multiple
thread/connection stuff?

Quoted text here. Click to load it


"Networking people" should have a basic grasp of bandwidth delay product :)

Quoted text here. Click to load it

I suspect there are some words about it in one of the editions of
Stevens' et al's TCP/IP Illustrated or Unix Network programming.

Quoted text here. Click to load it

There are a few other congestion control algorithms out there as well.
I believe that each have papers, if not actual RFC's discussing them.

Then there is the seminal (?) TCP congestion control and avoidance
paper by Van Jacobson.

rick jones
--
oxymoron n, commuter in a gas-guzzling luxury SUV with an American flag
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Re: Ethernet Performance Testing and throughput


Quoted text here. Click to load it

In summary - the key is that netperf seems to me to be
unix centric and I am not.

In rambling  :-)

Well it's not that netperf is unpalatable as such.
I have not really used it significantly due to the lack of
availability of MS Windows binaries that look to be from
a trustworthy source.

I am pretty sure that I had a look at it quite a few years
ago, maybe on solaris.

I have always worked in MS Windows environments
and I first started to use ttcp that Microsoft shipped
as part of their IPV6 protocol stack. I was confident
that this would be free of malicious code. I dont think
that it is available now on the MS web site.

I then needed to test using multicast and I found that
I could do this with iperf. I found some executable
code that I was reasonably happy with.

If there is a reasonably safe looking windows binary for
netperf then I will for sure have another look at it.

I really, really don't want to get into installing cygwin
and compiling it. Maybe I can do it and maybe I can't
get it to go.

I am far from sure but I suspect that I have considered netpref
a few times but difficulties in locating a trustworthy looking
executable for Windows has swayed me away from it in
each case.

If you could point me to a Windows binary then I will
for sure have a look. I would MUCH prefer a native
build that did not need cygwin.

Regarding the multiple stream thing. It was not a key
driver in selecting iperf. I did not even realise that it did it
until recently. It is though very handy as I think that
multiple streams are a very reliable way of working
towards determining the true 'network' performance.
i.e. it removes the influence of a number
of potential bandwidth delay product limitations
of the system.

In the absence of a native multiple stream in windows
you can do something like

for /L %i in (1, 1, 9) do start cmd /c iperf -p200%i ...

at a  guess from my dodgy memory

or "cmd /k" of you need the results to remain on
the screen

As usual a bit of a ramble - hope it makes some sense.


Re: Ethernet Performance Testing and throughput


Quoted text here. Click to load it



Indeed, while netperf does compile and run under Windows (without need
of cygwin - it will compile under the DDK but that too is a download),
I do not distribute netperf in binary form and I can see where this
makes its use under Windows more difficult.

rick jones
--
The computing industry isn't as much a game of "Follow The Leader" as
it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
                                                    - Rick Jones
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Re: Ethernet Performance Testing and throughput


Bod43@hotmail.co.uk wrote in part:
Quoted text here. Click to load it

Personally, I use ttcp  (there are small MS-Win32 compiled
versions available, look for ttcpw.exe)

I can usually staturate 100 in both directions.  Gig is much
harder because the PCI bus isn't fast enough.  When I can't get
simultaneous saturations, then there's a problem to chase:

1)  If one way is well above the other, it might be homemade cabling.
    
2)  If UDP runs fine, but TCP does not, then the protocol
    stacks probably do no have correct TCPWindow sizes
    (a past weakness of MS-Windows).


-- Robert


Site Timeline