concerns about jumbo frames

I'm considering a switch of our RHEL3 fileserver to use jumbo frames
in an attempt to improve Samba and NFS performance. Our entire LAN
is gigabit, and connected via switches that support jumbo frames. So
I expect that switching shouldn't be a problem. But I have a few
Since our server will also need to communicate with the outside world,
I need to make sure that PMTU discovery can't break. I understand
that it should get back an ICMP fragmentation-needed packet if a
packet encounters a link with a smaller MTU. But how reliable is
this? Are there routers that don't send those back? (I'd expect it
to break even the standard 1500-byte MTU if there were.) Is this
something to worry about?
Also, I'm curious if anything needs to be done to get NFS to take
advantage of the larger MTU. According to the RFCs, you shouldn't
send an packet larger than 576 bytes without knowledge that the
receiving end can handle it. In TCP, the exchange of MTU information
is handled during the initial handshake. But if NFS is being sent
over UDP, how would it know it can use larger packets?
Finally, I've seen the arguments against jumbo frames (the percentage
increase isn't worth it) and suggestions to try other things (like
change the window size or acking procedures). Any specific advice on
things like that would be appreciated. Given that this is a
production server, I won't want to try anything too unusual, but if
there are fairly standard/simple changes then I wouldn't mind giving
them a go.
Damian Menscher
Reply to
Damian Menscher
Loading thread data ...
All you need to do is make sure that your gateway router a) enforces a 1500 mtu on outgoing traffic and b) supports PMTU properly. Then the jumbo frames will stop at the door.
Reply to
Bruce Richardson
:Since our server will also need to communicate with the outside world, :I need to make sure that PMTU discovery can't break. I understand :that it should get back an ICMP fragmentation-needed packet if a :packet encounters a link with a smaller MTU. But how reliable is :this? Are there routers that don't send those back? (I'd expect it
There are, and they do.
There few routers that -cannot- send back the appropriate ICMPs, but there is an unfortunate trend to turn the ICMPs off or to filter them out.
The reasons vary, I'm sure, but one of the reasons is that people are tending to believe that their firewalls or routers should be "invisible to the outside world", so they block outbound icmp. This is of course nothing more than "security through obscurity", but over the course of several years of working in the Cisco newsgroups, I can only recall one time that someone said "Oops, I didn't think about that" and agreed to abandon the idea of "invisibility".
Sometimes the problem is just that people haven't heard about PMTU Discovery, or that it has slipped their mind. If it's just slipped their mind then they usually turn on the icmp as soon as they are reminded, but if they haven't heard about PMTU Discovery then it can be a Problem to convince them that it's a real problem -- *they* have never experienced the problem and so to them the problem must not exist or else must academic (sort of like being told that there is still a law on the books saying that all "horseless carriages" must be proceeded by a person on foot waving a red flag.")
:Is this :something to worry about?
In your situation, I'm not sure it's worth worrying about. I would expect that your border routers are likely to reject the jumbo packets before they get onto the Internet [or Internet II]. In the more general case, though, it continues to be a problem that plagues many people: ADSL providers who use PPPoE are (historically) particularily bad at ensuring that the proper icmp makes it back. It isn't uncommon for ADSL providers to simply tell all their customers that they must lower their MTU, usually to between 1360 and 1492. For example, my residential provider used to use a higher encapsulation overhead than they do now, and at that time if one didn't lower one's MTU drastically, then one could not even successfully load the provider's home page.
Reply to
Walter Roberson
You must be aware that _all_ nodes participating in this LAN will have to be configured for (the same) jumbo frame MTU.
Yes, it is something to worry about. And the worries are not under your control - it's routers, loadbalancers, and firewalls in remote places, e.g. in front of
formatting link

The "normal" 1500 byte ethernet MTU almost always works in those icmp-broken cases. Thus, the easiest way to circumvent problems, is to have a second ethernet link on all machines where you need to communicate with the outside world, running that link with the ethernet standard MTU. If you can't easily put a second interface on the machines, maybe VLAN trunking would work, with one (internal) VLAN configured for jumbo frames, and the other (external) VLAN configured with the standard MTU.
best regards Patrick
Reply to
Patrick Schaaf
I would start some measurment to assure that the current status and sort out any packet drops and other misbehavior. Also arrange a "measurment client" where scripted "typical transactions" are performed. With this as a base you might start testing if jumboframes gives anything. You should also dump the networking gears counters and monitor them during traffic. Any packet drops, buffer shortages or simular will affect your performace. Aslo look out for networking-layer wierdness ( wrong netmasks etc)
Yes, the world outside your LAN will be affected as jumboframes is a "layer-2" thingy, and not standardized in any way.
NFS will be using your jumboframes automatically. This both for nfs/UDP and nfs/TCP.
But as i read that you use linux, you might find the sources of your performace trouble here, depending on kernel version.
> Damian Menscher > -- > -=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=- > -=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=- > -=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=- > -=#|
formatting link
Fax:(217)333-9819 |#=- > -=#| The above opinions are not necessarily those of my employers. |#=-
Reply to
and beyond that, the MSS exchange at the beginning of a TCP connection will "cover" 90+% of the situations to begin with - all those remote sites will _probably_ still be on standard MTUs and so will be sending MSS options of ~1460 bytes, which your TCP will be honoring anyway. The time you really need PMTU to work is when you connect to a remote site that is also useing jumbo frame. Then you have what has been called a "dumbell" network - larger on the ends than in the middle - precisely the situation for which PMTU Discovery was invented in the first place.
As others point-out, some folks filter the ICMPs - probably concerned about them being spoofed or something I guess. That tends to break PMTU Discovery - or at least slow things down by forcing extra retransmissions while TCP's walk their PTMU tables.
rick jones
Reply to
Rick Jones Forums website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.