concerns about jumbo frames

I'm considering a switch of our RHEL3 fileserver to use jumbo frames in an attempt to improve Samba and NFS performance. Our entire LAN is gigabit, and connected via switches that support jumbo frames. So I expect that switching shouldn't be a problem. But I have a few concerns:

Since our server will also need to communicate with the outside world, I need to make sure that PMTU discovery can't break. I understand that it should get back an ICMP fragmentation-needed packet if a packet encounters a link with a smaller MTU. But how reliable is this? Are there routers that don't send those back? (I'd expect it to break even the standard 1500-byte MTU if there were.) Is this something to worry about?

Also, I'm curious if anything needs to be done to get NFS to take advantage of the larger MTU. According to the RFCs, you shouldn't send an packet larger than 576 bytes without knowledge that the receiving end can handle it. In TCP, the exchange of MTU information is handled during the initial handshake. But if NFS is being sent over UDP, how would it know it can use larger packets?

Finally, I've seen the arguments against jumbo frames (the percentage increase isn't worth it) and suggestions to try other things (like change the window size or acking procedures). Any specific advice on things like that would be appreciated. Given that this is a production server, I won't want to try anything too unusual, but if there are fairly standard/simple changes then I wouldn't mind giving them a go.

Damian Menscher

Reply to
Damian Menscher
Loading thread data ...

All you need to do is make sure that your gateway router a) enforces a 1500 mtu on outgoing traffic and b) supports PMTU properly. Then the jumbo frames will stop at the door.

Reply to
Bruce Richardson

In article , Damian Menscher wrote: :Since our server will also need to communicate with the outside world, :I need to make sure that PMTU discovery can't break. I understand :that it should get back an ICMP fragmentation-needed packet if a :packet encounters a link with a smaller MTU. But how reliable is :this? Are there routers that don't send those back? (I'd expect it

There are, and they do.

There few routers that -cannot- send back the appropriate ICMPs, but there is an unfortunate trend to turn the ICMPs off or to filter them out.

The reasons vary, I'm sure, but one of the reasons is that people are tending to believe that their firewalls or routers should be "invisible to the outside world", so they block outbound icmp. This is of course nothing more than "security through obscurity", but over the course of several years of working in the Cisco newsgroups, I can only recall one time that someone said "Oops, I didn't think about that" and agreed to abandon the idea of "invisibility".

Sometimes the problem is just that people haven't heard about PMTU Discovery, or that it has slipped their mind. If it's just slipped their mind then they usually turn on the icmp as soon as they are reminded, but if they haven't heard about PMTU Discovery then it can be a Problem to convince them that it's a real problem -- *they* have never experienced the problem and so to them the problem must not exist or else must academic (sort of like being told that there is still a law on the books saying that all "horseless carriages" must be proceeded by a person on foot waving a red flag.")

:Is this :something to worry about?

In your situation, I'm not sure it's worth worrying about. I would expect that your border routers are likely to reject the jumbo packets before they get onto the Internet [or Internet II]. In the more general case, though, it continues to be a problem that plagues many people: ADSL providers who use PPPoE are (historically) particularily bad at ensuring that the proper icmp makes it back. It isn't uncommon for ADSL providers to simply tell all their customers that they must lower their MTU, usually to between 1360 and 1492. For example, my residential provider used to use a higher encapsulation overhead than they do now, and at that time if one didn't lower one's MTU drastically, then one could not even successfully load the provider's home page.

Reply to
Walter Roberson

You must be aware that _all_ nodes participating in this LAN will have to be configured for (the same) jumbo frame MTU.

Yes, it is something to worry about. And the worries are not under your control - it's routers, loadbalancers, and firewalls in remote places, e.g. in front of

formatting link

The "normal" 1500 byte ethernet MTU almost always works in those icmp-broken cases. Thus, the easiest way to circumvent problems, is to have a second ethernet link on all machines where you need to communicate with the outside world, running that link with the ethernet standard MTU. If you can't easily put a second interface on the machines, maybe VLAN trunking would work, with one (internal) VLAN configured for jumbo frames, and the other (external) VLAN configured with the standard MTU.

best regards Patrick

Reply to
Patrick Schaaf

I would start some measurment to assure that the current status and sort out any packet drops and other misbehavior. Also arrange a "measurment client" where scripted "typical transactions" are performed. With this as a base you might start testing if jumboframes gives anything. You should also dump the networking gears counters and monitor them during traffic. Any packet drops, buffer shortages or simular will affect your performace. Aslo look out for networking-layer wierdness ( wrong netmasks etc)

Yes, the world outside your LAN will be affected as jumboframes is a "layer-2" thingy, and not standardized in any way.

NFS will be using your jumboframes automatically. This both for nfs/UDP and nfs/TCP.

But as i read that you use linux, you might find the sources of your performace trouble here, depending on kernel version.

Reply to

and beyond that, the MSS exchange at the beginning of a TCP connection will "cover" 90+% of the situations to begin with - all those remote sites will _probably_ still be on standard MTUs and so will be sending MSS options of ~1460 bytes, which your TCP will be honoring anyway. The time you really need PMTU to work is when you connect to a remote site that is also useing jumbo frame. Then you have what has been called a "dumbell" network - larger on the ends than in the middle - precisely the situation for which PMTU Discovery was invented in the first place.

As others point-out, some folks filter the ICMPs - probably concerned about them being spoofed or something I guess. That tends to break PMTU Discovery - or at least slow things down by forcing extra retransmissions while TCP's walk their PTMU tables.

rick jones

Reply to
Rick Jones Forums website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.