I'm considering a switch of our RHEL3 fileserver to use jumbo frames in an attempt to improve Samba and NFS performance. Our entire LAN is gigabit, and connected via switches that support jumbo frames. So I expect that switching shouldn't be a problem. But I have a few concerns:
Since our server will also need to communicate with the outside world, I need to make sure that PMTU discovery can't break. I understand that it should get back an ICMP fragmentation-needed packet if a packet encounters a link with a smaller MTU. But how reliable is this? Are there routers that don't send those back? (I'd expect it to break even the standard 1500-byte MTU if there were.) Is this something to worry about?
Also, I'm curious if anything needs to be done to get NFS to take advantage of the larger MTU. According to the RFCs, you shouldn't send an packet larger than 576 bytes without knowledge that the receiving end can handle it. In TCP, the exchange of MTU information is handled during the initial handshake. But if NFS is being sent over UDP, how would it know it can use larger packets?
Finally, I've seen the arguments against jumbo frames (the percentage increase isn't worth it) and suggestions to try other things (like change the window size or acking procedures). Any specific advice on things like that would be appreciated. Given that this is a production server, I won't want to try anything too unusual, but if there are fairly standard/simple changes then I wouldn't mind giving them a go.