I'll try to make this brief but it's a little complicated... and baffling.
The scenario is 3 pix firewalls (two 515Es and one 501) all with IPSec VPNs running between them. They've been this way for a couple years with no problems. Starting this week, running apps in the tunnel between firewall A (client) and firewall B (server) is causing the apps to lockup... and it's only when using this particular tunnel. Running the same app to the same server from firewall C clients is fine (in or out of tunnel). I can even run the same app from the same client behind firewall A to the same server behind firewall B outside the tunnel with no problems... this one tunnel is locking up applications for not apparent reason. Pings through the tunnel and outside the tunnel are fine and I'm not seeing any IPSec error packets. I've rebooted everything and also removed and rebuilt the tunnels with no effect. Basically I see no connectivity errors or IPSec errors to explain this.
As a side note, the app I'm testing with is SSH (but other apps display similar symptoms). Inside this one tunnel, I can sign on and basically work except I see pauses and skips when displaying files... and then eventually the session locks up. Running SSH between the same 2 boxes outside the tunnel will run flawlessly and smooth (as did the tunnel sessions prior to this week). To make this more confusing, the rest of the tunnels on all these firewalls are fine... no issues.
I'm thinking I may need a priest. Good thing it's St Patty's Day because I need a beer too.
Suggestions?