If your PIX VPN is acting strangely, such as traffic going one direction but not the other, or if your tunnels just never get past NAT-T negotations, or if you get a debug about a malformed payload with respect to a tunnel that traffic appears to be getting through... then
Triple-check the order of the transform sets on each side. (And the order of the isakmp policies too.)
If your transform set order is not identical between the two ends, or your isakmp policies are in a different order, you can have trouble trouble trouble.
The two ends each iterate through the transform list that the *other* end offers, finding the first one on that list that the local machine has permitted as a transform set for the remote peer. Effectively, the
-remote- machine gets to choose which of the transform sets that will used. When you put the transform set into a particular order, you are not controlling what your first choice is out of what the other end supports: you are controlling what you will offer the other end and it will have to pick the first one on your list that it supports.
Because of this algorithm, the two ends can end up using different transform sets to talk to each other. This often doesn't work very well :(
[I have a suspicion that there are particular problems if one end adopts AH and the other one doesn't, but I haven't proven that to be the cause yet.]I think I posted something along these lines once before, but that was before I had encountered some of the weird symptoms I found tonight...