I finally managed to implement a Site-to-Site tunnel using IPSEC between ISA back-to-back on one site and and a PIX on the other.
When testing I noticed that it takes some time to establish the connection. Debug showed the following message several times during negotiating: "ISAKMP: reserved not zero on payload 8!" "ISAKMP: malformed payload"
This message comes up serveral times and then finally the connection starts working. Cisco stated that this message means that the shared key does not match however, I cheked this (of course) and still the message comes up. Both in the end the tunnel comes up and traffic is allowed and works.
The problem here is the relative long time needed to establish the tunnel causes time-out problems on applications (RDP e.g.)
I already tried to disable PFS and also checked IKE timers etc.
Does anyone know the solution for this.