Strangest IPSec thing...

This is one of the bugs that we found. It turns out that IPSec engine 'trusts" the router. i.e. it's just an app running on the router. So it expects the router to *not* use the packet if it's sourced from a down interface. Turns out, this doesn't happen. It will happily use the IP from a downed interface.

We saw this on the 7200's too.

You have a decent ops team if they spotted this! If I were in the office, I could give you the exact TAC case number that we filed. I

Well, my OPS team is pretty good, but on this, it was not a big trouble to spot this since IPSec never got up on backup int because other router rejected SA packets from address that it doesn't have a route to it. (of course, because interface was "down").

I have to admit that I just hoped that they wrong, but it looks like shit really happens...

Well, on this router IOS is much lower than 12.2.19 so we'll just have to upgrade it.

Some update to this. It seems that consultant that implemented this did not done prescribed formal testing of the solution. The problem was that just "some" production traffic is encryped while other (mostly internet) traffic is not. When he was testing, he used "ping" which is not encrypted and he tought everything is working just because he got pings back after bringing up the backup interface.

Nobody ever tested backup using some production traffic (that should be encrypted) so, error was not seen until main link died for the first time (few days ago).

Bug would be found before putting solution into production if testing was right. And yes, after upgrade, bug is not present anymore.

Conclusion: bugs are not that big problem if human factor doesn't make mistakes at testing the solution...