Failover from primary router to secondary router with HSRP

Okay, I've got a fun one here. I have several remote sites that are on a full-mesh fiber network. I am just being handed trunked VLANs from the service provider. All of the remote sites have their WAN interface on the same VLAN, along with two head-end sites. I have 3560s at the remote sites and 6500s with Sup 720s at the head-end sites. I'm running OSPF over this WAN and traffic is balanced between the head-end sites by OSPF. This works great.

For backup, each remote site has a 2611 router hooked up to a DSL modem. The 2611 is set up to run a VPN back to a 3000 series concentrator I have back at the head-end. The DSL bandwidth is much smaller than my normal WAN bandwidth, so I limit the traffic allowed over the VPN to only the most necessary business traffic. If they fail over they won't have full functionality, but they will be able to perform basic business functions.

In order to have traffic fail over automatically, I am running HSRP between the 3560 and the 2611 at each remote site. The virtual IP is the default gateway for the remote site's LAN. I set the priority on the 3560 to 105 and leave the priority on the 2611 at the default of

100. I monitor the WAN interface on the 3560, so when it goes down the 3560's priority drops to 95 and the 2611 takes over the gateway IP. Traffic then flows over the VPN and my concentrator and OSPF take care of the routing back in my core at the head-end.

Here's the problem: This works great when I totally lose my connection from the service provider. The interface on the 3560 goes down and failover occurs as expected. However, when the service provider has upstream problems things don't fail over because the local link never goes down, so the 3560 becomes a black hole and traffic never moves to the VPN over DSL.

Is there a way to make this work without additional hardware? I know I can run GRE tunnels back over the IPSec and do OSPF over them, but the 3000 series doesn't do GRE tunnels, at least as far as I can tell. Are there any non-GRE solutions?

I wish I had $1 for every bogus redundant setup using HSRP for WAN fail over. As you noticed, it doesn't work except under very limited conditions (which, unfortunately for unsuspecting users, happens to be the test most frequently used to demonstrate that the new configuration works).

You need to run a routing protocol across the WAN link. You do not need to run a routing protocol across the VPN link (although you do need to test it regularly so you have a reasonable chance of having a working link when you finally do need it).

Easiest approach: use HSRP to protect against the 3560 failing (the role HSRP is designed to cover). Use a routing protocol between the

3560 and HQ to detect when the WAN link fails. Use a floating static route to send traffic to the 2611 when the WAN link fails. Don't forget to do the equivalent at the HQ end so that return traffic also goes over the DSL VPN. You can leave the HSRP config as is to eliminate the extra hop when the WAN link fails hard.

Good luck and have fun!