Pix fail-over questions

You may wish to investigate the Cisco Press Article.

Cisco PIX: Failover Demystified

A pair of identical PIX Firewall devices must be used for failover to function.

To determine the system requirements and the steps needed to configure failover with a failover cable or LAN-based failover, refer to:

Using PIX Firewall Failover for PIX operating on code 6.x and earlier

For PIX operating on code 7.x and later, refer to:

Configuring Failover

For general information on failover, refer to:

How Failover Works on the Cisco Secure PIX Firewall

-----------------------------------------------

How to replace the primary PIX Firewall in a failover environment PIX version 6.3 and earlier in Models PIX 515 - 535.

It is important that the secondary unit is running a current version of the configuration that exists on the primary PIX Firewall. Issue the following command on the primary, if possible, to copy the current configuration to the standby unit:

pixfirewall# configure terminal pixfirewall (config t)# copy standby

The configuration will be copied over to the standby unit. Once this is completed, remove the primary PIX from the setup by shutting down the primary unit. Make sure that the secondary PIX has taken over before physically removing the primary PIX. This can done by issuing the following command on the secondary unit:

secondarypix# configure terminal

secondarypix (config t)# show failover

The output should look like the following:

pixfirewall (config)# show failover

Failover On

Cable status:Normal

Reconnect timeout 0:00:00

Poll frequency 15 seconds

failover replication http

This host:Secondary - Active

Active time:30 (sec)

Interface FailLink (172.16.31.2):Normal

Interface 4th (172.16.16.1):Normal

Interface int5 (192.168.168.1):Normal

Interface intf2 (192.168.1.1):Normal

Interface outside (209.165.200.225):Normal

Interface inside (10.1.1.4):Normal

At this point, the secondary PIX should show up as active in the above command output. Now it is possible to safely remove the primary PIX physically.

Note: If the secondary unit is not active at this point, troubleshooting will be necessary.

Once the old primary unit has been removed, you will need to physically install the new primary unit. The hardware and software must match the secondary unit. Be sure that the new primary PIX is running the same PIX operating system version and has the same hardware configuration as the secondary. If these requirement are not met, the new primary PIX will not perform properly in a failover environment.

Once the new primary PIX has been attached properly and is running, you will need the existing configuration on the new unit. This can be done through TFTP or entered manually. The entire configuration is not necessary at this time, but the interfaces must be set up properly and failover commands must be in place.

Issue the show failover command again on the secondary unit.

The above information from the previous show failover command will appear along with the following:

Other host: Secondary - Standby

Active time:0 (sec)

Interface FailLink (172.16.31.2) : Normal

This will verify that the primary PIX is properly attached using failover.

Issue the copy standby command on the secondary unit. This will copy the full configuration from the secondary PIX to the new primary PIX.

Issue the failover active command on the primary PIX so that it will resume the active role for the failover connection.

Issue the show active command on the new primary to make sure that it is identified as the primary failover device.

On the new primary PIX, issue the write memory command to save the configuration to memory.

-----------------------------------------------

How to replace the secondary PIX Firewall in a failover environment PIX version 6.3 and earlier in Models PIX 515 - 535.

Verify that the primary PIX Firewall is actually the active PIX for failover, by issuing the following command on the primary:

pixfirewall# configure terminal pixfirewall (config t)# show failover

The output should look like the following:

Failover On

Cable status:Normal

Reconnect timeout 0:00:00

Poll frequency 15 seconds

failover replication http

This host:Primary - Active

Active time:30 (sec)

Interface FailLink (172.16.31.2):Normal

Interface 4th (172.16.16.1):Normal

Interface int5 (192.168.168.1):Normal

Interface intf2 (192.168.1.1):Normal

Interface outside (209.165.200.225):Normal

Interface inside (10.1.1.4):Normal

Other host: Secondary - Standby

Active time:0 (sec)

Interface FailLink (172.16.31.1) : Normal

After you have confirmed that the primary PIX is actually the active PIX, remove the secondary PIX from the setup by shutting down the secondary unit.

Make sure that the primary PIX is still active before physically removing the secondary PIX by issuing the show failover command again.

Once the old secondary unit has been removed, you will need to physically install the new secondary unit. The hardware and software for the new secondary PIX must match the primary unit. If these requirement are not met, the new primary PIX will not perform properly in a failover environment.

Once the new secondary PIX has been attached properly and is running, you will need the existing configuration on the new unit. This can be done through TFTP or entered manually.

The entire configuration is not necessary at this time, but the interfaces must be set up properly and failover commands must be in place.

Issue the show failover command again on the secondary unit.

The above information from the previous show failover command will appear along with the following:

Other host: Secondary - Standby

Active time:0 (sec)

Interface FailLink (172.16.31.1) : Normal

This will verify that the secondary PIX is properly attached using failover.

Issue the copy standby command on the primary unit.

This will copy the full configuration from the primary PIX to the new secondary PIX.

Sincerely,

Brad Reese BradReese.Com - Cisco Network Engineer Directory

Hendersonville Road, Suite 17 Asheville, North Carolina USA 28803 USA & Canada: 877-549-2680 International: 828-277-7272 Fax: 775-254-3558 AIM: R2MGrant Website:

Thanks for the info, Brad. I found a few of the Cisco.com articles after posting my message. However that doesn't answer my first question. In fact it make it more confusing. The docs state that the hardware that you insert into a failover situation (after removing it due to failure or whatever) has to exactly match the current active Pix. If that's the case then how do you ever upgrade the code or RAM on the cluster of Pixs? If I remove a Pix 515e w/ a VAC+, 128MB of RAM and 6.3(4) I'll have to replace it with an exact model according to the docs. I can't even upgrade it to 6.3(5).

The way I see it, the only way to upgrade the code is to first break failover on the standy. Then upgrade it accordingly. Once it's ready to go I'll have to schedule downtime, first to physically disconnect the active Pix and then to physically connect the interfaces on the upgraded Pix (or shut and unshut interfaces appropriately). Then I can upgrade the former active Pix and put it back online. Finally I'll have to re-enable failover on everything.

This would definitely cause downtime due to the state table being lost and for when I physically move cables. Isn't there a better to accomplish this? The docs that I've read make no mention of upgrading a pair of Pixs in failover mode. I do see that v7 adds a plethora of useful failover features though.

Thanks J

That is more or less the case. Do a search of this group for articles on failover pix upgrade. You will find a variety of opinions and approaches, ranging from "just do it" (and pray) to "paranoid conservative" (from me).

Bottom line is that you WILL suffer some down time, but with adequate planning you can reduce the downtime to seconds although you will drop all translations and VPNs at least once. You will also be running for a period of time with only one PIX, so you will be exposed to the potential for significant downtime if it chooses that instant to fail.

Good luck and have fun!

Thanks for the info, Brad. I found a few of the Cisco.com articles after posting my message. However that doesn't answer my first question. In fact it make it more confusing. The docs state that the hardware that you insert into a failover situation (after removing it due to failure or whatever) has to exactly match the current active Pix. If that's the case then how do you ever upgrade the code or RAM on the cluster of Pixs? If I remove a Pix 515e w/ a VAC , 128MB of RAM and 6.3(4) I'll have to replace it with an exact model according to the docs. I can't even upgrade it to 6.3(5).

The way I see it, the only way to upgrade the code is to first break failover on the standy. Then upgrade it accordingly. Once it's ready to go I'll have to schedule downtime, first to physically disconnect the active Pix and then to physically connect the interfaces on the upgraded Pix (or shut and unshut interfaces appropriately). Then I can upgrade the former active Pix and put it back online. Finally I'll have to re-enable failover on everything.

This would definitely cause downtime due to the state table being lost and for when I physically move cables. Isn't there a better to accomplish this? The docs that I've read make no mention of upgrading a pair of Pixs in failover mode. I do see that v7 adds a plethora of useful failover features though.

Thanks J

Excellent that Dr. Vincent C. Jones, PE

Is contributing to this thread as Vince Jones is the author of the revered Addison Wesley Book:

High Availability Networking with Cisco

and the Addison Wesley article:

Configuration for Transparently Redundant Firewalls

Which was adapted from Vince's book, High Availability Networking with Cisco.

--------------------------------------------------------

How to upgrade the PIX Firewall software in a failover scenario.

The PIXes must be running version 5.1(x) or later to use this procedure because it uses the copy tftp flash command.

It was introduced in PIX 5.1(x).

--------------------------------------------------------

To upgrade PIXes in a failover set by establishing a console connection to the PIXes, perform these steps:

Step 1: Force a failover to the secondary PIX by issuing the

no failover active

command on the primary PIX, or power off the primary PIX.

This causes the secondary PIX to become active.

Step 2: Disconnect all network cables from the primary PIX ( including the failover cable ).

Step 3: Power on the primary PIX and attach a PC with a TFTP server on it. Connect the inside interface of the primary PIX to the TFTP server with a crossover cable.

Step 4: Issue the

copy tftp flash

to upgrade the primary.

Perform the upgrade procedures for the primary PIX as given in:

Upgrading Software for the Cisco Secure PIX Firewall

Step 5: Reload the primary PIX and verify the new version, license keys and features, configuration and so on.

Step 6: Power off the primary PIX.

Step 7: Re-connect all cables to the primary PIX.

Step 8: Quickly power off the secondary PIX, then immediately power on primary PIX.

Verify that the primary PIX is now passing traffic running the new version of code.

Note: Your downtime occurs while the primary is booting.

Step 9: Once the primary PIX is up, it becomes active and passes traffic.

Step 10: Repeat steps two through seven for the secondary PIX.

Step 11: Power on the secondary PIX. It becomes the standby unit.

Wait two minutes and verify that the secondary PIX is in standby mode and that all interfaces have a status of Normal.

Both PIXes are now running the upgraded version and back to normal operation.

--------------------------------------------------------

To upgrade PIXes in a failover set through Telnet and/or Secure Shell ( SSH ) session, perform these steps.

Step 1: Open separate remote Telnet sessions to both the primary and seconday PIXes.

Step 2: Perform a show failover to ensure that the primary PIX is the active PIX.

Note: If it is not, issue the failover active command from the show primary PIX to make it so.

Step 3: Enter config t mode on both PIXes.

Step 4: For recovery purposes, in case something goes wrong, copy the running config command from the primary PIX to a text file, or issue the

write net

command from the primary PIX to save the config to the TFTP server.

Step 5: Issue the write standby command, followed by the write memory command on the primary PIX to insure that the secondary PIX is up-to-date, in case something happens to the primary PIX .

Step 6: Issue the copy tftp flash command on the primary PIX and wait until it downloads the new PIX image from the TFTP server successfully.

Step 7: If the step 6 download is successful on the primary PIX, repeat step six on the secondary PIX.

Note: At this point, a new PIX image has been downloaded to both the primary and secondary PIX.

Step 8: On the primary PIX, press the Reset button on the front of the PIX.

An alternative is turn off the power and turn it back on.

Step 9: Wait five seconds, then press the Reset button on the front of the seconday PIX, or turn off the power and turn it back on.

Step 10: Wait about one minute for both PIXes to complete their resets, then perform a separate Telnet to both primary and secondary PIXs.

Step 11: Verify through the

show version

command output on both PIXes that the new PIX image has been installed successfully.

Step 12: Verify through the show failover command output on the primary PIX that it is active, and verify that the secondary is in standby mode.

Both PIXes are now running the upgraded version and back to normal operation.

--------------------------------------------------------

To upgrade PIXes in a failover set remotely, initiate two Telnet ( or SSH ) sessions to the PIXes.

Initiate one session to the primary and one session to the secondary.

Replace steps eight and nine ( see above ) with these steps:

Step 1: On the secondary ( standby ) PIX, type reload to reboot the PIX.

Step 2: On the primary ( active ) PIX, type reload to reboot the PIX.

Note: The primary must be reloaded before the secondary comes back on line.

This gives you only a few seconds to perform these steps.

Step 3: Wait about one minute for both PIXes to complete their resets, then initiate another Telnet ( or SSH ) session to both PIXes.

Step 4: Verify through the show version command output that both PIXes are running the new image.

Step 5: Verify through the show failover command output that the secondary is active.

Step 6: On the primary PIX, issue the failover active command to force it to become the active PIX.

Step 7: Both PIXes are now running the upgraded image and back to normal operation.

When upgrading remotely, the secondary comes up first, so it is active when this process is complete.

You issue the failover active command from the primary to force it to be active.

Note: Downtime occurs when both PIXes are powered off and as the primary PIX boots up.

This downtime is necessary because the PIXes cannot communicate to one another on different versions of code.

--------------------------------------------------------

PIX version 7.0 introduced the ability to perform software upgrades of failover pairs without impacting network uptime or connections flowing through the units.

Version 7.0 introduced the ability to do inter-version state sharing between security appliance failover pairs, allowing you to perform software upgrades to maintenance releases for example, Version 7.0(1) upgrading to 7.0(2)) without impacting traffic flowing through the pair.

In active or standby failover environments or active/active environments where the pair is not oversubscribed, more that 50 percent load on each pair member.

--------------------------------------------------------

To perform the upgrade without impacting connections, back up current configurations and perform these steps:

Step 1: Copy the image files to the device, and select the image to boot by issuing the

boot system

command.

Step 2: Reload the standby unit, which causes it to boot the new image.

Step 3: When the standby unit is in standby_ready state, force a failover so it becomes active.

Re-load the new standby unit. Now both units are running the new image.

For more information refer to:

Configuring Failover

For the process to work smoothly, it is better to reload the standby unit in the procedure instead of the active unit.

In a situation where the units are Active/Active, and you do not want to impact connections on the secondary active unit, you must change the failover groups on the unit to achieve the same active/standby state before performing the steps.

If it is necessary to upgrade to a later version of software, you must upgrade to the next version for the upgrade to avoid impacting network uptime.

For example, if you need to upgrade from version 7.0(1) to version

7.0(3), you must upgrade from version 7.0(1) to version 7.0(2), then to version 7.0(3).

For detailed information visit:

Performing Zero Downtime Upgrades for Failover Pairs

Hope this helps.

Brad Reese Cisco Repair

Brad,

Excellent information. The first of Dr. Jones' books that you cited has been on my Amazon wishlist for a while now. I think it's time to go ahead and pick it up. I think I have what I need to formulate a solid MOP for the upgrade. I also think I have a strong case for justifying an upgrade to v7. That will prompt a separate question to the group though. This particular application demands 5 9s or better. If v7 offers a better way of meeting that requirement then I can justify the upgrade.

Thanks again for the wealth of useful information. J