Published on

Palo Alto PAN-OS

Authors
  • Name
    Jackson Chen

Reference

PAN-OS Upgrade Guide

https://docs.paloaltonetworks.com/pan-os/10-1/pan-os-upgrade/upgrade-pan-os/upgrade-the-firewall-pan-os/upgrade-an-ha-firewall-pair

Review the PAN-OS 10.1 Release Notes and then use the following procedure to upgrade a pair of firewalls in a high availability (HA) configuration. This procedure applies to both active/passive and active/active configurations.

To avoid downtime when upgrading firewalls that are in a high availability (HA) configuration, update one HA peer at a time: For active/active firewalls, it doesn’t matter which peer you upgrade first (though for simplicity, this procedure shows you how to upgrade the active-primary peer first). For active/passive firewalls, you must suspend (fail over) and upgrade the active (primary) peer first. After you upgrade the primary peer, you must unsuspend the primary peer to return it to a functional state (passive). Next, you must suspend the passive (secondary) peer to make the primary peer active again. After the primary peer is active and the secondary peer is suspended, you can continue the upgrade. To prevent failover during the upgrade of the HA peers, you must make sure preemption is disabled before proceeding with the upgrade. You only need to disable preemption on one peer in the pair.

Active passive firewall upgrade overview

# PA Active/Passive Firewalls
1. Must suspend (fail over)
2. Upgrade the active (primary) peer first
3. After successfully upgrade the primary peer, must un-suspend the primary peer to return it to a functional state (passive)
4. Next, you must suspend the passive (secondary) peer to make the primary peer active again
5. After the primary peer is active, and the secondary peer is suspended, you can continue the upgrade to secondary peer.

Note:
    To prevent failover during the upgrade of the HA peer,
    you must make sure pre-emption is disable before proceeding with the upgrade.
    You only need to disable preemption on one peer in the pair.

Upgrade HA firewalls across multiple feature PAN-OS releases

When upgrading HA firewalls across multiple feature PAN-OS releases, you must upgrade each HA peer to the same feature PAN-OS release on your upgrade path before continuing. For example, you are upgrading HA peers from PAN-OS 9.1 to PAN-OS 10.1. You must upgrade both HA peers to PAN-OS 10.0 before you can continue upgrading to the target PAN-OS 10.1 release. When HA peers are two or more feature releases apart, the firewall with the older release installed enters a suspended state with the message Peer version too old.

Note:
1. When upgrading HP firewalls across multiple feature PAN-OS releases, 
    you must upgrade each HA peer to the same feature PAN-OS release on your upgrade path before continue

2. When HA peers are two ore more features release apart,
    the firewall with the older release installed enter a suspended state with the message "Peer version too old"
Important:
To avoid impacting traffic, plan to upgrade within the outage window. 
Ensure the firewalls are connected to a reliable power source. 
A loss of power during an upgrade can make firewalls unusable.

Upgrade rocess

Step 1 - Save a backup of the current configuration file

Note: Although the firewall automatically creates a backup of the configuration, 
    it is a best practice to create and externally store a backup before you upgrade.
1. Select Device -> Setup -> Operations
2. click Export named configuration snapshot
3. Select the XML file that contains your running configuration (for example, running-config.xml) 
4. and click OK to export the configuration file.
5. Save the exported file to a location external to the firewall. 
    You can use this backup to restore the configuration if you have problems with the upgrade.

Step 2 - Select DeviceSupport and Generate Tech Support File. Click Yes when prompted to generate the tech support file.

Step 3 - Ensure that each firewall in the HA pair is running the latest content release version.

Refer to the release notes for the minimum content release version you must install for a PAN-OS 10.1 release. Make sure to follow the Best Practices for Applications and Threats Content Updates.

1. Select Device -> Dynamic Updates, 
    and check which Applications or Applications and Threats to determine which update is Currently Installed.

2. If the firewalls are not running the minimum required content release version or a later version required for PAN-OS 10.1, 
    Check Now to retrieve a list of available updates.
3. Locate and Download the desired content release version.
    After you successfully download a content update file, 
    the link in the Action column changes from Download to Install for that content release version.
4. Install the update. You must install the update on both peers.

Step 4 - Determine the Upgrade Path to PAN-OS 10.1 You cannot skip the installation of any feature release versions in the path from the currently running PAN-OS version to PAN-OS 10.1

Review PAN-OS Upgrade Checklist, the known issues and changes to default behavior in the Release Notes and Upgrade/Downgrade Considerations for each release through which you pass as part of your upgrade path.

Note: 
    The firewall automatically switches to using the device certificate for authentication with CDL ingestion and query endpoints on upgrade to PAN-OS 10.1.

    If you do not install the device certificate prior to upgrade to PAN-OS 10.1, 
    the firewall continues to use the existing logging service certificates for authentication.

Step 6 - Disable preemption on the first peer in each pair. You only need to disable this setting on one firewall in the HA pair but ensure that the commit is successful before you proceed with the upgrade.

1. Select Device -> High Availability and edit the Election Settings.
2. If enabled, disable (clear) the Preemptive setting and click OK.
3. Commit the change.

Step 7 - Suspend the primary HA peer to force a failover.

# (Active/passive firewalls) 
For firewalls in an active/passive HA configuration, suspend and upgrade the active HA peer firs

# (Active/active firewalls) 
For firewalls in an active/active HA configuration, suspend and upgrade the active-primary HA peer first.

1. Select Device =. High Availability -> Operational Commands and Suspend local device for high availability.
2. In the bottom-right corner, verify that the state is suspended.
    The resulting failover should cause the secondary HA peer to transition to active state.
    Note: 
        The resulting failover verifies that HA failover is functioning properly before you upgrade.

Step 8 - Install PAN-OS 10.1 on the suspended HA peer

1. On the primary HA peer, select Device -> Software and click Check Now for the latest updates.
    Only the versions for the next available PAN-OS release are displayed. 
    For example, if the PAN-OS 10.0 is installed on the firewall, then only PAN-OS 10.1 releases are displayed.
2. Locate and Download PAN-OS 10.1.0.
Note:
    If your firewall does not have internet access from the management port, 
    you can download the software image from the Palo Alto Networks Support Portal and then manually Upload it to your firewall.
3. After you download the image (or, for a manual upgrade, after you upload the image), Install the image.
4. After the installation completes successfully, reboot using one of the following methods:
    . If you are prompted to reboot, click Yes.
    . If you are not prompted to reboot, select Device -> Setup -> Operations and Reboot Device.
5. After the device finishes rebooting, 
    view the High Availability widget on the Dashboard and verify that the device you just upgraded is in sync with the peer.

Step 9 - Restore HA functionality to the primary HA peer.

1. Select Device -> High Availability -> Operational Commands 
    and Make local device functional for high availability.
2. In the bottom-right corner, verify that the state is Passive. 
    For firewalls in an active/active configuration, verify that the state is Active.
3. Wait for the HA peer running configuration to synchronize.
    In the Dasbhoard, monitor the Running Config status in the High Availability widget.

Step 10 - On the secondary HA peer, suspend the HA peer.

1. Select Device -> High Availability -> Operational Commands and Suspend local device for high availability.
2. In the bottom-right corner, verify that the state is suspended.
    The resulting failover should cause the primary HA peer to transition to Active state.

Step 11 - Install PAN-OS 10.1 on the secondary HA peer.

1. On the second peer, select Device -> Software and click Check Now for the latest updates.
2. Locate and Download PAN-OS 10.1.0.
3. After you download the image, Install it.
4. After the installation completes successfully, reboot using one of the following methods:
    . If you are prompted to reboot, click Yes.
    . If you are not prompted to reboot, select Device -> Setup -> Operations and Reboot Device.

Step 12 - Restore HA functionality to the secondary HA peer.

1. Select Device -> High Availability -> Operational Commands 
    and Make local device functional for high availability.
2. In the bottom-right corner, verify that the state is Passive. 
    For firewalls in an active/active configuration, verify that the state is Active.
3. Wait for the HA peer running configuration to synchronize.
    In the Dasbhoard, monitor the Running Config status High Availability widget.

Step 13 - Re-enable preemption on the HA peer where it was disabled in the previous step.

1. Select Device -> High Availability and edit the Election Settings.
2. Enable (check) the Preemptive setting and click OK.
3. Commit the change.

Step 14 - Verify that both peers are passing traffic as expected.

1. In an active/passive configuration, only the active peer should be passing traffic; 
    both peers should be passing traffic in an active/active configuration.
2. Run the following CLI commands to confirm that the upgrade succeeded:
    . (Active peers only) To verify that active peers are passing traffic, run the show session all command.
        $ show sessioin all
    . To verify session synchronization, 
        run the show high-availability interface ha2 command and make sure that the Hardware Interface counters on the CPU table are increasing as follows:
        $ show high-availability interface ha2
    Note:
        If you enabled HA2 keep-alive, the hardware interface counters on the passive peer will show both transmit and receive packets. 
        This occurs because HA2 keep-alive is bi-directional, which means that both peers transmit HA2 keep-alive packets.
    . In an active/passive configuration, only the active peer shows packets transmitted; 
        the passive peer will show only packets received.

How to troubleshoot your PAN-OS Upgrade

https://docs.paloaltonetworks.com/pan-os/10-1/pan-os-upgrade/upgrade-pan-os/troubleshoot-pan-os-upgrade

SYMPTOM                                         RESOLUTION
----------------------------------------------------------------------------------------------------------
The software warranty license expired.          
                                            From the CLI, delete the expired license key:
                                            1. Enter delete license key <software license key>.
                                            2. Enter delete license key Software_Warranty<expiredate>.key.

The latest PAN-OS software versions were not available.
                                            You can only see software versions that are one feature release ahead of the current installed version. 
                                            For example, if you have an 8.1 release installed, only 9.0 releases will be available to you. 
                                            To see 9.1 releases, you first have to upgrade to 9.0.

Checking for dynamic updates failed.
                                            This issue occurs due to a network connectivity error. 
                                            See the KnowledgeBase article Dynamic Updates Display Error After Clicking On Check Now Button.

No valid device certificate was found.
                                            In PAN-OS 9.1 and later versions, a device certificate must be installed. 
                                            To install the certificate:
                                            1. Log in to the Customer Support Portal.
                                            2. Select Generate OTP (AssetsDevice Certificates).
                                            3. In Device Type, select Generate OTP for Next-Gen Firewalls.
                                            4. Select your PAN-OS device serial number.
                                            5. Generate OTP and copy the one-time-password.
                                            6. Log in to the firewall as an admin user.
                                            7. Select Device Certificate 
                                                (Device -> Setup -> Management -> Device -> Certificate and Get Certificate)
                                            8. Paste the OTP and click OK.

The software image file failed to load onto the software manager due to an image authentication error.
                                            To update the software image list, click Check Now. 
                                            This establishes a new connection to the update server.

The VMware NSX plugin version was not compatible with the new software version.
                                            The VMware NSX plugin was automatically installed upon upgrade to 8.0. 
                                            If you are not using the plugin, you can uninstall it.

The reboot time after upgrading to PAN-OS 9.1 was longer than expected.
                                            Upgrade to Applications and Threats Content Release Version 8221 or later. 
                                            For more information on minimum software and content versions, 
                                            see <xref to 10.1 Associated Software and Content Versions>.

The device did not have support even when licenses are active.
                                            In Device -> Software, click Check Now
                                            This updates the licensing information on the firewall by establishing a new connection to the update server.
                                            If this does not work from the web interface, use request system software check.

The firewall did not have a DHCP address assigned to it by the DHCP server.
                                            Configure a security policy rule allowing the traffic from the ISP DHCP server to the internal networks.

The firewall continuously boots into maintenance mode.
                                            In the CLI, Access the Maintenance Recovery Tool (MRT). 
                                            In the MRT window, select Continue -> Disk Image. 
                                            Select either Reinstall <current version> or Revert to <previous version>. 
                                            Once the revert or reinstall operation completes, 
                                            select Reboot.

In an HA configuration, the firewall goes into a suspended state after upgrading the peer firewall with an error that the firewall is too old.
                                            Upgrading one firewall to a version that is more than one major release ahead will result in a network outage. 
                                            You must upgrade both firewalls only one major release ahead before upgrading to the next major release.
                                            Downgrade the peer firewall to the version that the suspended firewall stopped at.