Cisco ASR 903 Series router ISSU upgrade

Cisco ASR900 series is a popular platform in service provider domain. If you remember the famous 7600 platform,  ASR 900(ASR903/907and 914) is next generation to it. Majorly these devices are classified as SP access platforms. ASR 903 routers are known for high speed data plane(forwarding) because of the ASIC based data plane architecture.

In this blog post I will discuss about the ISSU upgrade procedure on these devices. ISSU upgrade on these devices are considered hit-less(zero impact) if the IO(input/output) plane doesn’t go for reload. IO plane go for reload when the IO firmware also gets upgraded along with the main image. So in case you are upgrading the router from very old image then the IO plane will also go for reload and you will see impact. In general ISSU is non impacting upgrade as the upgrade will reload the primary and standby RSP cards one by one hence at a time there will a one control and data plane available.

Following are some of the per-requisite:

  1. The router should be booted in sub-package mode.
  2. There should be two RSP cards in the router.

Check this blog to know the process of booting router in sub-package mode :

Once you have the router booted in sub-package mode you just need to follow below process to perform ISSU.

ASR903#show version
<----Outpt snipped------->
Uptime for this control processor is 3 minutes
System returned to ROM by reload
System image file is "bootflash:/packages.conf" -------------> this confirms that the router was booted in subpackage mode
Last reload reason: Reload Command
ASR903#show platform
Chassis type: ASR-903
Slot      Type                State                 Insert time (ago)
--------- ------------------- --------------------- -----------------
 0/3          A900-IMA8T1Z                 ok            00:02:47
R0            A900-RSP3C-200-S         ok, active        00:04:28
R1            A900-RSP3C-200-S         ok, standby       00:04:28
F0                                     ok, active                                                 00:04:28
F1                                     ok, standby                                            00:04:28
P0                Unknown                                    N/A                                never
P1            A900-PWR550-A         ok                    00:03:44
P2            A903-FAN              ok                    00:03:43
Slot      CPLD Version        Firmware Version
--------- ------------------- ---------------------------------------
R0        1802052C            15.6(18r)S
R1        1802052C            15.6(18r)S
F0        1802052C            15.6(18r)S
F1        1802052C            15.6(18r)S

Once you have the desired image saved into bootflash , running following command is sufficient to start the ISSU process.

ISSU Procedure :

Overall following steps will be done when ISSU is performed.

  1. Image will be decompressed on standby RSP.
  2. Standby RSP will be reloaded with new image
  3. Once the standby RSP comes up, the process of decompressing image on active RSP starts.
  4. Now the active RSP reloads and comes up as standby
  5. This concludes the ISSU process.

Below are the logs  taken from ASR903 router while ISSU was done.

ASR903#request platform software package install node file bootflash: inasr900rsp3-universalk9.03.18.06.SP.156-2.SP6-ext.bin interface-module-delay 150

Following are the steps that the device goes through. 
Mon May 13 17:24:03 UTC 2019 --- Starting disk space verification ---
Mon May 13 17:24:06 UTC 2019 Finished disk space verification
Mon May 13 17:24:06 UTC 2019
Mon May 13 17:24:06 UTC 2019 --- Starting initial file path checking ---
Mon May 13 17:24:29 UTC 2019 Copying bootflash:asr900rsp3-universalk9.03.18.06.SP.156-2.SP6-ext.bin to stby-bootflash:asr900rsp3-universalk9.03.18.06.SP.156-2.SP6-ext.bin
Mon May 13 17:25:33 UTC 2019 Finished initial file path checking
Mon May 13 17:25:33 UTC 2019
Mon May 13 17:25:33 UTC 2019 --- Starting config-register verification ---
Mon May 13 17:25:36 UTC 2019 Finished config-register verification
Mon May 13 17:25:36 UTC 2019
Mon May 13 17:25:36 UTC 2019 --- Starting image file expansion ---
Mon May 13 17:25:36 UTC 2019 Expanding image file: bootflash:asr900rsp3-universalk9.03.18.06.SP.156-2.SP6-ext.bin
Mon May 13 17:27:32 UTC 2019 Image file expanded and copied
Mon May 13 17:27:32 UTC 2019 Expanding image file: stby-bootflash:asr900rsp3-universalk9.03.18.06.SP.156-2.SP6-ext.bin
Mon May 13 17:29:27 UTC 2019 Image file expanded and copied
Mon May 13 17:29:27 UTC 2019 Finished image file expansion
Mon May 13 17:29:27 UTC 2019
Mon May 13 17:29:27 UTC 2019 STAGE 1: Installing software on standby RP
Mon May 13 17:29:27 UTC 2019 ===================================================
Mon May 13 17:29:28 UTC 2019 --- Starting local lock acquisition on R0 ---
Mon May 13 17:29:28 UTC 2019 Finished local lock acquisition on R0
Mon May 13 17:29:28 UTC 2019
Mon May 13 17:29:28 UTC 2019 --- Starting installation state synchronization ---
Mon May 13 17:29:28 UTC 2019 Finished installation state synchronization
Mon May 13 17:29:28 UTC 2019
Mon May 13 17:29:31 UTC 2019 --- Starting local lock acquisition on R1 ---
Mon May 13 17:29:31 UTC 2019 Finished local lock acquisition on R1
Mon May 13 17:29:31 UTC 2019
Mon May 13 17:29:31 UTC 2019 --- Starting file path checking ---
Mon May 13 17:29:31 UTC 2019 Finished file path checking
Mon May 13 17:29:31 UTC 2019
Mon May 13 17:29:31 UTC 2019 --- Starting image file verification ---
Mon May 13 17:29:31 UTC 2019 Checking image file names
Mon May 13 17:29:31 UTC 2019 Locating image files and validating name syntax
Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-espbase.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-rpaccess.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-rpbase.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-rpcontrol.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-rpios-universalk9.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-sipbase.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-sipspa.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:29:31 UTC 2019 Verifying image file locations
Mon May 13 17:29:31 UTC 2019 Inspecting image file types
Mon May 13 17:29:37 UTC 2019 WARNING: In-service installation of IOSD package
Mon May 13 17:29:37 UTC 2019 WARNING: requires software redundancy on target RP
Mon May 13 17:29:37 UTC 2019 WARNING: or on-reboot parameter
Mon May 13 17:29:37 UTC 2019 WARNING: Automatically setting the on-reboot flag
Mon May 13 17:29:37 UTC 2019 WARNING: In-service installation of RP Base package
Mon May 13 17:29:37 UTC 2019 WARNING: requires software reboot of target RP
Mon May 13 17:29:42 UTC 2019 Processing image file constraints
Mon May 13 17:29:48 UTC 2019 Creating candidate provisioning file
Mon May 13 17:29:54 UTC 2019 Finished image file verification
Mon May 13 17:29:54 UTC 2019
Mon May 13 17:29:54 UTC 2019 --- Starting candidate package set construction ---
Mon May 13 17:29:54 UTC 2019 Verifying existing software set
Mon May 13 17:29:57 UTC 2019 Processing candidate provisioning file
Mon May 13 17:30:25 UTC 2019 Constructing working set for candidate package set
Mon May 13 17:30:25 UTC 2019 Constructing working set for running package set
Mon May 13 17:30:25 UTC 2019 Checking command output
Mon May 13 17:30:25 UTC 2019 Constructing merge of running and candidate packages
Mon May 13 17:30:25 UTC 2019 Checking if resulting candidate package set would be complete
Mon May 13 17:30:25 UTC 2019 Finished candidate package set construction
Mon May 13 17:30:25 UTC 2019
Mon May 13 17:30:25 UTC 2019 --- Starting ISSU compatibility verification ---

Mon May 13 17:30:26 UTC 2019 Verifying image type compatibility
Mon May 13 17:30:26 UTC 2019 Checking IPC compatibility for candidate software
Mon May 13 17:30:34 UTC 2019 Checking candidate package set infrastructure compatibility
Mon May 13 17:30:34 UTC 2019 Checking infrastructure compatibility with running software
Mon May 13 17:30:34 UTC 2019 Checking infrastructure compatibility with running software ... skipped
Mon May 13 17:30:34 UTC 2019 Checking package specific compatibility
Mon May 13 17:30:38 UTC 2019 Finished ISSU compatibility verification
Mon May 13 17:30:38 UTC 2019
Mon May 13 17:30:38 UTC 2019 --- Starting list of software package changes ---
Mon May 13 17:30:38 UTC 2019 No old package files removed
Mon May 13 17:30:38 UTC 2019 No new package files added
Mon May 13 17:30:38 UTC 2019 Finished list of software package changes
Mon May 13 17:30:38 UTC 2019
Mon May 13 17:30:38 UTC 2019 --- Starting commit of software changes ---
Mon May 13 17:30:38 UTC 2019 Updating provisioning rollback files
Mon May 13 17:30:38 UTC 2019 Creating pending provisioning file
Mon May 13 17:30:41 UTC 2019 Committing provisioning file
Mon May 13 17:30:44 UTC 2019 Finished commit of software changes
Mon May 13 17:30:44 UTC 2019
Mon May 13 17:30:44 UTC 2019 SUCCESS: Software provisioned. New software will load on reboot.
Mon May 13 17:30:52 UTC 2019 STAGE 2: Restarting standby RP
Mon May 13 17:30:52 UTC 2019 ===================================================
Mon May 13 17:30:52 UTC 2019 --- Starting standby reload ---
Mon May 13 17:30:53 UTC 2019 Finished standby reload
Mon May 13 17:30:54 UTC 2019
Mon May 13 17:30:54 UTC 2019 --- Starting wait for Standby RP to reach terminal redundancy state ---
*May 13 17:37:21.890: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*May 13 17:37:22.894: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)Mon May 13 17:37:24 UTC 2019 Finished wait for Standby RP to reach terminal redundancy state
Mon May 13 17:37:24 UTC 2019
Mon May 13 17:37:24 UTC 2019 STAGE 3: Installing sipspa package on local RP
Mon May 13 17:37:24 UTC 2019 ===================================================
Mon May 13 17:37:25 UTC 2019 --- Starting local lock acquisition on R0 ---
Mon May 13 17:37:25 UTC 2019 Finished local lock acquisition on R0
Mon May 13 17:37:25 UTC 2019
Mon May 13 17:37:25 UTC 2019 --- Starting installation state synchronization ---
Mon May 13 17:37:25 UTC 2019 Finished installation state synchronization
Mon May 13 17:37:25 UTC 2019
Mon May 13 17:37:25 UTC 2019 --- Starting file path checking ---
Mon May 13 17:37:25 UTC 2019 Finished file path checking
Mon May 13 17:37:25 UTC 2019
Mon May 13 17:37:26 UTC 2019 --- Starting image file verification ---
Mon May 13 17:37:26 UTC 2019 Checking image file names
Mon May 13 17:37:26 UTC 2019 Locating image files and validating name syntax
Mon May 13 17:37:26 UTC 2019 Found asr900rsp3-sipspa.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:37:26 UTC 2019 Verifying image file locations
Mon May 13 17:37:26 UTC 2019 Inspecting image file types
Mon May 13 17:37:28 UTC 2019 Processing image file constraints
Mon May 13 17:37:29 UTC 2019 Creating candidate provisioning file
Mon May 13 17:37:31 UTC 2019 Finished image file verification
Mon May 13 17:37:31 UTC 2019
Mon May 13 17:37:31 UTC 2019 --- Starting candidate package set construction ---
Mon May 13 17:37:31 UTC 2019 Verifying existing software set
Mon May 13 17:37:34 UTC 2019 Processing candidate provisioning file
Mon May 13 17:38:02 UTC 2019 Constructing working set for candidate package set
Mon May 13 17:38:02 UTC 2019 Constructing working set for running package set
Mon May 13 17:38:02 UTC 2019 Checking command output
Mon May 13 17:38:02 UTC 2019 Constructing merge of running and candidate packages
Mon May 13 17:38:02 UTC 2019 Checking if resulting candidate package set would be complete
Mon May 13 17:38:02 UTC 2019 Finished candidate package set construction
Mon May 13 17:38:02 UTC 2019
Mon May 13 17:38:02 UTC 2019 --- Starting ISSU compatibility verification ---
Mon May 13 17:38:02 UTC 2019 Verifying image type compatibility
Mon May 13 17:38:02 UTC 2019 Checking IPC compatibility with running software
Mon May 13 17:38:20 UTC 2019 Checking candidate package set infrastructure compatibility
Mon May 13 17:38:20 UTC 2019 Checking infrastructure compatibility with running software
Mon May 13 17:38:20 UTC 2019 Checking package specific compatibility
Mon May 13 17:38:23 UTC 2019 Finished ISSU compatibility verification
Mon May 13 17:38:23 UTC 2019
Mon May 13 17:38:23 UTC 2019 --- Starting impact testing ---
Mon May 13 17:38:23 UTC 2019 Checking operational impact of change
Mon May 13 17:38:24 UTC 2019 Finished impact testing
Mon May 13 17:38:24 UTC 2019
Mon May 13 17:38:24 UTC 2019 --- Starting list of software package changes ---
Mon May 13 17:38:24 UTC 2019 No old package files removed
Mon May 13 17:38:24 UTC 2019 No new package files added
Mon May 13 17:38:24 UTC 2019 Finished list of software package changes
Mon May 13 17:38:24 UTC 2019
Mon May 13 17:38:24 UTC 2019 --- Starting commit of software changes ---
Mon May 13 17:38:24 UTC 2019 Updating provisioning rollback files
Mon May 13 17:38:24 UTC 2019 Creating pending provisioning file
Mon May 13 17:38:28 UTC 2019 Committing provisioning file
Mon May 13 17:38:31 UTC 2019 Finished commit of software changes
Mon May 13 17:38:31 UTC 2019
Mon May 13 17:38:31 UTC 2019 --- Starting analysis of software changes ---
Mon May 13 17:38:32 UTC 2019 Running software has no changes, reconciliation skipped
Mon May 13 17:38:32 UTC 2019 Finished analysis of software changes
Mon May 13 17:38:32 UTC 2019
Mon May 13 17:38:32 UTC 2019 SUCCESS: Finished installing software.
Mon May 13 17:38:35 UTC 2019 STAGE 4: Installing software on active RP
Mon May 13 17:38:35 UTC 2019 ===================================================
Mon May 13 17:38:36 UTC 2019 --- Starting local lock acquisition on R0 ---
Mon May 13 17:38:36 UTC 2019 Finished local lock acquisition on R0
Mon May 13 17:38:36 UTC 2019
Mon May 13 17:38:36 UTC 2019 --- Starting installation state synchronization ---
Mon May 13 17:38:36 UTC 2019 Finished installation state synchronization
Mon May 13 17:38:36 UTC 2019 --- Starting file path checking ---
Mon May 13 17:38:36 UTC 2019 Finished file path checking
Mon May 13 17:38:36 UTC 2019 --- Starting image file verification ---
Mon May 13 17:38:36 UTC 2019 Checking image file names
Mon May 13 17:38:36 UTC 2019 Locating image files and validating name syntax
Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-espbase.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-rpaccess.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-rpbase.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-rpcontrol.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-rpios-universalk9.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-sipbase.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-sipspa.03.18.06.SP.156-2.SP6-ext.pkg
Mon May 13 17:38:37 UTC 2019 Verifying image file locations
Mon May 13 17:38:37 UTC 2019 Inspecting image file types
Mon May 13 17:38:42 UTC 2019 WARNING: In-service installation of IOSD package
Mon May 13 17:38:42 UTC 2019 WARNING: requires software redundancy on target RP
Mon May 13 17:38:42 UTC 2019 WARNING: or on-reboot parameter
Mon May 13 17:38:42 UTC 2019 WARNING: Automatically setting the on-reboot flag
Mon May 13 17:38:42 UTC 2019 WARNING: In-service installation of RP Base package
Mon May 13 17:38:42 UTC 2019 WARNING: requires software reboot of target RP
Mon May 13 17:38:48 UTC 2019 Processing image file constraints
Mon May 13 17:38:53 UTC 2019 Creating candidate provisioning file
Mon May 13 17:39:00 UTC 2019 Finished image file verification
Mon May 13 17:39:00 UTC 2019 --- Starting candidate package set construction ---
Mon May 13 17:39:00 UTC 2019 Verifying existing software set
Mon May 13 17:39:03 UTC 2019 Processing candidate provisioning file
Mon May 13 17:39:31 UTC 2019 Constructing working set for candidate package set
Mon May 13 17:39:31 UTC 2019 Constructing working set for running package set
Mon May 13 17:39:31 UTC 2019 Checking command output
Mon May 13 17:39:31 UTC 2019 Constructing merge of running and candidate packages
Mon May 13 17:39:31 UTC 2019 Checking if resulting candidate package set would be complete
Mon May 13 17:39:31 UTC 2019 Finished candidate package set construction
Mon May 13 17:39:31 UTC 2019 --- Starting ISSU compatibility verification ---
Mon May 13 17:39:32 UTC 2019 Verifying image type compatibility
Mon May 13 17:39:32 UTC 2019 Checking IPC compatibility for candidate software
Mon May 13 17:39:40 UTC 2019 Checking candidate package set infrastructure compatibility
Mon May 13 17:39:40 UTC 2019 Checking infrastructure compatibility with running software
Mon May 13 17:39:40 UTC 2019 Checking infrastructure compatibility with running software ... skipped
Mon May 13 17:39:40 UTC 2019 Checking package specific compatibility
Mon May 13 17:39:44 UTC 2019 Finished ISSU compatibility verification
Mon May 13 17:39:44 UTC 2019 --- Starting list of software package changes ---
Mon May 13 17:39:44 UTC 2019 No old package files removed
Mon May 13 17:39:44 UTC 2019 No new package files added
Mon May 13 17:39:44 UTC 2019 Finished list of software package changes
Mon May 13 17:39:44 UTC 2019 --- Starting commit of software changes ---
Mon May 13 17:39:44 UTC 2019 Updating provisioning rollback files
Mon May 13 17:39:44 UTC 2019 Creating pending provisioning file
Mon May 13 17:39:47 UTC 2019 Committing provisioning file
Mon May 13 17:39:50 UTC 2019 Finished commit of software changes
Mon May 13 17:39:50 UTC 2019 SUCCESS: Software provisioned. New software will load on reboot.
Mon May 13 17:39:54 UTC 2019 STAGE 5: Restarting active RP (switchover to stdby)
Mon May 13 17:39:54 UTC 2019 ===================================================
Mon May 13 17:39:54 UTC 2019 --- Starting active reload ---
Mon May 13 17:39:54 UTC 2019 Finished active reload
Mon May 13 17:39:54 UTC 2019
Mon May 13 17:39:54 UTC 2019 SUCCESS: node ISSU finished successfully.
ASR903#
*May 13 17:39:54.343: %CMRP-3-RP_RESET: R0/0: cmand: RP is resetting : EHSA state has been set to NON_PARTICIPATE
<---Output snipped--->
*May 13 17:45:45.435: %CMRP-6-FP_HA_STATUS: R0/0: cmand: F0 redundancy state is Standby
*May 13 17:45:55.340: %PLATFORM-6-RF_PROG_SUCCESS: RF state STANDBY HOT
ASR903-stby>
Standby console disabled

Leave a Reply