Cisco ASR900 series is a popular platform in service provider domain. If you remember the famous 7600 platform, ASR 900(ASR903/907and 914) is next generation to it. Majorly these devices are classified as SP access platforms. ASR 903 routers are known for high speed data plane(forwarding) because of the ASIC based data plane architecture.
In this blog post I will discuss about the ISSU upgrade procedure on these devices. ISSU upgrade on these devices are considered hit-less(zero impact) if the IO(input/output) plane doesn’t go for reload. IO plane go for reload when the IO firmware also gets upgraded along with the main image. So in case you are upgrading the router from very old image then the IO plane will also go for reload and you will see impact. In general ISSU is non impacting upgrade as the upgrade will reload the primary and standby RSP cards one by one hence at a time there will a one control and data plane available.
Following are some of the per-requisite:
- The router should be booted in sub-package mode.
- There should be two RSP cards in the router.
Check this blog to know the process of booting router in sub-package mode :
Once you have the router booted in sub-package mode you just need to follow below process to perform ISSU.
ASR903#show version
<----Outpt snipped------->
Uptime for this control processor is 3 minutes
System returned to ROM by reload
System image file is "bootflash:/packages.conf" -------------> this confirms that the router was booted in subpackage mode
Last reload reason: Reload Command
ASR903#show platform Chassis type: ASR-903 Slot Type State Insert time (ago) --------- ------------------- --------------------- ----------------- 0/3 A900-IMA8T1Z ok 00:02:47 R0 A900-RSP3C-200-S ok, active 00:04:28 R1 A900-RSP3C-200-S ok, standby 00:04:28 F0 ok, active 00:04:28 F1 ok, standby 00:04:28 P0 Unknown N/A never P1 A900-PWR550-A ok 00:03:44 P2 A903-FAN ok 00:03:43 Slot CPLD Version Firmware Version --------- ------------------- --------------------------------------- R0 1802052C 15.6(18r)S R1 1802052C 15.6(18r)S F0 1802052C 15.6(18r)S F1 1802052C 15.6(18r)S
Once you have the desired image saved into bootflash , running following command is sufficient to start the ISSU process.
ISSU Procedure :
Overall following steps will be done when ISSU is performed.
- Image will be decompressed on standby RSP.
- Standby RSP will be reloaded with new image
- Once the standby RSP comes up, the process of decompressing image on active RSP starts.
- Now the active RSP reloads and comes up as standby
- This concludes the ISSU process.
Below are the logs taken from ASR903 router while ISSU was done.
ASR903#request platform software package install node file bootflash: inasr900rsp3-universalk9.03.18.06.SP.156-2.SP6-ext.bin interface-module-delay 150 Following are the steps that the device goes through. Mon May 13 17:24:03 UTC 2019 --- Starting disk space verification --- Mon May 13 17:24:06 UTC 2019 Finished disk space verification Mon May 13 17:24:06 UTC 2019 Mon May 13 17:24:06 UTC 2019 --- Starting initial file path checking --- Mon May 13 17:24:29 UTC 2019 Copying bootflash:asr900rsp3-universalk9.03.18.06.SP.156-2.SP6-ext.bin to stby-bootflash:asr900rsp3-universalk9.03.18.06.SP.156-2.SP6-ext.bin Mon May 13 17:25:33 UTC 2019 Finished initial file path checking Mon May 13 17:25:33 UTC 2019 Mon May 13 17:25:33 UTC 2019 --- Starting config-register verification --- Mon May 13 17:25:36 UTC 2019 Finished config-register verification Mon May 13 17:25:36 UTC 2019 Mon May 13 17:25:36 UTC 2019 --- Starting image file expansion --- Mon May 13 17:25:36 UTC 2019 Expanding image file: bootflash:asr900rsp3-universalk9.03.18.06.SP.156-2.SP6-ext.bin Mon May 13 17:27:32 UTC 2019 Image file expanded and copied Mon May 13 17:27:32 UTC 2019 Expanding image file: stby-bootflash:asr900rsp3-universalk9.03.18.06.SP.156-2.SP6-ext.bin Mon May 13 17:29:27 UTC 2019 Image file expanded and copied Mon May 13 17:29:27 UTC 2019 Finished image file expansion Mon May 13 17:29:27 UTC 2019 Mon May 13 17:29:27 UTC 2019 STAGE 1: Installing software on standby RP Mon May 13 17:29:27 UTC 2019 =================================================== Mon May 13 17:29:28 UTC 2019 --- Starting local lock acquisition on R0 --- Mon May 13 17:29:28 UTC 2019 Finished local lock acquisition on R0 Mon May 13 17:29:28 UTC 2019 Mon May 13 17:29:28 UTC 2019 --- Starting installation state synchronization --- Mon May 13 17:29:28 UTC 2019 Finished installation state synchronization Mon May 13 17:29:28 UTC 2019 Mon May 13 17:29:31 UTC 2019 --- Starting local lock acquisition on R1 --- Mon May 13 17:29:31 UTC 2019 Finished local lock acquisition on R1 Mon May 13 17:29:31 UTC 2019 Mon May 13 17:29:31 UTC 2019 --- Starting file path checking --- Mon May 13 17:29:31 UTC 2019 Finished file path checking Mon May 13 17:29:31 UTC 2019 Mon May 13 17:29:31 UTC 2019 --- Starting image file verification --- Mon May 13 17:29:31 UTC 2019 Checking image file names Mon May 13 17:29:31 UTC 2019 Locating image files and validating name syntax Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-espbase.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-rpaccess.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-rpbase.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-rpcontrol.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-rpios-universalk9.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-sipbase.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:29:31 UTC 2019 Found asr900rsp3-sipspa.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:29:31 UTC 2019 Verifying image file locations Mon May 13 17:29:31 UTC 2019 Inspecting image file types Mon May 13 17:29:37 UTC 2019 WARNING: In-service installation of IOSD package Mon May 13 17:29:37 UTC 2019 WARNING: requires software redundancy on target RP Mon May 13 17:29:37 UTC 2019 WARNING: or on-reboot parameter Mon May 13 17:29:37 UTC 2019 WARNING: Automatically setting the on-reboot flag Mon May 13 17:29:37 UTC 2019 WARNING: In-service installation of RP Base package Mon May 13 17:29:37 UTC 2019 WARNING: requires software reboot of target RP Mon May 13 17:29:42 UTC 2019 Processing image file constraints Mon May 13 17:29:48 UTC 2019 Creating candidate provisioning file Mon May 13 17:29:54 UTC 2019 Finished image file verification Mon May 13 17:29:54 UTC 2019 Mon May 13 17:29:54 UTC 2019 --- Starting candidate package set construction --- Mon May 13 17:29:54 UTC 2019 Verifying existing software set Mon May 13 17:29:57 UTC 2019 Processing candidate provisioning file Mon May 13 17:30:25 UTC 2019 Constructing working set for candidate package set Mon May 13 17:30:25 UTC 2019 Constructing working set for running package set Mon May 13 17:30:25 UTC 2019 Checking command output Mon May 13 17:30:25 UTC 2019 Constructing merge of running and candidate packages Mon May 13 17:30:25 UTC 2019 Checking if resulting candidate package set would be complete Mon May 13 17:30:25 UTC 2019 Finished candidate package set construction Mon May 13 17:30:25 UTC 2019 Mon May 13 17:30:25 UTC 2019 --- Starting ISSU compatibility verification --- Mon May 13 17:30:26 UTC 2019 Verifying image type compatibility Mon May 13 17:30:26 UTC 2019 Checking IPC compatibility for candidate software Mon May 13 17:30:34 UTC 2019 Checking candidate package set infrastructure compatibility Mon May 13 17:30:34 UTC 2019 Checking infrastructure compatibility with running software Mon May 13 17:30:34 UTC 2019 Checking infrastructure compatibility with running software ... skipped Mon May 13 17:30:34 UTC 2019 Checking package specific compatibility Mon May 13 17:30:38 UTC 2019 Finished ISSU compatibility verification Mon May 13 17:30:38 UTC 2019 Mon May 13 17:30:38 UTC 2019 --- Starting list of software package changes --- Mon May 13 17:30:38 UTC 2019 No old package files removed Mon May 13 17:30:38 UTC 2019 No new package files added Mon May 13 17:30:38 UTC 2019 Finished list of software package changes Mon May 13 17:30:38 UTC 2019 Mon May 13 17:30:38 UTC 2019 --- Starting commit of software changes --- Mon May 13 17:30:38 UTC 2019 Updating provisioning rollback files Mon May 13 17:30:38 UTC 2019 Creating pending provisioning file Mon May 13 17:30:41 UTC 2019 Committing provisioning file Mon May 13 17:30:44 UTC 2019 Finished commit of software changes Mon May 13 17:30:44 UTC 2019 Mon May 13 17:30:44 UTC 2019 SUCCESS: Software provisioned. New software will load on reboot. Mon May 13 17:30:52 UTC 2019 STAGE 2: Restarting standby RP Mon May 13 17:30:52 UTC 2019 =================================================== Mon May 13 17:30:52 UTC 2019 --- Starting standby reload --- Mon May 13 17:30:53 UTC 2019 Finished standby reload Mon May 13 17:30:54 UTC 2019 Mon May 13 17:30:54 UTC 2019 --- Starting wait for Standby RP to reach terminal redundancy state --- *May 13 17:37:21.890: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded *May 13 17:37:22.894: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)Mon May 13 17:37:24 UTC 2019 Finished wait for Standby RP to reach terminal redundancy state Mon May 13 17:37:24 UTC 2019 Mon May 13 17:37:24 UTC 2019 STAGE 3: Installing sipspa package on local RP Mon May 13 17:37:24 UTC 2019 =================================================== Mon May 13 17:37:25 UTC 2019 --- Starting local lock acquisition on R0 --- Mon May 13 17:37:25 UTC 2019 Finished local lock acquisition on R0 Mon May 13 17:37:25 UTC 2019 Mon May 13 17:37:25 UTC 2019 --- Starting installation state synchronization --- Mon May 13 17:37:25 UTC 2019 Finished installation state synchronization Mon May 13 17:37:25 UTC 2019 Mon May 13 17:37:25 UTC 2019 --- Starting file path checking --- Mon May 13 17:37:25 UTC 2019 Finished file path checking Mon May 13 17:37:25 UTC 2019 Mon May 13 17:37:26 UTC 2019 --- Starting image file verification --- Mon May 13 17:37:26 UTC 2019 Checking image file names Mon May 13 17:37:26 UTC 2019 Locating image files and validating name syntax Mon May 13 17:37:26 UTC 2019 Found asr900rsp3-sipspa.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:37:26 UTC 2019 Verifying image file locations Mon May 13 17:37:26 UTC 2019 Inspecting image file types Mon May 13 17:37:28 UTC 2019 Processing image file constraints Mon May 13 17:37:29 UTC 2019 Creating candidate provisioning file Mon May 13 17:37:31 UTC 2019 Finished image file verification Mon May 13 17:37:31 UTC 2019 Mon May 13 17:37:31 UTC 2019 --- Starting candidate package set construction --- Mon May 13 17:37:31 UTC 2019 Verifying existing software set Mon May 13 17:37:34 UTC 2019 Processing candidate provisioning file Mon May 13 17:38:02 UTC 2019 Constructing working set for candidate package set Mon May 13 17:38:02 UTC 2019 Constructing working set for running package set Mon May 13 17:38:02 UTC 2019 Checking command output Mon May 13 17:38:02 UTC 2019 Constructing merge of running and candidate packages Mon May 13 17:38:02 UTC 2019 Checking if resulting candidate package set would be complete Mon May 13 17:38:02 UTC 2019 Finished candidate package set construction Mon May 13 17:38:02 UTC 2019 Mon May 13 17:38:02 UTC 2019 --- Starting ISSU compatibility verification --- Mon May 13 17:38:02 UTC 2019 Verifying image type compatibility Mon May 13 17:38:02 UTC 2019 Checking IPC compatibility with running software Mon May 13 17:38:20 UTC 2019 Checking candidate package set infrastructure compatibility Mon May 13 17:38:20 UTC 2019 Checking infrastructure compatibility with running software Mon May 13 17:38:20 UTC 2019 Checking package specific compatibility Mon May 13 17:38:23 UTC 2019 Finished ISSU compatibility verification Mon May 13 17:38:23 UTC 2019 Mon May 13 17:38:23 UTC 2019 --- Starting impact testing --- Mon May 13 17:38:23 UTC 2019 Checking operational impact of change Mon May 13 17:38:24 UTC 2019 Finished impact testing Mon May 13 17:38:24 UTC 2019 Mon May 13 17:38:24 UTC 2019 --- Starting list of software package changes --- Mon May 13 17:38:24 UTC 2019 No old package files removed Mon May 13 17:38:24 UTC 2019 No new package files added Mon May 13 17:38:24 UTC 2019 Finished list of software package changes Mon May 13 17:38:24 UTC 2019 Mon May 13 17:38:24 UTC 2019 --- Starting commit of software changes --- Mon May 13 17:38:24 UTC 2019 Updating provisioning rollback files Mon May 13 17:38:24 UTC 2019 Creating pending provisioning file Mon May 13 17:38:28 UTC 2019 Committing provisioning file Mon May 13 17:38:31 UTC 2019 Finished commit of software changes Mon May 13 17:38:31 UTC 2019 Mon May 13 17:38:31 UTC 2019 --- Starting analysis of software changes --- Mon May 13 17:38:32 UTC 2019 Running software has no changes, reconciliation skipped Mon May 13 17:38:32 UTC 2019 Finished analysis of software changes Mon May 13 17:38:32 UTC 2019 Mon May 13 17:38:32 UTC 2019 SUCCESS: Finished installing software. Mon May 13 17:38:35 UTC 2019 STAGE 4: Installing software on active RP Mon May 13 17:38:35 UTC 2019 =================================================== Mon May 13 17:38:36 UTC 2019 --- Starting local lock acquisition on R0 --- Mon May 13 17:38:36 UTC 2019 Finished local lock acquisition on R0 Mon May 13 17:38:36 UTC 2019 Mon May 13 17:38:36 UTC 2019 --- Starting installation state synchronization --- Mon May 13 17:38:36 UTC 2019 Finished installation state synchronization Mon May 13 17:38:36 UTC 2019 --- Starting file path checking --- Mon May 13 17:38:36 UTC 2019 Finished file path checking Mon May 13 17:38:36 UTC 2019 --- Starting image file verification --- Mon May 13 17:38:36 UTC 2019 Checking image file names Mon May 13 17:38:36 UTC 2019 Locating image files and validating name syntax Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-espbase.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-rpaccess.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-rpbase.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-rpcontrol.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-rpios-universalk9.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-sipbase.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:38:36 UTC 2019 Found asr900rsp3-sipspa.03.18.06.SP.156-2.SP6-ext.pkg Mon May 13 17:38:37 UTC 2019 Verifying image file locations Mon May 13 17:38:37 UTC 2019 Inspecting image file types Mon May 13 17:38:42 UTC 2019 WARNING: In-service installation of IOSD package Mon May 13 17:38:42 UTC 2019 WARNING: requires software redundancy on target RP Mon May 13 17:38:42 UTC 2019 WARNING: or on-reboot parameter Mon May 13 17:38:42 UTC 2019 WARNING: Automatically setting the on-reboot flag Mon May 13 17:38:42 UTC 2019 WARNING: In-service installation of RP Base package Mon May 13 17:38:42 UTC 2019 WARNING: requires software reboot of target RP Mon May 13 17:38:48 UTC 2019 Processing image file constraints Mon May 13 17:38:53 UTC 2019 Creating candidate provisioning file Mon May 13 17:39:00 UTC 2019 Finished image file verification Mon May 13 17:39:00 UTC 2019 --- Starting candidate package set construction --- Mon May 13 17:39:00 UTC 2019 Verifying existing software set Mon May 13 17:39:03 UTC 2019 Processing candidate provisioning file Mon May 13 17:39:31 UTC 2019 Constructing working set for candidate package set Mon May 13 17:39:31 UTC 2019 Constructing working set for running package set Mon May 13 17:39:31 UTC 2019 Checking command output Mon May 13 17:39:31 UTC 2019 Constructing merge of running and candidate packages Mon May 13 17:39:31 UTC 2019 Checking if resulting candidate package set would be complete Mon May 13 17:39:31 UTC 2019 Finished candidate package set construction Mon May 13 17:39:31 UTC 2019 --- Starting ISSU compatibility verification --- Mon May 13 17:39:32 UTC 2019 Verifying image type compatibility Mon May 13 17:39:32 UTC 2019 Checking IPC compatibility for candidate software Mon May 13 17:39:40 UTC 2019 Checking candidate package set infrastructure compatibility Mon May 13 17:39:40 UTC 2019 Checking infrastructure compatibility with running software Mon May 13 17:39:40 UTC 2019 Checking infrastructure compatibility with running software ... skipped Mon May 13 17:39:40 UTC 2019 Checking package specific compatibility Mon May 13 17:39:44 UTC 2019 Finished ISSU compatibility verification Mon May 13 17:39:44 UTC 2019 --- Starting list of software package changes --- Mon May 13 17:39:44 UTC 2019 No old package files removed Mon May 13 17:39:44 UTC 2019 No new package files added Mon May 13 17:39:44 UTC 2019 Finished list of software package changes Mon May 13 17:39:44 UTC 2019 --- Starting commit of software changes --- Mon May 13 17:39:44 UTC 2019 Updating provisioning rollback files Mon May 13 17:39:44 UTC 2019 Creating pending provisioning file Mon May 13 17:39:47 UTC 2019 Committing provisioning file Mon May 13 17:39:50 UTC 2019 Finished commit of software changes Mon May 13 17:39:50 UTC 2019 SUCCESS: Software provisioned. New software will load on reboot. Mon May 13 17:39:54 UTC 2019 STAGE 5: Restarting active RP (switchover to stdby) Mon May 13 17:39:54 UTC 2019 =================================================== Mon May 13 17:39:54 UTC 2019 --- Starting active reload --- Mon May 13 17:39:54 UTC 2019 Finished active reload Mon May 13 17:39:54 UTC 2019 Mon May 13 17:39:54 UTC 2019 SUCCESS: node ISSU finished successfully. ASR903# *May 13 17:39:54.343: %CMRP-3-RP_RESET: R0/0: cmand: RP is resetting : EHSA state has been set to NON_PARTICIPATE <---Output snipped---> *May 13 17:45:45.435: %CMRP-6-FP_HA_STATUS: R0/0: cmand: F0 redundancy state is Standby *May 13 17:45:55.340: %PLATFORM-6-RF_PROG_SUCCESS: RF state STANDBY HOT ASR903-stby> Standby console disabled