This post is intended to convey the basic data collection needed for TAC to identify the cause of a router crash. The scope of this blog is limited to router platforms running IOS-XE software. For list of products supporting IOS-XE, please refer the following link:
https://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-xe/index.html#~stickynav=2
we would be briefly discussing about- what is a router crash and types of crashes, before we go the data collection needed for a crash. IOS XE Software is a modular operating system built on a Linux kernel on a Route Processor (RP), Embedded Services Processor (ESP), or SPA Interface Processor (SIP). The IOS daemon (IOSD) and other IOS XE processes run on the Linux kernel, so there are several types of crashes as listed below:
1- IOSD crash (on RP module) 2- SPA driver crash (on SIP module) 3- IOS-XE process crash (on RP, ESP, SIP) 4- QFP microcode crash (on ESP module) 5- Linux Kernel crash (on RP, ESP, SIP)
Types of Crashes | Crashinfo File Name (Syntax) | Example |
IOSD Crash | crashinfo_RP_SlotNumber_00_Date-Time-Zone | crashinfo_RP_00_00_20080807-063430-UTC |
SPA Driver Crash | crashinfo_SIP_SlotNumber_00_Date-Time-Zone | crashinfo_SIP_00_00_20080828-084907-UTC |
Types of Crashes | Core Dump File Name (Syntax) | Example |
IOSD Crash | hostname_RP_SlotNumber_ppc_linux_iosd-_ProcessID.core.gz | Router_RP_0_ppc_linux_iosd-_17407.core.gz |
SPA Driver Crash | hostname_SIP_SlotNumber_mcpcc-lc-ms_ProcessID.core.gz | Router_SIP_1_mcpcc-lc-ms_6098.core.gz |
IOS XE Process Crash | hostname_FRU_SlotNumber_ProcessName_ProcessID.core.gz | Router_RP_0_fman_rp_28778.core.gz Router_ESP_1_cpp_cp_svr_4497.core.gz |
Cisco QFP Crash | hostname_ESP_SlotNumber_cpp-mcplo-ucode_ID.core.gz | Router_ESP_0_cpp-mcplo-ucode_042308082102.core.gz |
Linux Kernel Crash | hostname_FRU_SlotNumber_kernel.core | Router_ESP_0_kernel.core |
Note: FRU stands for Field Replaceable Unit and could be RP/ESP/SIP. Depending on which module the crash happens the file name for IOS-XE crash and Kernel crash will vary. Other 3 type of Crashes have fixed syntax where module name does not vary.
Data collection:
1- Crashinfo file – Available in bootflash: or harddisk: and can be exported via TFTP/FTP using the command “copy bootflash: TFTP:” or “copy bootflash: FTP:”. Given that TFTP/FTP IP address is reachable from the router.
2- Core file – Available in bootflash: or harddisk: and can be exported via TFTP/FTP using the command “copy bootflash: TFTP:” or “copy bootflash: FTP:”. Given that TFTP/FTP IP address is reachable from the router.
3- Tracelogs – There could be too many tracelog files in your bootflash: which makes it difficult to extract one at a time. you can use the command syntax: “archive tar /create bootflash:/<filename>.tar bootflash:/tracelogs/” which will generate a “.tar” in the bootflash that has all the tracelogs in it and can be extracted over TFTP/FTP at once.
example: archive tar /create bootflash:/TACTRACELOGS.tar bootflash:/tracelogs/
4- show tech – This would help TAC to correlate available data and configuration with any known crashes previously reported on that platform. For example, certain crashes occur only when certain configuration is present on the router or only when certain stats reflect increasing counter values.
5- Syslogs/console from the time of issue