Symptom

Log snip : +++ 09:59:22 ng-rp-auto-1-r1 exec +++ show platform Node Type PLIM State Config State ----------------------------------------------------------------------------- 0/0/CPU0 LC GE IOS XR RUN PWR,NSHUT,MON 0/RP0/CPU0 RP(Active) N/A IOS XR RUN PWR,NSHUT,MON 0/RP1/CPU0 RP(Standby) N/A IOS XR RUN PWR,NSHUT,MON RP/0/RP0/CPU0:ng-rp-auto-1-r1# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + step1: Verify standby RP ready on router ng-rp-auto-1-r1 + ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2013-07-10 09:59:22: DEBUG:enaExecCLI: Entering enaExecCLI with args ng-rp-auto-1-r1 show redundancy -mode exec -array out_arr -ignore {} 2013-07-10 09:59:22: INFO: enaExecCLI Executing 'show redundancy' on router ng-rp-auto-1-r1, mode -enable (prev - enable) +++ 09:59:22 ng-rp-auto-1-r1 exec +++ show redundancy Redundancy information for node 0/RP0/CPU0: ========================================== Node 0/RP0/CPU0 is in ACTIVE role Partner node (0/RP1/CPU0) is in STANDBY role Standby node in 0/RP1/CPU0 is ready Standby node in 0/RP1/CPU0 is NSR-ready Reload and boot info ---------------------- RP reloaded Wed Jul 10 08:23:00 2013: 36 minutes ago Active node booted Wed Jul 10 08:23:00 2013: 36 minutes ago Standby node boot Wed Jul 10 08:23:00 2013: 36 minutes ago Standby node last went not ready Wed Jul 10 08:51:23 2013: 8 minutes ago Standby node last went ready Wed Jul 10 08:51:23 2013: 8 minutes ago Standby node last went not NSR-ready Wed Jul 10 08:51:34 2013: 7 minutes ago Standby node last went NSR-ready Wed Jul 10 08:51:54 2013: 7 minutes ago There have been 0 switch-overs since reload Active node reload "Initiating switch-over." Standby node reload "Initiating switch-over." RP/0/RP0/CPU0:ng-rp-auto-1-r1# --- 09:59:23 --- 2013-07-10 09:59:23: DIAG: ****************************************************************************** * PASS:Standby RP ready * ****************************************************************************** ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + step2: Do OIR card 0/RP0 on router ng-rp-auto-1-r1 + ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2013-07-10 09:59:23: DEBUG:enaExecCLI: Entering enaExecCLI with args ng-rp-auto-1-r1 hw-module location 0/RP0 reload force -mode adminexec -array out_array 2013-07-10 09:59:23: INFO: enaExecCLI Executing 'hw-module location 0/RP0 reload force' on router ng-rp-auto-1-r1, mode -admin (prev - enable) +++ 09:59:23 ng-rp-auto-1-r1 adminexec +++ admin ^M^M root connected from 127.0.0.1 using console on xr-vm_node0_RP0_CPU0 ^[[?7hcalvados-vm:0_RP0# --- 09:59:24 --- +++ 09:59:24 ng-rp-auto-1-r1 adminexec +++ hw-module location 0/RP0 reload force Wed Jul 10 13:59:21.500 UTC Reload hardware module ? [no,yes] yes result Card reload request on 0/RP0 succeeded. calvados-vm:0_RP0# --- 09:59:24 --- +++ 09:59:24 ng-rp-auto-1-r1 enable +++ exit Wed Jul 10 13:59:21.194 UTC RP/0/RP0/CPU0:ng-rp-auto-1-r1# --- 09:59:24 --- 2013-07-10 09:59:24: DIAG: ****************************************************************************** * PASS:Card 0/RP0 OIR successfully * ****************************************************************************** 2013-07-10 09:59:24: INFO: enaAfter: waiting for 300000 ms: Waiting time for card OIR +++ 10:04:35 ng-rp-auto-1-r1 exec +++ show platform Node Type PLIM State Config State ----------------------------------------------------------------------------- 0/0/CPU0 LC GE IOS XR RUN PWR,NSHUT,MON 0/RP0/CPU0 RP(Active) N/A IOS XR RUN PWR,NSHUT,MON 0/RP1/CPU0 RP(Standby) N/A IOS XR RUN PWR,NSHUT,MON RP/0/RP0/CPU0:ng-rp-auto-1-r1# [[?7hcalvados-vm:0_RP1# --- 10:08:08 --- +++ 10:08:08 ng-rp-auto-1-r1 adminexec +++ show platform Wed Jul 10 14:08:04.952 UTC Location Card Type HW State SW State Config State ---------------------------------------------------------------------------- 0/RP0 NC6-RP OPERATIONAL SW_INACTIVE NSHUT 0/RP1 NC6-RP OPERATIONAL OPERATIONAL NSHUT 0/FC0 NC6-FC OPERATIONAL N/A NSHUT 0/FC2 NC6-FC OPERATIONAL N/A NSHUT 0/FC3 NC6-FC OPERATIONAL N/A NSHUT 0/FT1 P-L-FANTRAY OPERATIONAL N/A NSHUT 0/0 PROTO-CXP-2XPITA OPERATIONAL OPERATIONAL NSHUT 0/PT1 PROTO-AC-PWRTRAY OPERATIONAL N/A NSHUT log for reference : http://ott-pixr1.cisco.com/cgi-bin/hfr-mpls/auto-view.php?file=/auto/ng_ott_auto/DEVELOPMENT_USERS//dmanohar/runinfo/panini_rp_rpfo_ha.2013Jul10_09:38:38

Conditions

Workaround

More Info

Collected the traces : ott2lab-as1:134> pwd /auto/ng_ott_auto/DEVELOPMENT_USERS/dmanohar/ena-tests/Platform-NG/chassis/rp/oir ott2lab-as1:135> ls -ltr total 616096 -rw-r--r-- 1 dmanohar eng 611840 Jul 10 10:39 RP0_sysmgr.log -rw-r--r-- 1 dmanohar eng 606331 Jul 10 10:39 RP1_sysmgr.log -rw-r--r-- 1 jiemiwan eng 297671 Jul 10 10:41 LC0_sysmgr.log -rw-r--r-- 1 jiemiwan eng 9942 Jul 10 10:42 RP0_syslog.local -rw-r--r-- 1 jiemiwan eng 9354310 Jul 10 10:43 RP0_fsdb.trc -rw-r--r-- 1 jiemiwan eng 25375397 Jul 10 10:43 RP0_ccc.trc -rw-r--r-- 1 jiemiwan eng 2085974 Jul 10 10:43 RP0_fgid.trc -rw-r--r-- 1 jiemiwan eng 46209202 Jul 10 10:43 RP0_shelf.trc -rw-r--r-- 1 jiemiwan eng 4451298 Jul 10 10:43 RP0_sdr.trc -rw-r--r-- 1 jiemiwan eng 34611861 Jul 10 10:44 RP0_cm.trc -rw-r--r-- 1 jiemiwan eng 15116941 Jul 10 10:44 RP0_pm.trc -rw-r--r-- 1 jiemiwan eng 3458533 Jul 10 10:44 RP0_vmm.trc -rw-r--r-- 1 jiemiwan eng 7202734 Jul 10 10:44 RP0_wdmon.trc -rw-r--r-- 1 jiemiwan eng 10163969 Jul 10 10:44 RP0_sfe.trc -rw-r--r-- 1 jiemiwan eng 15465 Jul 10 10:47 RP1_syslog.local -rw-r--r-- 1 jiemiwan eng 6450611 Jul 10 10:47 RP1_fsdb.trc -rw-r--r-- 1 jiemiwan eng 28164882 Jul 10 10:47 RP1_ccc.trc -rw-r--r-- 1 jiemiwan eng 2272780 Jul 10 10:47 RP1_fgid.trc -rw-r--r-- 1 jiemiwan eng 45718314 Jul 10 10:47 RP1_shelf.trc -rw-r--r-- 1 jiemiwan eng 4083281 Jul 10 10:47 RP1_sdr.trc -rw-r--r-- 1 jiemiwan eng 31241258 Jul 10 10:48 RP1_cm.trc -rw-r--r-- 1 jiemiwan eng 14946631 Jul 10 10:48 RP1_pm.trc -rw-r--r-- 1 jiemiwan eng 3468900 Jul 10 10:48 RP1_vmm.trc -rw-r--r-- 1 jiemiwan eng 7315322 Jul 10 10:48 RP1_wdmon.trc -rw-r--r-- 1 jiemiwan eng 10802277 Jul 10 10:48 RP1_sfe.trc -rw-r--r-- 1 dmanohar eng 13289 Jul 10 10:56 show_reboot.txt -rw-r--r-- 1 dmanohar eng 3250 Jul 10 10:57 show_sdr.txt ott2lab-as1:136> SM PD Team , can you please take a look at this , I see SM PI requested for board reload , have not seen any ack on this , don't see any BOARD reload requests in CCC , just see reset history buffer changes at around this time. hw-module location 0/RP0 reload force Wed Jul 10 13:59:21.500 UTC Reload hardware module ? [no,yes] yes result Card reload request on 0/RP0 succeeded. bash-3.2$ cat RP1_shelf.trc | grep -i PD-ACTION-RESET 13.59.29.722480512:1742:calvados/system_pkg/shelf_mgr/rack/rm_card_hw_handlers.c:234:rm_card_hw_handlers_235:shelf_mgr:RM: PD-ACTION-RESET rm_card_hw_handle_power_reset for slot_num=0 13.59.29.722915200:1742:calvados/system_pkg/shelf_mgr/rack/rm_card_reload_handlers.c:267:rm_card_reload_handlers_267:shelf_mgr:RM: rm_card_rel_done for slot_num=0 13.59.43.797067392:1742:calvados/panini_pkg/system/shelf_ctrl/lib/sctrl_inv_cpmi_client.c:1895:SCTRL_MAIN_CPMI_12_0:sctrl_inv_set_hw_state_cb(): hw state request for slot: 1. state:5 13.59.43.797069824:1742:calvados/panini_pkg/system/shelf_ctrl/lib/sctrl_inv_cpmi_client.c:1931:SCTRL_ERR_CPMI_12_ZZZ0:sctrl_inv_set_hw_state_cb: slot 1 req id 0 state 5 13.59.43.846832768:1742:calvados/panini_pkg/system/shelf_ctrl/lib/sctrl_inv_cpmi_client.c:1895:SCTRL_MAIN_CPMI_12_0:sctrl_inv_set_hw_state_cb(): hw state request for slot: 16. state:5 13.59.43.846834688:1742:calvados/panini_pkg/system/shelf_ctrl/lib/sctrl_inv_cpmi_client.c:1931:SCTRL_ERR_CPMI_12_ZZZ0:sctrl_inv_set_hw_state_cb: slot 16 req id 1 state 5