OPERATIONAL DEFECT DATABASE
...
...
Trying to allocate interfaces from M-series I/O module into production VDC after 10 minutes timeout this operation may fail with: 2018 Mar 27 14:02:56.278 RTR %$ VDC-2 %$ %VMM-2-VMM_SERVICE_ERR: VDC2: Service SAP Qosmgr SAP for slot 4 returned error 0x41170014 (Operation timed out) in if_bind sequence vdc_id: 2 vdc_name: RTR interfaces: Port Status ---- ---------- Eth3/1 OK Eth3/2 OK ... Eth3/22 OK Eth3/23 OK Eth3/24 OK Eth4/1 ERROR:Operation timed out (0x41170014) Eth4/2 ERROR:Operation timed out (0x41170014) Eth4/3 ERROR:Operation timed out (0x41170014) Eth4/4 ERROR:Operation timed out (0x41170014) Eth4/5 ERROR:Operation timed out (0x41170014) Eth4/6 ERROR:Operation timed out (0x41170014) Eth4/7 ERROR:Operation timed out (0x41170014) Eth4/8 ERROR:Operation timed out (0x41170014) Eth4/9 ERROR:Operation timed out (0x41170014) Eth4/10 ERROR:Operation timed out (0x41170014) Eth4/11 ERROR:Operation timed out (0x41170014) ... RTR2712# show system internal mts buffers details **Fast Sap Buffers are not displayed below** Node/Sap/queue Age(ms) SrcNode SrcSAP DstNode DstSAP OPC MsgId MsgSize RRToken Offset sup/4765/nper 215691 0x201 357 0x201 4765 7679 0xb41e 224 0x8000b41d 0xffdc404 sup/284/pers 1630019 0x201 4487 0x201 284 86017 0x2c92 4608 0x2c92 0xfab2004 sup/284/pers 222 0x201 4751 0x201 284 86017 0xbee3 4608 0xbee3 0xffde004 sup-1/284/pers 1453369 0x205 14798 0x205 284 86017 0x4e47 4608 0x4e47 0xffd8004 sup-1/284/pers 612 0x205 14799 0x205 284 86017 0x13263 4608 0x13263 0xfab0004 sup-1/351/log 1069009 0x205 377 0x205 0 6484 0xe6e9 12110 0 0xfabc004 <===
The exact conditions are not known so far, however: - this issue is constantly and easily reproducible with M2 I/O modules with specific running-configuration. 2018 Mar 27 14:00:26.267 RTR %IPQOSMGR-4-QOSMGR_LC_ERROR_MSG: Linecard 1 returned an error: Operation timed out 2018 Mar 27 14:02:56.277 RTR %IPQOSMGR-4-QOSMGR_LC_ERROR_MSG: Linecard 1 returned an error: Operation timed out 2018 Mar 27 14:02:56.278 RTR %VMM-2-VMM_SERVICE_ERR: VDC2: Service SAP Qosmgr SAP for slot 4 returned error 0x41170014 (Operation timed out) in if_bind sequence Looks like this issue is driven by "ACLMgr" component and thus it depends on certain ACL configuration, you might use in your environment. It might be ACLs for route-map, table-map, QoS, PACL, RACL... Further investigation is needed... RTR# show system internal mts buffers details ... sup-1/351/log 6258260 0x105 377 0x205 0 6484 0xe6e9 12110 0 0xfaac004 sup-1/351/log 79718 0x105 14933 0x105 351 15462 0x3c37 1155 0x3c37 0xfab2804 sup-1/351/log 59997 0x105 28 0x105 351 15460 0x960c 118 0x960c 0xfa45c04 sup-1/351/log 55003 0x105 28 0x105 351 15460 0x970b 118 0x970b 0xfab8e04 sup-1/351/log 49718 0x105 18417 0x105 351 15462 0x97a9 2084 0x97a9 0xfa2e004 RTR2# show system internal mts node sup-1 sap 351 description Aclmgr SAP
++> please, use "ascii" reload to rebuild PSS from scratch and do an ASCII replay: RTR# copy ru st vd [########################################] 100% Copy complete. RTR# reload ascii !!!WARNING! This command will erase binary configuration across all VDCs and reboot the system with ascii configuration. Do you wish to proceed anyway? (y/n) [n] <== y
++> SSO will not help: 2018 Mar 27 16:01:53.661998 sysmgr: process_ennvar_non_sysmgr_srv_get: received request to get ennvar for non sysmgr srv on vdc 1 2018 Mar 27 16:01:53.662108 sysmgr: process_ennvar_non_sysmgr_srv_get: sending response back to srv 2018 Mar 27 16:01:54.998 RTR %$ VDC-1 %$ %SYSMGR-2-GSYNC_SNAPSHOT_SRVFAILED: Service "aclmgr" on active supervisor failed to store its snapshot (error-id 0x801E003E). 2018 Mar 27 16:01:55.008863 sysmgr: bury_child: Service name: System Manager (gsync controller), Pid: 22929, exit code: 256, cwd: /var/sysmgr/work, core dump = 0 2018 Mar 27 16:01:55.062799 sysmgr: fsm_action_become_active: local ip addr = 0x105017f for vdc 2 2018 Mar 27 16:01:55.062968 sysmgr: fsm_action_become_active: setting mts_set_other_addr for act sup, vnode 1 to 0x105, ip = 0x105017f for vdc 2, ret_val = 0x0 2018 Mar 27 16:01:55.124 RTR %$ VDC-1 %$ %SYSMGR-STANDBY-2-SHUTDOWN_SYSTEM_LOG: vdc 2 will shut down soon. ++> "vdc reload" will not help: RTR# reload vdc RTR Are you sure you want to reload this vdc (y/n)? [no] yes RTR# show vdc Switchwide mode is m1 f1 m1xl f2 m2xl f2e f3 vdc_id vdc_name state mac type lc ------ -------- ----- ---------- --------- ------ 1 PTR active e4:c7:22:07:a9:c1 Admin None 2 RTR suspend in progress e4:c7:22:07:a9:c2 Ethernet m1 m1xl m2xl f2e ++> "binary"/normal reload will not help as well
Click on a version to see all relevant bugs
Cisco Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
