Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
1. QOS related show commands hang and return no output # show policy-map interface Bundle-Ether900 Mon May 9 07:39:34.162 UTC Bundle-Ether900 direction input: Service Policy not installed Bundle-Ether900 direction output: Error obtaining statstics: 'sysdb' detected the 'warning' condition 'An EDM took too long to process a request and was timed out' #show policy-map targets Mon May 9 07:44:12.162 UTC No targets found 2. It's not possible to remove or add any policy-map to the configuration. !!% The process 'qos_ma' took too long to respond to a verification request and was timed out ! 3. Following log messages appear: LC/0/0/CPU0:Apr 20 16:03:15.759 UTC: sysdb_svr_local[187]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM request for 'oper/qosea/node/8220/flow-qos/summary' from 'qos_ea_show_interface' (jid 69007, node 0/RP0/CPU0). No response from 'qos_ma_ea' (jid 342, node 0/0/CPU0) within the timeout period (100 seconds) RP/0/RP0/CPU0:Apr 20 16:14:34.880 UTC: sysdb_svr_local[212]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM request for 'oper/qos_ma/node/20/stats/after_clear/if/Bundle-Ether200/dir/output/' from 'qos_ma_show_stats' (jid 65574, node 0/RP0/CPU0). No response from 'qos_ma' (jid 1275, node 0/RP0/CPU0) within the timeout period (100 seconds)
On ASR9903 running 7.1.3 issue has been reproduced with following command sequence: 1. show qos flow-aware summary - takes some time to return output and returns following: #show qos flow-aware summary Mon May 9 07:31:30.936 UTC Node: 0/0/CPU0 -------------------------------------------- Flow QoS Summary not retrieved. Check if the server (qos_ma_ea) is spawned on the LC. 2. qos_ma gets blocked by ifmgr on LC0 upon either "show policy-map interface Bundle-Ether" command is issued If service-policy is applied on that BE interface or during commit which supposed to add service-policy under BE, which is even worth cause commit stucks for an undefined period. # show policy-map interface Bundle-Ether900 Mon May 9 07:39:34.162 UTC Bundle-Ether900 direction input: Service Policy not installed Bundle-Ether900 direction output: Error obtaining statstics: 'sysdb' detected the 'warning' condition 'An EDM took too long to process a request and was timed out'
process restart qos_ma_ea loc 0/0/CPU0
RP/0/RP1/CPU0:ASR-9903-B#show im status Mon May 9 08:19:12.576 UTC View: OWN - Owner, L3P - Local 3rd Party, G3P - Global 3rd Party, LDP - Local Data Plane GDP - Global Data Plane, RED - Redundancy, UL - UL Op: IFC - intf-create, IFD - intf-delete, CADD - caps-add, CREM - caps-remove BC - basecaps, CNSU - caps ns update, ATTR - attr-change, IDAT - init-data REPL - repl-ul-intf, REPL - repl-nodeid, REPL - repl-wildcard, SYNC - resync REG - registration, CFG - cfg-change, ACTV - act-virtual, LOOK - lookup OCHN - owner-channel, ISSU - issu-recreation, Clients that IM is waiting for. These clients may be preventing the completion of an in-progress operation and should be investigated further. |Node | JID | Process Name | Op | Wait time,s | FS | -----|--------------------|-----|--------------------|----|-------------|----| 1 0/0/CPU0 237 qos_ma_ea CADD 0:15:35 Clients with IM operations in progress. These processes are expected to be blocked on IM until the operation can complete. The 'Waiting for' column references the table above. Node | JID | Process Name |Op | Wait time,s | Waiting for | --------------------|-----|--------------------|----|-------------|------------------------------------------------------------| 0/RP1/CPU0 1304 qos_ma CADD 0:15:35 1 Error conditions: ID | Count | Description & First Instance | --------|-------|--------------------------------------------------------------------------------------------------------------| ops.01 1 Client with blocked ops Client qos_ma on node 0/RP1/CPU0 is waiting for client qos_ma_ea on node 0/0/CPU0 (flow id 0x001-2f60) Informational conditions. Note that these may not represent a problem or the system may have since recovered: ID | Count | Description & First Instance | --------|-------|--------------------------------------------------------------------------------------------------------------| conn.04 1 Disconnected client in ZOMBIE state Node 0/0/CPU0, JID 237, cb handle 0x001e4060 (LDP 3P - ID 242), proc name LDP 3P - ID 242 trace.02 12 EA returned an error to an IM operation Node 0/0/CPU0, cb handle 0x80000060, proc name netio, rd id 0x002-01f2, flow id 0x002-01ed, op DPC, purpose DLD_DPC, error: 'Subsystem(2)' detected the 'success' condition 'Code(0)': Operation not permitted trace.04 33 Resources deleted as part of resync Node 0/0/CPU0, timestamp 145d 05h trace.06 66 Download nodeset inconsistency hit during resync. Invalid notifications may have been sent to clients Node 0/RP1/CPU0, timestamp 4d 57h stats.01 20 IM client involved in failed operations Node 0/RP0/CPU0, proc name igmp stats.02 7 Client started an operation that blocked for over 30 secs Node 0/RP0/CPU0, proc name eint_ma, purpose OP_OWNER_CHAN, time 0:00:32 stats.03 14 Client blocked for more than 30 secs on one operation Node 0/RP0/CPU0, proc name ipv4_ma, purpose OP_OWNER_CHAN, time 0:00:32 stats.04 6 Possible unbulked client Node 0/0/CPU0, proc name cdp, purpose OP_REG, op count 307 Checking connections and node states.........OK Checking for unfinished operations.........FAIL Checking for trace errors....................OK Checking for statistical anomalies...........OK ----------------------------------------------- Overall result.............................FAIL RP/0/RP1/CPU0:ASR-9903-B#show processes blocked location all Mon May 9 08:20:03.794 UTC node: node0_0_CPU0 ------------------------------------------ Jid Pid Tid ProcessName State TimeInState Blocked-on node: node0_RP0_CPU0 ------------------------------------------ Jid Pid Tid ProcessName State TimeInState Blocked-on 156 7250 7392 lpts_fm Reply 0687:13:41.0804 6201 node0_RP1_CPU0 PID:6201 node: node0_RP1_CPU0 ------------------------------------------ Jid Pid Tid ProcessName State TimeInState Blocked-on 0 3364 3364 sh_proc_ng_blocked Reply 0000:00:00.0210 5091 procfs_server 0 31420 31420 config Reply 0000:16:27.0068 5078 sysdb_mc 1304 7317 7317 qos_ma Reply 0000:16:26.0882 6198 ifmgr 379 7288 7431 lpts_fm Reply 0000:00:04.0526 6201 lpts_pa