Symptom

Diagnostic test TestUnusedPortLoopback/TestPortTxMonitoring running on the background every 90sec can trigger different issues such as: - TCAM being filled up with diag/GOLD MAC addresses (i.e. f0e0.d0f0.e0d0 and 0002.0304.0506): #sh plat hard fed act fwd res tcam util CAM Utilization for ASIC [0] Table Max Values Used Values -------------------------------------------------------------------------------- Unicast MAC addresses 65536/1024 57/1023 <<<<< L3 Multicast entries 16384/1024 0/9 L2 Multicast entries 16384/1024 0/11 Directly or indirectly connected routes 49152/65536 1524/4394 #sh plat hard fed act fwd-asic resource tcam table l2 asic 0 reg 1 Printing entries for region L2_CONTROL (1) type 3 asic 0 ======================================================== OftTable 0 Index 1 (A:0,C:0) Valid KEY - vlan:0 mac:0xf0e0d0f0e0d0 l3_if:1 gpn:0 epoch:0 static:0 flood_en:0 vlan_lead_wless_flood_en: 0 client_home_asic: 0 learning_peerid 0, learning_peerid_valid 0 MASK - vlan:0 mac:0xffffffffffff l3_if:1 gpn:0 epoch:0 static:0 flood_en:0 vlan_lead_wless_flood_en: 0 client_home_asic: 0 learning_peerid 0, learning_peerid_valid 0 SRC_AD - need_to_learn:0 lrn_v:0 catchall:0 static_mac:0 chain_ptr_v:0 chain_ptr: 0 static_entry_v:0 auth_state:0 auth_mode:0 auth_behavior_tag:0 traf_m:0 is_src_ce:0 DST_AD - si:0x9e bridge:0 replicate:0 blk_fwd_o:1 v4_rmac:0 v6_rmac:0 catchall:0 ign_src_lrn:0 port_mask_o:0 afd_cli_f:0 afd_lbl:0 prio:1 dest_mod_idx:0 destined_to_us:0 pv_trunk:0 smr:0 OftTable 0 Index 2 (A:0,C:0) Valid KEY - vlan:0 mac:0xf0e0d0f0e0d0 l3_if:0 gpn:0 epoch:0 static:0 flood_en:0 vlan_lead_wless_flood_en: 0 client_home_asic: 0 learning_peerid 0, learning_peerid_valid 0 MASK - vlan:0 mac:0xffffffffffff l3_if:0 gpn:0 epoch:0 static:0 flood_en:0 vlan_lead_wless_flood_en: 0 client_home_asic: 0 learning_peerid 0, learning_peerid_valid 0 SRC_AD - need_to_learn:0 lrn_v:0 catchall:0 static_mac:0 chain_ptr_v:0 chain_ptr: 0 static_entry_v:0 auth_state:0 auth_mode:0 auth_behavior_tag:0 traf_m:0 is_src_ce:0 DST_AD - si:0x9e bridge:0 replicate:0 blk_fwd_o:1 v4_rmac:0 v6_rmac:0 catchall:0 ign_src_lrn:0 port_mask_o:0 afd_cli_f:0 afd_lbl:0 prio:1 dest_mod_idx:0 destined_to_us:0 pv_trunk:0 smr:0 OftTable 0 Index 3 (A:0,C:0) Valid KEY - vlan:0 mac:0x203040506 l3_if:1 gpn:0 epoch:0 static:0 flood_en:0 vlan_lead_wless_flood_en: 0 client_home_asic: 0 learning_peerid 0, learning_peerid_valid 0 MASK - vlan:0 mac:0xffffffffffff l3_if:1 gpn:0 epoch:0 static:0 flood_en:0 vlan_lead_wless_flood_en: 0 client_home_asic: 0 learning_peerid 0, learning_peerid_valid 0 SRC_AD - need_to_learn:0 lrn_v:0 catchall:0 static_mac:0 chain_ptr_v:0 chain_ptr: 0 static_entry_v:0 auth_state:0 auth_mode:0 auth_behavior_tag:0 traf_m:0 is_src_ce:0 DST_AD - si:0x9e bridge:0 replicate:0 blk_fwd_o:1 v4_rmac:0 v6_rmac:0 catchall:0 ign_src_lrn:0 port_mask_o:0 afd_cli_f:0 afd_lbl:0 prio:1 dest_mod_idx:0 destined_to_us:0 pv_trunk:0 smr:0 - Interfaces setting the Loopback bit forever and never getting cleared. show controll eth Te2/1/0/21 phy detail Te2/1/0/21 (port: 21) ---------------------------------------------------- 0000 : 0041 Control Register : 0000 0000 0100 0001 <<< Bit 0 - Local loopback set 0001 : 0082 Control STATUS : 0000 0000 1000 0010 0002 : ae02 Phy ID 1 : 1010 1110 0000 0010 0003 : 53d0 Phy ID 2 : 0101 0011 1101 0000 0004 : 0111 Auto-Negotiation Advertisement : 0000 0001 0001 0001 0005 : 008a Auto-Negotiation Link Partner : 0000 0000 1000 1010 0006 : 0000 Auto-Negotiation Expansion Reg : 0000 0000 0000 0000 0007 : 000d Next Page Transmit Register : 0000 0000 0000 1101 0008 : bbe1 Link Partner Next page Register : 1011 1011 1110 0001 - Crash while invaliding the TCAM for invalid TCAM entry - timing issue.

Conditions

TestUnusedPortLoopback and TestPortTxMonitoring running recurrently as health monitoring diag test. show diagnostic content switch 2 module 1 switch 2 module 1: Diagnostics test suite attributes: X/* - Not a health monitoring test / NA A/I - Monitoring is active / Monitoring is inactive Test Interval Thre- ID Test Name Attributes day hh:mm:ss.ms shold ==== ================================== ============ =============== ===== 1) TestGoldPktLoopback -------------> *BPN*X**I not configured n/a 2) TestPhyLoopback -----------------> *BPD*X**I not configured n/a 3) TestThermal ---------------------> *B*N****A 000 00:01:30.00 1 4) TestScratchRegister -------------> *B*N****A 000 00:01:30.00 5 5) TestUnusedPortLoopback ----------> *BPN****A 000 00:01:30.00 1 <<<< 6) TestPortTxMonitoring ------------> *BPN****A 000 00:01:15.00 1 <<<<

Workaround

Disable diagnostic tests for all modules: no diagnostic monitor switch 1 module 1 test TestUnusedPortLoopback no diagnostic monitor switch 1 module 1 test TestPortTxMonitoring Save configuration, and then reload, to clear any TCAM/Loopback issue.

Further Problem Description