Symptoms

Multiple interfaces dropping frames causing serve performance issue to customers environment..MDS switch logging the following PMON alerts:-logging log:PMON-SLOT1-3-RISING_THRESHOLD_REACHED: Credit Loss Reco has reached the rising threshold PMON-SLOT1-3-RISING_THRESHOLD_REACHED: TX Credit Not Available has reached the rising threshold

Cause

Onboard Logging log review:The following IPA errors indicate packets are being corrupted after being received. The module these are occurring on should be replaced`show hardware internal errors`|------------------------------------------------------------------------|| Device:Tbird Que Driver Role:QUE Mod: 1 || Device Statistics Category :: ERROR|------------------------------------------------------------------------|Instance:0Cntr Name Value Ports----- ---- ----- ----- 24 THB_IPA_IPA0_CNT_BAD_CRC 0000000000637620 1-4 - 25 THB_IPA_IPA0_CNT_CORRUPT 0000000000637620 1-4 - The following EBM errors are the same corrupted packets being received at the egress linecard. These will likely occur on several modules that are receiving these corrupted packets. These modules are not faulty. Replace the module generating the IPA errors and these will stop:`show hardware internal errors`|------------------------------------------------------------------------|| Device:Tbird Fwd Driver Role:L2 Mod: 2 || Device Statistics Category :: ERROR|------------------------------------------------------------------------|Instance:1Cntr Name Value Ports----- ---- ----- ----- 300 THB_EBM0_ACC_ERR_QUEUE_DROP_PKT 0000000000000017 9-12 - <-------- 301 THB_EBM0_CNT_ERR_QUEUE_DROP_SF 0000000000000010 9-12 - <--------13414 TBIRD_FWD_EBM_0_PACK0_EPR_DROP 0000000000000001 9-12 - 13422 TBIRD_FWD_EBM_0_PACK1_EPR_DROP 0000000000000005 9-12 - 13430 TBIRD_FWD_EBM_0_PACK2_EPR_DROP 0000000000000001 9-12 - 13438 TBIRD_FWD_EBM_0_PACK3_EPR_DROP 0000000000000003 9-12 - 13727 TBIRD_FWD_EPR0_PKT_CRC_ERR 0000000000000001 9-16 - 13755 TBIRD_FWD_EPR1_PKT_CRC_ERR 0000000000000005 9-16 - 13783 TBIRD_FWD_EPR2_PKT_CRC_ERR 0000000000000001 9-16 - 13811 TBIRD_FWD_EPR3_PKT_CRC_ERR 0000000000000003 9-16 - |------------------------------------------------------------------------|| Device:Tbird Fwd Driver Role:L2 Mod: 3 || Device Statistics Category :: ERROR|------------------------------------------------------------------------|Instance:2Cntr Name Value Ports----- ---- ----- ----- 300 THB_EBM0_ACC_ERR_QUEUE_DROP_PKT 0000000000068016 17-20 - 301 THB_EBM0_CNT_ERR_QUEUE_DROP_SF 0000000000057196 17-20 - |------------------------------------------------------------------------|| Device:Tbird Fwd Driver Role:L2 Mod: 4 || Device Statistics Category :: ERROR|------------------------------------------------------------------------|Instance:0Cntr Name Value Ports----- ---- ----- ----- 300 THB_EBM0_ACC_ERR_QUEUE_DROP_PKT 0000000000004504 1-4 - 301 THB_EBM0_CNT_ERR_QUEUE_DROP_SF 0000000000003598 1-4 - |------------------------------------------------------------------------|| Device:Tbird Fwd Driver Role:L2 Mod: 5 || Device Statistics Category :: ERROR|------------------------------------------------------------------------| 300 THB_EBM0_ACC_ERR_QUEUE_DROP_PKT 0000000000020069 1-4 - 301 THB_EBM0_CNT_ERR_QUEUE_DROP_SF 0000000000019715 1-4 - The EBM errors are the same corrupted packets being received at the egress linecard. These will likely occur on several modules that are receiving these corrupted packets. These modules are not faulty. Replace the module generating the IPA errors and these will stop:Occurs on MDS DS-X9232-256K9 and DS-X9248-256K9 modules only.

Resolution

Fix:Replace the linecard logging the following errors:THB_IPA_IPA0_CNT_BAD_CRC THB_IPA_IPA0_CNT_CORRUPT Note: port-monitor has 3 special counters that can alert on the above condition. Since the default "slowdrain" policy are mostly used these are not monitored .The counters are: err-pkt-from-port err-pkt-to-xbar err-pkt-from-xbar See also for additional information :CSCum09652 Add Gen4 IPA and EFI errors to port-monitor xbar errors

Support Cases