OPERATIONAL DEFECT DATABASE
...
...
The interface reliability drops to 254/255 and does not recover on its own.
Issue seen during regular operation
Workaround: • Clearing the counters of the interface brings the reliability back to 255/255 • This stays at 255/255 for couple of days, but the reliability then comes back to 254/255
Problem Analysis: All the connected interfaces of the switch are showing reliability as 254/255 LEAF# show int Eth1/1 - 36 | grep tx reliability 254/255, txload 105/255, rxload 76/255 reliability 254/255, txload 106/255, rxload 78/255 reliability 254/255, txload 52/255, rxload 72/255 reliability 254/255, txload 54/255, rxload 73/255 reliability 255/255, txload 1/255, rxload 1/255 reliability 255/255, txload 1/255, rxload 1/255 reliability 254/255, txload 56/255, rxload 72/255 reliability 255/255, txload 1/255, rxload 1/255 reliability 254/255, txload 103/255, rxload 76/255 Interface details for once such connected interface: Ethernet1/3 is up admin state is up, Dedicated Interface Belongs to po1 Port description is TO-AMBAPMNRCSR002-Hu0/6/0/2 Hardware: 10000/100000/25000/40000/auto Ethernet, address: 0000.0000.0000 (bia 00ee.ab67.3c29) MTU 9216 bytes, BW 100000000 Kbit, DLY 1 usec reliability 254/255, txload 41/255, rxload 4/255 Encapsulation ARPA, medium is broadcast Port mode is routed full-duplex, 100 Gb/s FEC (forward-error-correction) : cl91-rs-fec Beacon is turned off Auto-Negotiation is turned on Input flow-control is off, output flow-control is off Auto-mdix is turned off Rate mode is dedicated Switchport monitor is off EtherType is 0x8100 EEE (efficient-ethernet) : n/a Last link flapped 00:57:47 Last clearing of "show interface" counters never 50 interface resets 30 seconds input rate 1779333808 bits/sec, 421085 packets/sec 30 seconds output rate 16173039704 bits/sec, 2256115 packets/sec Load-Interval #2: 5 minute (300 seconds) input rate 1812507624 bps, 425607 pps; output rate 16144015392 bps, 2242664 pps L3 in Switched: ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes L3 out Switched: ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes RX 2292065643622 unicast packets 6616470 multicast packets 606 broadcast packets 2292072260698 input packets 1392318124524733 bytes 24437767047 jumbo packets 0 storm suppression bytes 0 runts 0 giants 1 CRC 0 Stomped CRC 0 no buffer 1 input error 0 short frame 0 overrun 0 underrun 0 ignored 0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble 0 input discard 0 input buffer drop 0 input total drop 0 Rx pause TX 9753836893732 unicast packets 664511 multicast packets 5860508 broadcast packets 9753843418747 output packets 8823621475694699 bytes 63545619884 jumbo packets 9 output error 0 collision 0 deferred 0 late collision 0 lost carrier 0 no carrier 0 babble 0 output discard 0 output buffer drops 0 output total drops 0 Tx pause eth-1/3 9 Total 2292063676382 1392313476534240 9753798089589 8823580698777018 Unicast 2291337228798 1392315416320770 9754326157538 8824048284560471 Multicast 0 0 0 0 Flood 879862 111442452 0 0 Total Drops 849995672 3399484 Storm Drops(bytes) 0 Buffer 0 0 Error 0 3399484 Forward 849995672 LB 0 AFD RED 0 Class 0 Admit 2291335873500 9751237432614 Drop 4036631 3396328 Class 1 Admit 0 38167855 Drop 0 3156 Class 2 Admit 0 2480586860 Drop 0 0 Class 3 Admit 447791 787627 Drop 0 0 Class 4 Admit 0 0 Drop 0 0 Class 5 Admit 0 0 Drop 0 0 Class 6 Admit 0 0 Drop 0 0 Class 7 Admit 0 0 Drop 0 0 Class SUP Admit 847746527 569182582 Drop 0 0 Class 9 Admit 0 0 Drop 0 0 Mac RMON RX_PKTOK 2292063676381 RX_PKTTOTAL 2292063676382 RX_FCS_ERR 1 RX_ANY_ERR 1 RX_OCTETSOK 1392313476534122 RX_OCTETS 1392313476534240 RX_UCAST 2292057059327 RX_MCAST 6616449 RX_BCAST 606 RX_PAUSE 0 RX_INRANGEERR 0 RX_UNDERSIZE 0 RX_OVERSIZE 0 RX_FRAGMENT 0 RX_JABBER 0 RX_USER_PAUSE 0 RX_CRCERR 0 RX_TOOLONG 0 RX_VLAN 2292058005121 RX_DISCARD 0 RX_PKT_LT64 0 RX_PKT_64 506168714 RX_PKT_65 1181936616201 RX_PKT_128 146666563899 RX_PKT_256 34641546207 RX_PKT_512 40087897634 RX_PKT_1024 863787201318 RX_PKT_1519 24437682342 RX_PKT_2048 0 RX_PKT_4096 21 RX_PKT_8192 0 RX_PKT_GT9216 54 TX_PKTOK 9753798089571 TX_PKTTOTAL 9753798089589 TX_FRM_ERROR 9 TX_OCTETSOK 8823580698762157 TX_OCTETS 8823580698777018 TX_UCAST 9753791564585 TX_MCAST 664509 TX_BCAST 5860491 TX_PAUSE 0 TX_USER_PAUSE 0 TX_VLAN 9753797630207 TX_PKT_LT64 1 TX_PKT_64 58152577044 TX_PKT_65 2007742770662 TX_PKT_128 1438060093833 TX_PKT_256 199179949766 TX_PKT_512 199583370745 TX_PKT_1024 5787533991729 TX_PKT_1519 63545316058 TX_PKT_2048 19597 TX_PKT_4096 185 TX_PKT_8192 0 TX_PKT_GT9216 0 TX_PAUSE_PRI0 0 TX_PAUSE_PRI1 0 TX_PAUSE_PRI2 0 TX_PAUSE_PRI3 0 TX_PAUSE_PRI4 0 TX_PAUSE_PRI5 0 TX_PAUSE_PRI6 0 TX_PAUSE_PRI7 0 RX_PAUSE_PRI0 0 RX_PAUSE_PRI1 0 RX_PAUSE_PRI2 0 RX_PAUSE_PRI3 0 RX_PAUSE_PRI4 0 RX_PAUSE_PRI5 0 RX_PAUSE_PRI6 0 RX_PAUSE_PRI7 0 TX_PAUSE_QUANTA_PRI0 0 TX_PAUSE_QUANTA_PRI1 0 TX_PAUSE_QUANTA_PRI2 0 TX_PAUSE_QUANTA_PRI3 0 TX_PAUSE_QUANTA_PRI4 0 TX_PAUSE_QUANTA_PRI5 0 TX_PAUSE_QUANTA_PRI6 0 TX_PAUSE_QUANTA_PRI7 0 RX_PAUSE_QUANTA_PRI0 0 RX_PAUSE_QUANTA_PRI1 0 RX_PAUSE_QUANTA_PRI2 0 RX_PAUSE_QUANTA_PRI3 0 RX_PAUSE_QUANTA_PRI4 0 RX_PAUSE_QUANTA_PRI5 0 RX_PAUSE_QUANTA_PRI6 0 RX_PAUSE_QUANTA_PRI7 0 RX_STANDARD_PAUSE 0 [10/21/2022 3:01:22 PM] Macsec RMON MACSEC_RX_BYTES_VALIDATED 0 MACSEC_RX_BYTES_DECRYPTED 0 MACSEC_RX_PKTS_VALIDATED 0 MACSEC_RX_PKTS_DECRYTPED 0 MACSEC_RX_PKTS_UNUSED_SA 0 MACSEC_RX_PKTS_NOT_USING_SA 0 MACSEC_RX_LATE 0 MACSEC_RX_OK 0 MACSEC_RX_DELAYED 0 MACSEC_RX_UNCHECKED 0 MACSEC_RX_NOTOK 0 MACSEC_RX_INVALID 0 MACSEC_TX_BYTES_PROTECTED 0 MACSEC_TX_BYTES_ENCRYPTED 0 MACSEC_TX_PKTS_PROTECTED 0 MACSEC_TX_PKTS_ENCRYPTED 0 Troubleshooting done so far: • We started with checking the issue on physical layer. • The cable and SFP of one of the interfaces were replaced, the issue persisted. • We proceeded to perform RMA of the switch. The issue persisted, and the reliability was back to 254/255 on all the connected interfaces. Lab repro outcome: • The Lab switch was connected to IXIA device where we could generate packets. • The idea was that if huge number of packets are sent to the interface and there are some drops, then will it lead to reliability going down. • The reliability did not go down whatsoever. We tried with 2 different versions. Below are the logs: Details from customer output: ++ 106982 Class 0 drops. ++ 15166462 Sup redirects, classified as drop. ++ 15273444 total forward drops. bgl-aci05-leaf3# show inter ethernet 1/26 | grep "packets\|reliab" reliability 255/255, txload 1/255, rxload 1/255 30 seconds input rate 0 bits/sec, 0 packets/sec 30 seconds output rate 0 bits/sec, 0 packets/sec 3445970942 unicast packets 0 multicast packets 0 broadcast packets 3445970942 input packets 231944979596 bytes 0 jumbo packets 0 storm suppression bytes 0 unicast packets 428 multicast packets 0 broadcast packets 428 output packets 175928 bytes 0 jumbo packets bgl-aci05-leaf3# vsh_lc -c "show platform internal counters port 26" Stats for port 26 (note: forward drops includes sup redirected packets too) IF LPort Input Output Packets Bytes Packets Bytes eth-1/26 101 Total 3594769852 241468109836 429 176393 Unicast 0 0 429 176393 Multicast 0 0 0 0 Flood 0 0 0 0 Total Drops 3594728489 1 Storm Drops(bytes) 0 Buffer 28020 0 Error 0 1 Forward 3594700469 LB 0 AFD RED 0 Class 0 Admit 0 0 Drop 2850709795 0 Class 1 Admit 0 0 Drop 0 0 Class 2 Admit 0 0 Drop 0 0 Class 3 Admit 0 0 Drop 64 0 Class 4 Admit 0 0 Drop 0 0 Class 5 Admit 0 0 Drop 0 0 Class 6 Admit 0 0 Drop 0 0 Class 7 Admit 0 0 Drop 0 0 Class SUP Admit 744024837 429 Drop 0 0 Class 9 Admit 0 0 Drop 0 0 Mac RMON RX_PKTOK 3594769852 RX_PKTTOTAL 3594769852 RX_FCS_ERR 0 RX_ANY_ERR 0 RX_OCTETSOK 241468109836 RX_OCTETS 241468109836 RX_UCAST 3594769852 RX_MCAST 0 RX_BCAST 0 RX_PAUSE 0 RX_INRANGEERR 0 RX_UNDERSIZE 0 RX_OVERSIZE 0 RX_FRAGMENT 0 RX_JABBER 0 RX_USER_PAUSE 0 RX_CRCERR 0 RX_TOOLONG 0 RX_VLAN 2850709827 RX_DISCARD 0 RX_PKT_LT64 0 RX_PKT_64 744060031 RX_PKT_65 2850709827 RX_PKT_128 0 RX_PKT_256 0 RX_PKT_512 0 RX_PKT_1024 0 RX_PKT_1519 0 RX_PKT_2048 0 RX_PKT_4096 0 RX_PKT_8192 0 RX_PKT_GT9216 0 TX_PKTOK 429 TX_PKTTOTAL 429 TX_FRM_ERROR 0 TX_OCTETSOK 176393 TX_OCTETS 176393 TX_UCAST 0 TX_MCAST 429 TX_BCAST 0 TX_PAUSE 0 TX_USER_PAUSE 0 TX_VLAN 0 TX_PKT_LT64 0 TX_PKT_64 0 TX_PKT_65 0 TX_PKT_128 0 TX_PKT_256 429 TX_PKT_512 0 TX_PKT_1024 0 TX_PKT_1519 0 TX_PKT_2048 0 TX_PKT_4096 0 TX_PKT_8192 0 TX_PKT_GT9216 0 TX_PAUSE_PRI0 0 TX_PAUSE_PRI1 0 TX_PAUSE_PRI2 0 TX_PAUSE_PRI3 0 TX_PAUSE_PRI4 0 TX_PAUSE_PRI5 0 TX_PAUSE_PRI6 0 TX_PAUSE_PRI7 0 RX_PAUSE_PRI0 0 RX_PAUSE_PRI1 0 RX_PAUSE_PRI2 0 RX_PAUSE_PRI3 0 RX_PAUSE_PRI4 0 RX_PAUSE_PRI5 0 RX_PAUSE_PRI6 0 RX_PAUSE_PRI7 0 TX_PAUSE_QUANTA_PRI0 0 TX_PAUSE_QUANTA_PRI1 0 TX_PAUSE_QUANTA_PRI2 0 TX_PAUSE_QUANTA_PRI3 0 TX_PAUSE_QUANTA_PRI4 0 TX_PAUSE_QUANTA_PRI5 0 TX_PAUSE_QUANTA_PRI6 0 TX_PAUSE_QUANTA_PRI7 0 RX_PAUSE_QUANTA_PRI0 0 RX_PAUSE_QUANTA_PRI1 0 RX_PAUSE_QUANTA_PRI2 0 RX_PAUSE_QUANTA_PRI3 0 RX_PAUSE_QUANTA_PRI4 0 RX_PAUSE_QUANTA_PRI5 0 RX_PAUSE_QUANTA_PRI6 0 RX_PAUSE_QUANTA_PRI7 0 RX_STANDARD_PAUSE 0 Macsec RMON MACSEC_RX_BYTES_VALIDATED 0 MACSEC_RX_BYTES_DECRYPTED 0 MACSEC_RX_PKTS_VALIDATED 0 MACSEC_RX_PKTS_DECRYTPED 0 MACSEC_RX_PKTS_UNUSED_SA 0 MACSEC_RX_PKTS_NOT_USING_SA 0 MACSEC_RX_LATE 0 MACSEC_RX_OK 0 MACSEC_RX_DELAYED 0 MACSEC_RX_UNCHECKED 0 MACSEC_RX_NOTOK 0 MACSEC_RX_INVALID 0 MACSEC_TX_BYTES_PROTECTED 0 MACSEC_TX_BYTES_ENCRYPTED 0 MACSEC_TX_PKTS_PROTECTED 0 MACSEC_TX_PKTS_ENCRYPTED 0 bgl-aci05-leaf3# acidiag version 5.2.4d bgl-aci05-leaf3# ++ no change in behavior when the version s changed.
Click on a version to see all relevant bugs
Cisco Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
