OPERATIONAL DEFECT DATABASE
...
...
Overruns and performance issues observed with crypto configuration enabled on C8500L-8S4X. The throughput degrades over time due to a resource leaks impacting packet reception on some active interfaces. If flow control has been negotiated with devices connected to GigE or TenGigE interfaces, a pause frame generation will be high coming from the C8500L-8S4X. Please observe below an interface affected by this problem: #show int te0/1/0 TenGigabitEthernet0/1/0 is up, line protocol is up 20875 input errors, 0 CRC, 0 frame, 20875 overrun, 0 ignored 0 lost carrier, 0 no carrier, 803 pause output interface TenGigabitEthernet0/1/0 ip address X.X.X.X X.X.X.X interface Tunnel1 ip address X.X.X.X X.X.X.X tunnel source tunnel mode gre multipoint tunnel protection ipsec profile PROFILE-NAME In the example above, the physical outbound interface for Tunnel1 is TenGigabitEthernet0/1/0. IPSEC is enabled under the Tunnel. For further isolation, please follow-up the steps below: Identify the FPE value for the affected interface, in this scenario Te0/1/0 #show platform hardware qfp active data infra bind Port Instance Bindings: ID Port IOS Port WRKR12 WRKR13 1 rcl0 rcl0 Rx Tx 2 ipc ipc Rx+Tx - 3 vxe_punti vxe_puntif Rx+Tx - 4 fpe0 GigabitEthernet0/0/0 Rx+Tx - 5 fpe1 GigabitEthernet0/0/1 Rx+Tx - 6 fpe2 GigabitEthernet0/0/2 - Rx+Tx 7 fpe3 GigabitEthernet0/0/3 Rx+Tx - 8 fpe4 GigabitEthernet0/0/4 - Rx+Tx 9 fpe5 GigabitEthernet0/0/5 Rx+Tx - 10 fpe6 GigabitEthernet0/0/6 - Rx+Tx 11 fpe7 GigabitEthernet0/0/7 Rx+Tx - 12 fpe8 TenGigabitEthernet0/1/0 - Rx+Tx 13 fpe9 TenGigabitEthernet0/1/1 Rx+Tx - 14 fpe10 TenGigabitEthernet0/1/2 - Rx+Tx 15 fpe11 TenGigabitEthernet0/1/3 Rx+Tx - Recollect the data plane PPE Usage, looking at the column "Credits Usage". In this example the interested line for the affected interface is FPE8: ROUTER#show platform hardware qfp active datapath infrastructure sw-cio Credits Usage: ID Port Wght Global WRKR0 WRKR1 WRKR2 WRKR3 WRKR4 WRKR5 WRKR6 WRKR7 WRKR8 WRKR9 WRKR10 WRKR11 WRKR12 WRKR13 WRKR14 WRKR15 Total 1 rcl0 1: 6048 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 6144 1 rcl0 128: 6048 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 6144 2 ipc 1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 vxe_punti 1: 461 0 0 0 0 0 0 0 0 0 0 0 0 51 0 0 0 512 4 fpe0 1: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 4 fpe0 2: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 5 fpe1 1: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 5 fpe1 2: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 6 fpe2 1: 1952 0 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 2048 6 fpe2 2: 1952 0 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 2048 7 fpe3 1: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 7 fpe3 2: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 8 fpe4 1: 1952 0 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 2048 8 fpe4 2: 1952 0 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 2048 9 fpe5 1: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 9 fpe5 2: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 10 fpe6 1: 1952 0 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 2048 10 fpe6 2: 1952 0 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 2048 11 fpe7 1: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 11 fpe7 2: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 12 fpe8 1: 0 0 0 0 0 0 0 0 0 0 0 0 0 3 57 0 0 60 <------------ should be close 2048 12 fpe8 2: 1952 0 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 2048 13 fpe9 1: 1179 0 0 0 0 0 0 0 0 0 0 0 0 86 16 0 0 1281 13 fpe9 2: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 14 fpe10 1: 1155 0 0 0 0 0 0 0 0 0 0 0 0 0 112 0 0 1267 14 fpe10 2: 1952 0 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 2048 15 fpe11 1: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 15 fpe11 2: 1952 0 0 0 0 0 0 0 0 0 0 0 0 96 0 0 0 2048 This is an evidence of leaked credits, at idle (no traffic). Under traffic it is expected that this command shows low values, but at idle we would expect the credit values to return to their original count prior to traffic and subsequent loss.
This issue will occur on C8500L-8S4X platform configured with crypto during periods of increased loads.
The router must be rebooted to recover the lost resources. The reboot can be delayed by moving connections to the ports generating pause frames to previously unused ports, however, the problem might be observed after some time. For a permanent solution, the router software must be upgraded to resolve the issue to 17.6.3.
This loss of packets leads to interface credits being lost, resulting in less packets being processed and decreased performance. The loss of credits can be observed using the IOS exec command `show platform hardware qfp active datapath infrastructure sw-cio` at idle (no traffic). Under traffic it is expected that this command shows low values, but at idle we would expect the credit values to return to their original count prior to traffic and subsequent loss.
Click on a version to see all relevant bugs
Cisco Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
