Symptom
ACL configured to allow traffic but traffic is either dropped or "ICMP unreachable" is being received.
Conditions
Has been seen with large ACLs on the device and high TCAM usage for these ACLs.
Standalone:
show platform software access-list F0 summary
show platform software fed active acl usage
show platform hardware fed active fwd-asic resource tcam utilization
VSS:
show platform software access-list switch active F0 summary
show platform software fed switch active acl usage
show platform hardware fed switch active fwd-asic resource tcam utilization
Can verify drops with a fed packet capture or a PSV Packet Tracer:
https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xe-gibraltar-16121/216746-configure-punt-inject-fed-packet-capture.html
https://community.cisco.com/t5/networking-documents/use-serviceability-features-to-troubleshoot-your-cat9k-as-a/ta-p/4190774?utm_medium=Referral&utm_source=Slides&utm_campaign=CL_Catalyst9k&utm_term=CiscoCommunity,%20Networking,%20Catalyst&attachment-id=195000
Workaround
Taking impacted ACE in the ACL and changing the sequence number corrects this problem but can not put the ACE back in it's original sequence without a reload.
Further Problem Description
Issue is tied to TCAM programming being incorrect in some rare scenarios. Software level commands will report the flow(s) as allowed but at a TCAM hardware programming level the values will be incorrect.