Symptom

MCP crashes are seen on ACI leaf switches running 4/2(6). A core file is also generated: leaf# show cores VDC Module Instance Process-name PID Date(Year-Month-Day Time) --- ------ -------- --------------- -------- ------------------------- 1 1 1 mcp 20081 02/13/2021 19:17:40

Conditions

The version running is 4.2(6). MCP per VLAN is being used across many interfaces on the leaf switch. Run the following command on the switch to see the total number of VLANs with MCP enabled per port: show mcp internal info interface all | grep "Number of VLANS in MCP packets are sent" For example: show mcp internal info interface all | grep "Number of VLANS in MCP packets are sent" Number of VLANS in MCP packets are sent: 256 Number of VLANS in MCP packets are sent: 256 Number of VLANS in MCP packets are sent: 256 Number of VLANS in MCP packets are sent: 256 This means there are 256* 4 ports = 1024 VLANs enabled. The limit in the scalability guide is 2000 as of 5.2. If MCP scale numbers for the running version is exceeded, the likelihood of switch crash increases. Running outside of scale numbers is unsupported.

Workaround

If the total MCP scale is below valid scalability numbers for the running version (2000 per leaf as of 5.2), contact Cisco TAC to apply a workaround. If the total MCP scale is above documented scalability limits (2000 per leaf as of 5.2), The running configuration is beyond scale and is currently in an unsupported state. The first step towards remediation is to disable per port VLAN MCP to bring it within supported scale numbers. fabric —> access policies —> Policies —> global —> MCP instance default Instead of disabling it entirely, new MCP policies with state "Enabled" can be created and applied to specific interfaces where MCP should be enabled. The default MCP policy should then be set to disabled so that it's not implicitly enabled on all interfaces. This should help to reduce the scale. In all cases, configuration should remain within documented scale numbers for the running release to ensure system stability: https://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html

Further Problem Description