Symptom

Conditions

Any one of the following condition can cause this issue: 1) Device having more than 700 interfaces configured with " vpc orphan-port suspend " command. 2) Hardware failure on the device.

Workaround

1) Limit the configuration of "vpc orphan-port suspend" to 400. 2) save the configuration. 3) Then reload the switch. Once all the FEX comes back online you can proceed with the ISSU.

Further Problem Description

ISSU fails while upgrading from 7.1(4)N1.1 to 7.3(5)N1.1. sh install all failure-reason Service "vdc_mgr" returned error: VDC_MGR has detected a potential issue and blocked upgrade (0x413C0017)(vdc: 1) Service "vdc_mgr" has sent the following message: VDC is not active. All VDC must be in active state for upgrade. Resume any suspended VDCs and delete or restart all failed VDCs (vdc: 1) Service "stp" has sent the following message: Port: port-channel404 in VLAN1802 is Designated. Topology change could occur during ISSU. Upgrade needs to be disruptive!!! (vdc: 1) Service "vpc" has sent the following message: STP Preupgrade Check failed on VPC peer switch (vdc: 1) Install has failed. VDC_MGR has detected a potential issue and blocked upgrade (0x413C0017) ================================================================================ Install has failed. VDC_MGR has detected a potential issue and blocked upgrade (0x413C0017) Please contact the Customer Support Team ================================================================================ You can see vdc creation failure error messages on the logs: %KERN-2-SYSTEM_MSG: [ 230.761721] mts_do_msg_input() failing since no space available in 450 (src_sap = 450, opc = 106533) - kernel %KERN-2-SYSTEM_MSG: [ 230.761728] node=4 sap=450 rq=419(26660797) lq=0(0) pq=1(6800) nq=0(0) sq=0(0) buf_in_transit=500, bytes_in_transit=26671552 - kernel %VMM-2-VMM_TIMEOUT: VDC1: Service SAP 450 for slot 1 timed out in INSERT_SEQ sequence %IM-3-IM_SEQ_ERROR: Error (Sequence timeout) while communicating with component opcode:N/A (for:RID_VDC: 1) %VDC_MGR-3-VDC_ERROR: vdc_mgr: GIM did not return success for vdc creation: error-code: 0x41850006 %VDC_MGR-3-VDC_ERROR: vdc_mgr: There has been a failure at gim (pre_cr/del) %VDC_MGR-2-VDC_CRITICAL: vdc_mgr has hit a critical error: Creation of default vdc failed! %VDC_MGR-5-VDC_STATE_CHANGE: vdc 1 state changed to failed Reloaded the device but the issue still existed. Had other devices which upgraded successfully and those devices had over 431 interfaces configured with "vpc orphan-port suspend". The devices that failed to upgrade had over 790 interfaces configured with this command. We then removed the command from all the interfaces and reloaded the switch and the upgrade was now successful: #sh install all failure-reason No install all failure-reason