Symptom

After reload of N9K-C93108TC-FX3P and N9K-C93108TC-FX3H device some RJ45 interfaces might not come up. This can happen with any reload reason (power up/down, upgrade, SW reset, crash). Issue is related only to the front panel interfaces Ethernet 1/1-48 (MGMT port is not affected)

Conditions

Only Cisco Nexus N9K-C93108TC-FX3P and N9K-C93108TC-FX3H Switch is affected no other PID are affected. Only copper front panel ports Eth1/1 - 48 can be affected. (mgmt port is not affected). There is no specific trigger and issue might happen with any configuration type present. After consecutive reloads, different ports in the same PHY group can be affected. Only switches with MI FPGA version 0x10 or lower are affected. In order to verify: switch# show version module 1 epld EPLD Device Version --------------------------------------- MI FPGA 0x10 IO FPGA 0x8 Issue is also applicable to nexus device acting in FEX mode. Ie issue is independent of the mode (switch/fex)

Workaround

Further Problem Description

There is no workaround available to recover the impacted ports. To prevent the issue, do not reload the switch. Updating the electronic programmable logic devices (EPLD) proactively will also prevent the issue from occurring. Solution The solution is to upgrade the EPLD to Cisco NX-OS System software release independent EPLD n9000-epld.CSCwh84282.img - https://software.cisco.com/download/home/286326561/type/282253564/release/10.2(6) Note: While available to download under the 10.2(6)M release tree, the EPLD n9000-epld.CSCwh84282.img image is software release agnostic and can be installed on devices with Cisco NX-OS System software releases other than Release 10.2(6). This EPLD is supported on Cisco NX-OS Software releases 9.3.x and 10.2.x and higher (such as Release 10.3.x, Release 10.4.x and so on). When installing NXOS and EPLD with single CLI that install both with single reload (install all nxos epld ISSUE WILL NOT BE RESOLVED after first reload you MUST RELOAD DEVICE SECOND TIME. Procedure: 1. install nxos NXOS-IMAGENAME epld EPLD-IMAGENAME 2. device reload as part of installation 3. manual reload (2nd reload after installation) - this step is necessary to recover all ports (even if epld version is 0x11) NOTE: As FX3 devices in FEX mode does not automatically reload when parent is manually reloaded after EPLD/NXOS installation on the parent device FX3 in FEX mode must be manually reloaded one by one When installing NXOS and EPLD with separate CLI you MUST RELOAD device after NXOS and EPLD installation is completed - procedure: Procedure: 1. install epld EPLD-IMAGENAME module all 2. device reload as part of EPLD installation 3. install nxos NXOS-IMAGENAME 4. device reload as part of NXOS installatoion 3. manual reload (3nd reload after installation) - this step is necessary to recover all ports (even if epld version is 0x11) NOTE: As FX3 devices in FEX mode does not automatically reload when parent is manually reloaded after EPLD/NXOS installation on the parent device FX3 in FEX mode must be manually reloaded one by one The following NX-OS EPLD updates that were released with Cisco NX-OS System software versions 9.3(13) and 10.4(2)F already contain the fix for this problem: NX-OS EPLD update for release 9.3(13) - https://software.cisco.com/download/home/286317419/type/282253564/release/9.3(13) NX-OS EPLD update for release 10.4(2)F - https://software.cisco.com/download/home/286317419/type/282253564/release/10.4(2) Note: This issue will not be resolved by a Cisco NX-OS Software update; the EPLD must be upgraded. (An EPLD upgrade requires a reload.) To install the EPLD, follow standard EPLD upgrade procedure using these steps: Copy the EPLD image to bootflash. Install epld bootflash:n9000-epld.CSCwh84282.img module 1 After the EPLD has been upgraded, affected interfaces will be recovered. To verify whether the correct EPLD was installed, use the CLI command show version module 1 epld. In the command output, MI FPGA versions must be 0x11 and higher. The following example shows the output of a fixed MI FPGA version: n9k# show version module 1 epld EPLD Device Version --------------------------------------- MI FPGA 0x11 IO FPGA 0x8 Issue is also applicable to nexus device acting in FEX mode. Ie issue is independent of the mode (switch/fex)