Symptoms

Using multiple network cards Intel X520 or Intel X540 in a single PowerEdge server may result from a message from VMware ESXi SR-IOV. VMware will report that there is an Incorrect Port-Virtual Function Enumeration.The issue can be experienced when trying to configure SR-IOV per Intel's white paper at:http://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/converged-network-adapter-sr-iov-on-esxi-5-1-brief.pdfhttp://www.intel.com/content/dam/www/public/us/en/documents/reference-guides/sr-iov-guide-ethernet-cna-x710-xl710-vmware-vsphere-6-brief.pdf When the issue is seen, after enabling virtual functions from the command line utility (CLI) with the max_vfs parameter, the port-virtual function enumeration appears inconsistent with the attempted assignment. Viewing the web GUI may also show inconsistent port-virtual function enumeration after enabling virtual functions from the CLI in this manner Solution VMware is aware of this issue and a fix is currently being investigated. The VMWare documentation recommends that users enable SR-IOV virtual functions via the vSphere Web Client:http://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.networking.doc%2FGUID-EE03DC6F-32CA-42EF-98FC-12FDE06C0BE0.html&resultof=%22%69%6f%76%22%20 For customer's attempting to use the CLI to configure SR-IOV, please ensure the latest drivers and web client are in place, then it is recommended to use the web client method to enable the virtual functions. Advanced CLI Workaround This CLI workaround is provided for advanced users. The ESX kernel scans and sorts the PCI devices in three different ways depending on the command used. Segment : Bus : Device : Function (Segment = Domain in Linux terms, normally it's just 0) 1. PCI tree scanning order, i.e. device-attached order 2. The VMKernel scans from the root bridge and all child nodes read from config space and in the 'depth-first-search' recursive way. There is no documentation, but it is somewhat similar to http://wiki.osdev.org/PCI#Recursive_Scan_With_Bus_Configuration. 3. vmnic name: i.e. vmnic1, vmnic1x, ...vmnic2,vmnic2x,.....,vmnic3 Then the various tools to list devices may follow any of these three methods: 1. lspci is sorted by S,B,D,F 2. 'esxcfg-nics-l' is sorted by vmnicX name 3. Web client is sorted by S,B,D,F 4. esxcfg-module -s max_vsf= is sorted by PCI tree scanning order 5. Dmesg shows the PCI devices sorted based on the PCI tree scanning order In order to correctly set/enable SR-IOV (set max_vfs via the command line), the user should enable a PCI device based on the PCI tree scanning order, which can be achieved by: -running dmesg to get the info and parse the info to the PCI device info, then set the max_vfs based on the order listed in the dmesg output-running lspci.py script (below) and set the max_vfs based on the order listed by the script. This script queries the PCI and sorts by PCI tree scanning order. The script is for example only, and it may not run on all ESX kernel versions. It was tested with ESX 6.0U3. In order to use the script below please save in a separate file with the .py extention (lspci.py for example). #!/usr/bin/env python import vmwareimport vmware.vsi as vsi VSI_HW_PCI_DEVICES = '/hardware/pci/devices'VSI_HW_PCI_ROOT = '/hardware/pci'VSI_SYS_MOD_ROOT = '/system/modules' modInfo = {} for m in vsi.list(VSI_SYS_MOD_ROOT): general = vsi.get("%s/%s/general" % (VSI_SYS_MOD_ROOT, m)) id = int(general['moduleId']) modInfo[id] = m for ptr in vsi.list(VSI_HW_PCI_DEVICES): dev = vsi.get("%s/%s/info" % (VSI_HW_PCI_DEVICES, ptr.strip())) s, b, d, f = dev['seg'], dev['bus'], dev['dev'], dev['func'] nodePath = "%s/seg/%d/bus/%d/slot/%d/func/%d" % (VSI_HW_PCI_ROOT, s, b, d, f) cfg = vsi.get("%s/seg/%d/bus/%d/slot/%d/func/%d/pciConfigHeader" % (VSI_HW_PCI_ROOT, s, b, d, f)) modName = modInfo[cfg['moduleID']] if cfg['moduleID'] != 0xffffffff else " " * 8 print("%04d:%02d:%02d:%d\t%-16s\t%s" % (s, b, d, f, modName, cfg['name']))

Cause

Resolution

Support Cases