OPERATIONAL DEFECT DATABASE
...
...
ESXi host Crashed with PSOD caused by brcmfcoe driver referencing __lpfc_sli_get_iocbq.Any of the following HPE servers running VMware ESXi 6.0, VMware ESXi 6.5 or VMware ESXi 6.7 and configured with an HPE FlexFabric 20Gb 2-port 650FLB Adapter with driver version 12.0.1211.0 (or prior): HPE ProLiant BL460c Gen10 Server Blade HPE ProLiant BL460c Gen9 Server Blade HPE ProLiant BL660c Gen9 ServerHost crashes with or without load.The host may crash with a wake NOT set" message is logged in the vmkernel logs as below 2019-01-20T11:36:22.861Z cpu7:32854)elxnet: elxnet_startQueue:5365: [vmnic5] RxQ, QueueIDVal:42019-01-20T11:37:07.966Z cpu19:31782666)WARNING: brcmfcoe: lpfc_sli_issue_iocb_wait:10765: 0:0330 IOCB wake NOT set, Data x24 x02019-01-20T11:37:07.969Z cpu19:31782666)World: 9762: PRDA 0x418044c00000 ss 0x0 ds 0x10b es 0x10b fs 0x0 gs 0x13b2019-01-20T11:37:07.969Z cpu19:31782666)World: 9764: TR 0x4020 GDT 0x4394b8521000 (0x402f) IDT 0x4180244ca000 (0xfff)2019-01-20T11:37:07.969Z cpu19:31782666)World: 9765: CR0 0x80010031 CR3 0x20cc2f9000 CR4 0x427682019-01-20T11:37:08.036Z cpu19:31782666)Backtrace for current CPU #19, worldID=31782666, rbp=0x4305405fae702019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851b8a8:[0x418024ca1f69]__lpfc_sli_get_iocbq@<None>#<None>+0x51 stack: 0x4305407ef330, 0x4302019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851b8b0:[0x418024ca27cd]lpfc_sli_get_iocbq@<None>#<None>+0x25 stack: 0x4305405fae70, 0x418022019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851b8d0:[0x418024c813cb]lpfc_mgmt_send_ct@<None>#<None>+0x98b stack: 0x0, 0x439400000028, 0x2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851ba20:[0x418024af3b6f]VmkMgmtDispatchCallback@com.vmware.vmkapi#v2_3_0_0+0x13f stack: 0x1,2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bae0:[0x418024af4a26]VmkMgmtCharIoctl@com.vmware.vmkapi#v2_3_0_0+0x5ea stack: 0x4305407ef2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bc70:[0x418024495130]VMKAPICharDevDevfsWrapIoctl@vmkernel#nover+0x80 stack: 0x439400000012019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bce0:[0x418024697694]DevFSIoctl@vmkernel#nover+0x684 stack: 0x43132dd9cc70, 0xbcd000000002019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bdb0:[0x418024673d38]FSSObjectIoctlCommon@vmkernel#nover+0x9c stack: 0x4394b851be43, 0x432019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851be00:[0x41802467743c]FSS_IoctlByFH@vmkernel#nover+0xbc stack: 0x1f08d7b0, 0x200, 0x1f08d72019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851be40:[0x4180249e8d4b]UserFile_PassthroughIoctl@<None>#<None>+0x3f stack: 0x43132dda2be8,2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851be60:[0x418024a30764]UserVmfs_Ioctl@<None>#<None>+0x24 stack: 0x43132dd9c0e0, 0x4180249cd2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bec0:[0x4180249f3074]LinuxFileDesc_Ioctl@<None>#<None>+0x5c stack: 0x4394b851bf30, 0x36,2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bef0:[0x4180249c8663]User_LinuxSyscallHandler@<None>#<None>+0xd7 stack: 0x0, 0x0, 0x0, 0x2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bf20:[0x41802448ee91]User_LinuxSyscallHandler@vmkernel#nover+0x1d stack: 0x0, 0x13b, 0x0,2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bf30:[0x4180244c8067]gate_entry_@vmkernel#nover+0x0 stack: 0x0, 0x36, 0x200, 0x1f08d7b0,2019-01-20T11:37:08.068Z cpu19:31782666)VMware ESXi 6.0.0 [Releasebuild-10474991 x86_64]#PF Exception 14 in world 31782666:ams-main IP 0x418024ca1f69 addr 0xe0Or a similar coredump to:2019-01-26T12:18:58.687Z cpu22:35927)WARNING: brcmfcoe: lpfc_sli_issue_iocb_wait:10765: 0:0330 IOCB wake NOT set, Data x24 x02019-01-26T12:18:58.697Z cpu2:33532)World: 9762: PRDA 0x418040800000 ss 0x0 ds 0x10b es 0x10b fs 0x0 gs 0x13b2019-01-26T12:18:58.697Z cpu2:33532)World: 9764: TR 0x4020 GDT 0x43929c721000 (0x402f) IDT 0x4180008ca000 (0xfff)2019-01-26T12:18:58.697Z cpu2:33532)World: 9765: CR0 0x80010031 CR3 0x20fa85f000 CR4 0x427682019-01-26T12:18:58.762Z cpu2:33532)Backtrace for current CPU #2, worldID=33532, rbp=0x43051d9d00102019-01-26T12:18:58.762Z cpu2:33532)0x439117e1b8e8:[0x4180010a1f69]__lpfc_sli_get_iocbq@<None>#<None>+0x51 stack: 0x43051d91de70, 0x4302019-01-26T12:18:58.762Z cpu2:33532)0x439117e1b8f0:[0x4180010a27cd]lpfc_sli_get_iocbq@<None>#<None>+0x25 stack: 0x43051d9d0010, 0x418002019-01-26T12:18:58.762Z cpu2:33532)0x439117e1b910:[0x4180010acb24]lpfc_sli4_handle_eqe@<None>#<None>+0x6ac stack: 0x430336d7a140, 0xa02019-01-26T12:18:58.762Z cpu2:33532)0x439117e1b9d0:[0x4180010ad6e1]lpfc_sli4_intr_bh_handler@<None>#<None>+0x89 stack: 0x28, 0x41000b112019-01-26T12:18:58.762Z cpu2:33532)0x439117e1ba00:[0x418000859199]IntrCookieBH@vmkernel#nover+0x299 stack: 0x0, 0x43051d95bca0, 0x43912019-01-26T12:18:58.762Z cpu2:33532)0x439117e1baa0:[0x41800083329e]BH_DrainAndDisableInterrupts@vmkernel#nover+0xe2 stack: 0x10100000002019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bb30:[0x4180008571b2]IDT_IntrHandler@vmkernel#nover+0x1ce stack: 0x0, 0x0, 0x0, 0x41800082019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bb60:[0x4180008c8067]gate_entry_@vmkernel#nover+0x0 stack: 0x0, 0x0, 0x0, 0x0, 0x418040802019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bc20:[0x418000b067fa]Power_HaltPCPU@vmkernel#nover+0x1ee stack: 0x417fc0a84f60, 0x41804092019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bc70:[0x418000a12dc8]CpuSchedIdleLoopInt@vmkernel#nover+0x2f8 stack: 0x39266d1018ad98, 0x2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bcf0:[0x418000a16522]CpuSchedDispatch@vmkernel#nover+0x15fe stack: 0x439117e27100, 0x43912019-01-26T12:18:58.762Z cpu2:33532)0x439117e1be10:[0x418000a17104]CpuSchedWait@vmkernel#nover+0x240 stack: 0x0, 0x430769ee8080, 0x5d012019-01-26T12:18:58.762Z cpu2:33532)0x439117e1be90:[0x418000a17445]CpuSchedTimedWaitInt@vmkernel#nover+0xc9 stack: 0x430700002001, 0x412019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bf10:[0x418000a17516]CpuSched_TimedWait@vmkernel#nover+0x36 stack: 0x430769ee8080, 0x41802019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bf30:[0x418000850a5d]helpFunc@vmkernel#nover+0x611 stack: 0x0, 0x430769e11b60, 0x33, 0x0,2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bfd0:[0x418000a17d9e]CpuSched_StartWorld@vmkernel#nover+0xa2 stack: 0x0, 0x0, 0x0, 0x0, 02019-01-26T12:18:58.795Z cpu2:33532)VMware ESXi 6.0.0 [Releasebuild-10474991 x86_64]#PF Exception 14 in world 33532:helper51-3 IP 0x4180010a1f69 addr 0xe0PTEs:0x160bba027;0x20f9fb1027;0x0;Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.
To avoid PSOD.
The completion handler associated with the IOCB was never called.
Restart of ESXi host is required after driver is installed.
To resolve the issue, upgrade the brcmfcoe driver: For ESXi 6.0, use brcmfcoe driver version 12.0.1110.39 - To download go to Customer ConnectFor ESXi 6.5 , use brcmfcoe driver version 12.0.1216.4 - To download go to Customer ConnectFor ESXi 6.7, use brcmfcoe driver version 12.0.1216.4. To download go to Customer Connect Disclaimer:VMware is not responsible for the reliability of any data,opinions,advice or statements made on third-party websites.Inclusion of such links does not imply that VMware endorses,recommends or accepts any responsibility for the content of such sites IMPORTANT: Refer to HPE customer advisory for more information
No workaround.
Procedure to install driver using various methods refer to VMware KB : How to download and install async drivers in ESXi 5.x/6.x
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
