OPERATIONAL DEFECT DATABASE
...
...
vmware-sps service starts properlyOn VCSA SSH, "wget localhost:22000/sms/HealthStatus ; cat HealthStatus ; rm HealthStatus" shows SPS health as yellowvCenter > Configure > Storage providers showing as offlinevCenter > Configure > Refreshing storage providers gives the error "Storage service not initialised" /var/log/vmware/vmware-sps/sps.log:2021-05-25T10:16:53.846+09:00 [pool-35-thread-4] ERROR opId=lro-2-2869eb71 com.vmware.vim.storage.common.VmodlErrorStrings - ProviderLoader initialization is ongoing.2021-05-25T10:16:53.846+09:00 [pool-35-thread-4] INFO opId=lro-2-2869eb71 com.vmware.vim.sms.StorageManagerImpl - Timer stopped: queryStorageContainer, Time taken: 1 ms.2021-05-25T10:16:53.846+09:00 [pool-35-thread-4] ERROR opId=lro-2-2869eb71 com.vmware.vim.storage.common.VmodlErrorStrings - Failed to query StorageContainer for input(sms.fault.ServiceNotInitialized) { faultCause = null, faultMessage = null}=== OR === /var/log/vmware/vmware-sps/sps-runtime.log.stderr growing quickly with no limitSeeing the following message in sps-runtime.log.stderr INFO: Client raised fatal(2) handshake_failure(40) alert: Failed to read record org.bouncycastle.tls.TlsFatalAlert: handshake_failure(40)Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
The certificate used by SMS to communicate with IOFilter VASA Providers is generated using signature algorithm sha1WithRSAEncryption. This was changed to sha256WithRSAEncryption in 7.0.2.This is due to the fact that vCenter 7.0.2 is FIPS compliant and having sha1WithRSAEncryption will cause IOFilter VP's get into offline/disconnected state.
To rectify this, delete the SMS certificate, unregister the offline providers, and restart the SPS service to register the storage providers with the SHA256 certificate. The below steps can also be used to reset the storage providers for troubleshooting.Please take an offline snapshot and/or backup of all nodes in the SSO domain. Do not skip this step.1. Stop the Storage Provider Service: service-control --stop vmware-sps2. Delete the SMS certificate from VECS: /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store sms --alias sms_self_signed3. Start the SPS service: service-control --start vmware-spsThis will have the effect of SMS self-signed certificate re-generated with sha256withRSAEncryption signing algorithm and this certificate will now be FIPS compliant. Regeneration of the cert will also cause all the storage providers to go offline as it's a new cert and no longer trusted; meaning step 5 will unregister them and clear them out.4. After step 3 above, Wait for some time for SPS service to get into initialized state and health status is GREEN. Use the wget command above to check. Initialisation can take a long time.5. Once after storage providers list is updated and providers are listed (there will be providers listed with offline/disconnected state), run the python script "unreg_vasa.py" and capture the output (needed only in case if the steps doesn't resolve the issue), python unreg_vasa.py -s <VC_IP_ADDRESS> NOTE: <VC_IP_ADDRESS> above need to be replaced with actual values. NOTE: The script waits 5 seconds between unregistering each VP. Please wait for it to finish.6. Restart vmware-sps service: vmon-cli -r sps This will have the effect of all the IOFilter VPs that was unregistered in step 5 to get registered again.
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
