OPERATIONAL DEFECT DATABASE
...
...
Attempting to the vCenter certificate replacement workflow fails with below error in the /var/log/vmware/vcf/operationsmanager/operationsmanager.log: 2021-05-11T05:07:00.287+0000 ERROR [vcf_om,6c92de84a35548da,391e] [c.v.v.c.vc.VCenterCertificatePlugin,om-exec-29] Replace certificate failed for resource: {"version":"7.0.2.00000-17694817","hostName":"<FQDN>","id":"54df8203-2e72-40b4-ba51-d1dc4af443e3","vmName":"<VM_NAME>","resourceType":"vcenter","credentials":[{"username":"root","secret":"*****","type":"ssh"},{"username":"administrator@vsphere.local","secret":"*****","type":"sso"}],"ipAddress":"<IP>","sans":["<FQDN>"],"master":false} : 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.error","value":{"error_type":"ERROR","messages":[{"args":["AFD Native Error Occured: 11"],"default_message":"Exception found (AFD Native Error Occured: 11)","id":"com.vmware.certificatemanagement.error"}]}}] : {}com.vmware.vcf.certmgmt.exceptions.CertificateReplacementException: 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.error","value":{"error_type":"ERROR","messages":[{"args":["AFD Native Error Occured: 11"],"default_message":"Exception found (AFD Native Error Occured: 11)","id":"com.vmware.certificatemanagement.error"}]}}] at com.vmware.vcf.certmgmt.vc.VCenterCertificatePluginService.replaceCertificate(VCenterCertificatePluginService.java:230) at com.vmware.vcf.certmgmt.vc.VCenterCertificatePluginService.uploadCertificate(VCenterCertificatePluginService.java:179) at com.vmware.vcf.certmgmt.vc.VCenterCertificatePlugin.replaceCertificate(VCenterCertificatePlugin.java:83)at com.vmware.vcf.certmgmt.vc.VCenterCertificatePlugin$$FastClassBySpringCGLIB$$874f9739.invoke(<generated>)at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) The SDDC Manager and vCenter re-trust sub task fails with below error: 2021-05-11T05:15:22.959+0000 ERROR [vcf_om,6c92de84a35548da,701f] [c.v.v.c.v.VCenterCertificatePluginService,om-exec-25] 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vapi.std.errors.Error"],"default_message":"Provider method imp... (482 bytes)]org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: [{"type":"com.vmware.vapi.std.errors.internal_server_error","value":{"error_type":"INTERNAL_SERVER_ERROR","messages":[{"args":["com.vmware.vapi.std.errors.Error"],"default_message":"Provider method imp... (482 bytes)] at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:100) at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:186) at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125) at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:780) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:738) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:712) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:600) Running /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh on the vCenter reports the following Error: Failed to trigger root cert refreshvecs-cli failed. Error 11: Possible errors:LDAP error: Administrative limit exceededWin Error: Operation failed with error ERROR_BAD_FORMAT (11) Adding a trusted root certificate to Certificate Management via UI fails with "Error occurred while adding trusted root certificates" Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.
If the vCenter certificate replacement fails or the SDDC Manager and vCenter re-trust fails the vCenter is unable to process any certificate related workflows, which can lead other workflows like add/ remove host, etc to fail.
This issue occurs if root certificate content is of length in the order of 233 multiples.
This is a known issue affecting all VCF 4.x releases and will be resolved in a future release.
To workaround this problem, please either use the attached fixCertRetrust.sh script or follow the steps below in Option 2.Option 1: Download the fixCertRetrust.sh script.Using a file transfer utility, like WinSCP, copy the file to /tmp/ on the the vCenter appliance.Open an SSH session to the vCenter appliance/Change the permssions on the file: chmod +x fixCertRetrust.sh Run the script: sh CertRepublish.sh Option 2: Open an SSH session to the vCenter appliance with the root user. Get the trusted root certificates list /usr/lib/vmware-vmafd/bin/dir-cli trustedcert list --login administrator@<domain> Take a backup of all the certificates; using the IDs the output from step 1 /usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id <ID> --outcert trustedCert_x.crt If there is no new line added after 64th character, execute: sed -i -e "s/.\{64\}/&\n/g" trustedCert_x.crt Verify the backed up certificate: openssl x509 -text -in trustedCert_x.crt Proceed to unpublish certificate from vmdir if the backup was successful: /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert trustedCert_x.crt Initiate VECS store refresh to delete the unpublished certificate from the VECS store /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh Publish the backed trusted root certificates back to vmdir /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert trustedCert_x.crt Verify if the certificate has been added: /usr/lib/vmware-vmafd/bin/dir-cli trustedcert list Initiate VECS store refresh to add the published certificate to vecs. /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh Verify VECS TRUSTED_ROOTS store entries: /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
