Symptoms

When an external vCenter manages multiple VxRail clusters, the VPXD memory usage may gradually increase and finally reach to a critical value and crash. Check the /var/log/vmware/vpxd.log file on the vCenter, you can see the following entries during the VXPD crash. yyyy-mm-ddThh:mm:ss.mss+05:30 error vpxd[38046] [Originator@6876 sub=Memory checker] Current value 4044496 exceeds hard limit 4042752. Shutting down process. yyyy-mm-ddThh:mm:ss.mss+05:30 panic vpxd[38046] [Originator@6876 sub=Default] --> --> Panic: Memory exceeds hard limit. Panic

Cause

VxRail Property Collector service causes vCenter VPXD service memory leak.

Resolution

VxRail has fixed this issue on its 8.0.331 release.If you are not ready for an update, follow the below steps to apply a hot patch on the VxRail manager. Follow VMware KB 2110031 (External Link) to clear the vCenter staled data.Download the property-collector-patch .zip file from the KB attachment. Extract the file, it contains 5 .zip files for different VxRail versions. Please select the appropriate patch .zip file based on your VxRail version, upload it to the /home/mystic directory on the VxRail Manager, and then run the following commands as the root user. Extract the file:7.0.45x cd /home/mystic/ unzip property-collector-patch-7.0.45x.zip 7.0.48x, 7.0.510 cd /home/mystic/ unzip property-collector-patch-7.0.48x_7.0.510.zip 7.0.520, 7.0.53x, 7.0.540, 7.0.550 cd /home/mystic/ unzip property-collector-patch-7.0.520_7.0.53x_7.0.540_7.0.550.zip 8.0.21x cd /home/mystic/ unzip property-collector-patch-8.0.210.zip 8.0.30x, 8.0.31x, 8.0.32x, 8.0.330 cd /home/mystic/ unzip property-collector-patch-8.0.30x_8.0.31x_8.0.32x_8.0.330.zip Verify that the files "vc_monitor.py" and "property-collector-patch.yaml" exist in the /home/mystic directory. Run below commands to update the property collector: cd /home/mystic/ kubectl patch deployment property-collector --patch-file property-collector-patch.yaml kubectl scale deployment property-collector --replicas=0 kubectl scale deployment property-collector --replicas=1 Run below command to check if the patch is applied successfully: kubectl exec -it $(kubectl get pods | grep -Eo "property-collector(\-|\w)+") -- grep Remove /home/app/server/vc_monitor.py The following message is expected if the patch is applied successfully: LOGGER.info("Remove property collector filter: {}".format(str(f))) NOTE: If you encounter this issue on other VxRail versions that are not covered by this KB, engage Dell Support and request a hot patch for that specific version.

Support Cases