Symptoms
Bosh commands are not working and a similar error is thrown
Error: Status: 500; ErrorMessage: <nil>; Description: There was a problem completing your request. Please contact your operations team providing the following information: service: p.pks, service-instance-guid: 9e5db8ec-d9da-4f77-b4d6-61345f3569e0, broker-request-id: 15ede0c7-a870-4ed1-8cb5-cc1dd762f710, operation: bind - error-message: gathering binding info Could not fetch VMs info for deployment "service-instance_9e5db8ec-d9da-4f77-b4d6-61345f3569e0": Listing deployment 'service-instance_9e5db8ec-d9da-4f77-b4d6-61345f3569e0' vms infos: Director responded with non-successful status code '500' response ''; ResponseError: <nil>
Cause
No enough space is left in the bosh director persistent disk
Resolution
1- To validate that the issue is indeed related to not having enough space left issue in the director persistent disk , please ssh to the bosh director following the instructions mentioned in the following doc https://docs.vmware.com/en/VMware-Tanzu-Operations-Manager/3.0/vmware-tanzu-ops-manager/install-trouble-advanced.html
2- Check the bosh director logs stored under var/vcap/sys/logs and you will find similar errors are prompted in the credhub and the postgres logs
"base/16386/pg_internal.init.4894": No space left on device
When running df -h you should find /dev/sdc1 fully consumed simlar to the below example
bosh/0:~$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 25M 3.9G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda1 2.9G 1.5G 1.3G 55% /
/dev/sdb2 56G 7.7G 45G 15% /var/vcap/data
tmpfs 16M 372K 16M 3% /var/vcap/data/sys/run
/dev/sdc1 50G 47G 0 100% /var/vcap/store
Also you can navigate to the operations manager GUI and from the bosh director tile , you can check the bosh director VM persistent disk consumption percentage as well .
3- Increase the persistent disk on the Bosh director tile from the resource config tab and Apply Changes which should solve the issue.
