Symptoms
While using TMC UI to create a new guest cluster on vSphere with Tanzu. Guest cluster stuck on running status on the TMC UI
On vSphere side, just one control plane is created and nothing more is up.From cloud-init-output.log:
umount /var/lib
umount: /var/lib: target is busy.
Cause
On TMC UI , While creating the TKC there is an option to add a disk to the control planes and the worker nodes.Issue happened when tried to add disk on /var/lib directory.TKC spec is attempting to expand storage capacity for the entire `/var/lib` directory, but data migration fails - `target is busy` means a process has a file open somewhere in the `/var/lib` directory.
Impact / Risks
TKC creation is stuck
Resolution
Solution is to use the following subdirectories instead of using /var/libFor control plane nodes:
Mount Path:
/var/lib/etcd
For worker nodes:
Mount Path:
/var/lib/containerd
From https://tanzu.vmware.com/content/blog/deploy-tanzu-kubernetes-clusters-data-volumes-tanzu-mission-control
