Symptoms
1. Alarms for platform services disk usage growing.2. Failure to generate a support bundle.Relevant Log LocationsThe issue can be confirmed by collecting the "diskusage.txt" file after executing the following commands:
napp-k exec -it `napp-k get pods | grep cluster | cut -d ' ' -f 1` -c cluster-api -- find /var/log/napp/supportbundle -type f -exec ls -lh {} \; > ./diskusage.txt
napp-k exec -it `napp-k get pods | grep cluster | cut -d ' ' -f 1` -c cluster-api -- df -h >> ./diskusage.txt
Purpose
From the 'df -h' command output, it's evident that the '/var/log/napp/supportbundle' directory on the 'cluster-api' pod is almost full or full. To address this issue, a patch named "cluster_api_patch.yaml" is suggested. This patch is to be applied to the 'cluster-api'. The steps to apply this patch are provided in 'Workaround Section'.
Cause
In certain scenarios, the 'rawflowcorrelator' and 'overflowcorrelator' pods can sometimes malfunction, leading to excessive and uncontrolled log generation. This abnormal behavior results in rapid disk space consumption, potentially causing indefinite growth in disk usage. This issue is particularly severe if these pods crash or restart frequently, exacerbating log accumulation and posing a risk to system performance and storage capacity.
Impact / Risks
Alarms will be seen about platform services disk usage, and it will not be possible to generate the NAPP support bundle.It is a known issue in 4.1.x and lower versions.
Resolution
This is now fixed in NAPP 4.1.2 and higher versionsNAPP 4.1.2 do not need to patch the setup.The patch "cluster_api_patch" attached to this KB has been tested on NAPP 4.1.1 release version and after applying the patch , flowcorelator files older than 2 days were deleted.
Workaround
Download the attached file "cluster_api.yaml" and move it to the NSX manager node.Execute the following command on the NSX shell from the same directory where the file is present:
napp-k patch deployment cluster-api --type merge --patch-file=cluster_api_patch.yaml
