Symptom
On Nexus 9K EoR switches (Nexus 9500 switches), the following error will be seen in logging (exact percent may be variable):
`show logging`
2023 Apr 11 16:08:32 Nexus9000-switch %SYSMGR-STANDBY-2-NON_VOLATILE_DB_FULL: System non-volatile storage usage is unexpectedly high at 80%.
The utilization can grow over time and can eventually hit 100%.
Specifically, when checking the bash shell on the standby supervisor, the following file will be consuming the vast majority of space in the /dev/sda3 file system:
Nexus9000-switch(standby)# run bash sudo su -
root@(none)#df -h | egrep -i pss
/dev/sda3 120M 97M 18M 85% /mnt/pss <<< checking for which file system /mnt/pss is mounted on
root@(none)#ls -la /mnt/pss/core*
-rw-rw-rw- 1 root root 83337073 Jan 26 00:06 /mnt/pss/core-dump.log <<< the offending file
-rw-r--r-- 1 root root 193 May 11 2020 /mnt/pss/core_handling.log
Conditions
- Nexus 9500 switch running impacted code
- specific syslog seen - %SYSMGR-STANDBY-2-NON_VOLATILE_DB_FULL: System non-volatile storage usage is unexpectedly high at 80%.
- file /mnt/pss/core-dump.log has grown to consume most of the available free space relative to all other files
