Issue

This is similar to RHEL-102929 but for RHEL8. I'm filing a different JIRA because it's harder to reproduce on RHEL8, but still it reproduces from time to time. When having rules that monitor file system activity (such as OSPP or STIG rules tracking deletion, mode change, etc.), updating the system or deleting a high number of files in a row leads to getting the following error message: Jul 11 09:41:14 vm-audit8 auditd[834]: Error receiving audit netlink packet (No buffer space available) Jul 11 09:41:15 vm-audit8 auditd[834]: Error receiving audit netlink packet (No buffer space available) This occurs despite having a large buffer (16384) and no other activity on the system. The error messages are usually printed multiple times (as shown above) but there is nothing in dmesg that would indicate the audit buffer was full. auditctl -s doesn't always report lost events. Reproducer Reproducing is possible with the following setup: Tune buffer to 16384 and add rules on "delete" -b 16384 -a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -F auid>=1000 -F auid!=4294967295 -k delete -a always,exit -F arch=b32 -S unlink -S unlinkat -S rename -S renameat -F auid>=1000 -F auid!=4294967295 -k delete Tune audit log file size to avoid rotations (rotations was leading to the error message in the past so it's better to rule that out as well) max_log_file = 64 Move /var/log/audit to a dedicated disk # mkfs.xfs /dev/vdb # echo "/dev/vdb /var/log/audit xfs defaults 0 0" >> /etc/fstab Tune disk I/O for the dedicated disk to be slow In QEMU/KVM+libvirt, edit "VirtIO Disk2" XML and add iotune: <target dev="vdb" bus="virtio"/> <iotune> <total_bytes_sec>1000000</total_bytes_sec> </iotune> Reboot Create a lot of files in a temporary file system # mkdir /scratch_tmp # mount -t tmpfs tmpfs /scratch_tmpfs # cd /scratch_tmpfs # mkdir dir1 ; for i in $(seq 1 1024); do echo $i > dir1/file$i; done # for i in $(seq 2 128); do cp -r dir1 dir$i; done # cd # mount -o remount,ro /scratch_tmp Note: I remount ro to be able to reproduce multiple times without recreating the files. Delete the files using a find+rm command # find /scratch_tmpfs -type f | xargs rm Alternatively use the C program attached as delete.c (it deletes faster hence reproduces more often): # gcc -o delete delete.c # ./delete /scratch_tmpfs Result Error messages: Jul 11 09:41:14 vm-audit8 auditd[834]: Error receiving audit netlink packet (No buffer space available) Jul 11 09:41:15 vm-audit8 auditd[834]: Error receiving audit netlink packet (No buffer space available) Additional information The end customer is hitting the issue when his 3rd party software written in C deletes a large temporary hierarchy. He reproduces on HyperV VM which has great I/O performance (in opposite to my reproducer that slows down the disk on purpose). For him, he can see 500 "delete" events occurring in 120ms, then issue occurs.

Release Notes

No. of Comments

Resolution

Unresolved