Symptoms
A syslog-ng issue with remote logging to an invalid remote syslog server may lead to logging, even locally, via syslog-ng to stop. CPU use of syslog-ng may increase.
For software version 13.1 only it may lead to BIG-IP unexpectedly rebooting due to
host watchdog timeout, typically within hours to
day or two after syslog-ng gets hung up.
The cessation of logging happens at the time of the last 'Syslog connection broken' in /var/log/messages before reboot.
That message will appear without a preceding 'Syslog connection established' just before it with same timestamp.
At this time syslog-ng typically spins, using near 100% CPU.
Impact
Very rarely syslog-ng hangs in a non-functional state. Sometimes, this may lead to an unexpected reboot of BIG-IP.
Loss of logs before restart and traffic disrupted while BIG-IP restarts.
Even if it doesn't reboot the loss of logging functionality can cause some daemons to block while logging and thus interrupt service.
Conditions
Invalid syslog-ng server configuration or broken connection from BIG-IP toward configured syslog-ng remote server.
A server is configured as a remote syslog destination on the BIG-IP, but it or an intervening system responds to stream of log messages by breaking connection eg by sending ICMP port unreachable to BIG-IP.
Syslog-ng will note the connection attempt and that it has broken usually in the same second, and do so every 60s when it retries.
There may be many of these log pairs, repeating every minute in /var/log/messages, such as:
Nov 25 03:14:01 localhost.localdomain notice syslog-ng[12452]: Syslog connection established; fd='14', server='AF_INET(192.168.1.1:514)', local='AF_INET(0.0.0.0:0)'
Nov 25 03:14:01 localhost.localdomain notice syslog-ng[12452]: Syslog connection broken; fd='14', server='AF_INET(192.168.1.1:514)', time_reopen='60'
The final log will of a broken connection only, usually one minute after the last established/broken pair in the very rare event that syslog-ng hangs.
Nov 25 03:15:01 localhost.localdomain notice syslog-ng[12452]: Syslog connection broken; fd='14', server='AF_INET(192.168.1.1:514)', time_reopen='60'
Workaround
Ensure syslog-ng server configuration is valid, and that the server is reachable. If a remote server is not reachable remove it from the BIG-IP syslog configuration.
If the system has encountered this issue it's important that syslog-ng is restarted if that (or equivalent such as reboot) hasn't already occurred, to resume its normal service and reduce risk of further issues.
bigstart restart syslog-ng
