OPERATIONAL DEFECT DATABASE
...
...
No hostname (or ipaddr for that matter) in SYSLOG msg content. tcpdump ----------------------------------------------------------------- Raw message received from 'v6-n9508-agg1' (192.168.10.191): root@avr-lubuntu[~]# tcpdump -nXli eth1 host 192.168.10.191 and port 514 16:15:23.043290 IP 192.168.10.191.514 > 192.168.12.220.514: SYSLOG local7.notice, length: 115 0x0000: 4500 008f 3227 0000 3f11 b04b c0a8 0abf E...2'..?..K.... 0x0010: c0a8 0cdc 0202 0202 007b 1346 3c31 3839 .........{.F<189 0x0020: 3e3a 2032 3031 3420 4665 6220 3139 2031 >:.2014.Feb.19.1 0x0030: 393a 3134 3a35 3920 4553 543a 2025 5653 9:14:59.EST:.%VS 0x0040: 4844 2d35 2d56 5348 445f 5359 534c 4f47 HD-5-VSHD_SYSLOG 0x0050: 5f43 4f4e 4649 475f 493a 2043 6f6e 6669 _CONFIG_I:.Confi 0x0060: 6775 7265 6420 6672 6f6d 2076 7479 2062 gured.from.vty.b 0x0070: 7920 6164 6d69 6e20 6f6e 2031 3932 2e31 y.admin.on.192.1 0x0080: 3638 2e31 322e 3232 3040 7074 732f 31 68.12.220@pts/1 Note above there is no information in the actual msg indicating where the msg came from. This is in contrast to messages received from ASR1Ks. Here is an example from 'v6-asr1k-pe1' (192.168.10.195) in response to a 'conf t' and 'ctrl-z': root@avr-lubuntu[~]# tcpdump -nXli eth1 host 192.168.10.195 and port 514 16:31:51.719438 IP 192.168.10.195.57055 > 192.168.12.220.514: SYSLOG local7.notice, length: 121 0x0000: 4500 0095 0a73 0000 fe11 18f5 c0a8 0ac3 E....s.......... 0x0010: c0a8 0cdc dedf 0202 0081 5798 3c31 3839 ..........W.<189 0x0020: 3e33 3135 353a 2076 362d 6173 7231 6b2d >3155:.v6-asr1k- 0x0030: 7065 313a 202a 4665 6220 3139 2031 373a pe1:.*Feb.19.17: 0x0040: 3330 3a31 322e 3934 303a 2025 5359 532d 30:12.940:.%SYS- 0x0050: 352d 434f 4e46 4947 5f49 3a20 436f 6e66 5-CONFIG_I:.Conf 0x0060: 6967 7572 6564 2066 726f 6d20 636f 6e73 igured.from.cons 0x0070: 6f6c 6520 6279 2061 646d 696e 206f 6e20 ole.by.admin.on. 0x0080: 7674 7931 2028 3139 322e 3136 382e 3132 vty1.(192.168.12 0x0090: 2e32 3230 29 .220) You can clearly see in the dump above (ASCII interpreted column on the right) that the hostname is part of the syslog message. On ASR1K we can specify "logging origin_id hostname". Something similar to this would be great on N9K, or maybe better put, in NX-OS (I'm not sure if this is a limitation of all NX-OS devices or just N9K).
Here is how the environment was configured: Unlike the ASR1Ks, since the Nexus 9Ks do not appear send hostname, nor provide any configuration knobs to specify an "origin-id", we had to key off ipaddr: root@avr-lubuntu[~]# cat /etc/rsyslog.d/40-network.conf :fromhost-ip, startswith, "192.168" /var/log/network/all.log :msg, contains, "v6-asr1k-pe1" /var/log/network/v6-asr1k-pe1.log & ~ :msg, contains, "v6-asr1k-pe2" /var/log/network/v6-asr1k-pe2.log & ~ :msg, contains, "v6-asr1k-pe3" /var/log/network/v6-asr1k-pe3.log & ~ :fromhost-ip, isequal, "192.168.10.191" /var/log/network/v6-n9508-agg1.log & ~ :fromhost-ip, isequal, "192.168.10.192" /var/log/network/v6-n9508-agg2.log & ~ :fromhost-ip, isequal, "192.168.10.193" /var/log/network/v6-n9396-p1a.log & ~ :fromhost-ip, isequal, "192.168.10.194" /var/log/network/v6-n9396-p1b.log & ~ :fromhost-ip, isequal, "192.168.10.198" /var/log/network/v6-n6001-trsw1.log & ~ :fromhost-ip, isequal, "192.168.10.199" /var/log/network/v6-n6001-trsw2.log & ~ :fromhost-ip, isequal, "192.168.100.51" /var/log/network/v6-gold101-csr1 & ~ :fromhost-ip, isequal, "192.168.101.51" /var/log/network/v6-gold101-csr2 & ~ :fromhost-ip, isequal, "192.168.112.51" /var/log/network/v6-gold101-vpx01 & ~ :fromhost-ip, isequal, "192.168.114.51" /var/log/network/v6-gold101-vpx02 & ~ :fromhost-ip, isequal, "192.168.100.52" /var/log/network/v6-gold102-csr1 & ~ :fromhost-ip, isequal, "192.168.101.52" /var/log/network/v6-gold102-csr2 & ~ :fromhost-ip, isequal, "192.168.112.52" /var/log/network/v6-gold102-vpx01 & ~ :fromhost-ip, isequal, "192.168.114.52" /var/log/network/v6-gold102-vpx02 & ~ Rsyslog listening on port 514: root@avr-lubuntu[~]# netstat -an --inet | grep 514 tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN tcp 0 0 172.18.116.220:22 10.20.215.212:53514 ESTABLISHED udp 0 0 0.0.0.0:514 0.0.0.0:* Logging directory, files are getting populated with data and being rotated correctly: adrichar@avr-lubuntu[~/svnwork/Configs/v6]% ll /var/log/network | grep v6-n9 -rw-r----- 1 syslog 50075 Feb 20 06:59 v6-n9396-p1a.log -rw-r----- 1 syslog 375062 Feb 18 19:24 v6-n9396-p1a.log.1 -rw-r----- 1 syslog 32410 Feb 20 06:59 v6-n9396-p1b.log -rw-r----- 1 syslog 345036 Feb 20 04:37 v6-n9396-p1b.log.1 -rw-r----- 1 syslog 19091 Feb 18 19:24 v6-n9396-p1b.log.2.gz -rw-r----- 1 syslog 621536 Feb 20 11:50 v6-n9508-agg1.log -rw-r----- 1 syslog 399265 Feb 19 04:54 v6-n9508-agg1.log.1 -rw-r----- 1 syslog 28196 Feb 17 04:53 v6-n9508-agg1.log.2.gz -rw-r----- 1 syslog 47129 Feb 15 04:59 v6-n9508-agg1.log.3.gz -rw-r----- 1 syslog 643928 Feb 20 11:50 v6-n9508-agg2.log -rw-r----- 1 syslog 511786 Feb 18 04:53 v6-n9508-agg2.log.1 -rw-r----- 1 syslog 32865 Feb 15 04:58 v6-n9508-agg2.log.2.gz Logfile of 'v6-n9508-agg1' root@avr-lubuntu[~]# tail /var/log/network/v6-n9508-agg1.log Feb 19 19:10:23 v6-n9508-agg1 : 2014 Feb 19 19:10:00 EST: %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed for user User Access Verification from console - login Feb 19 19:11:36 v6-n9508-agg1 : 2014 Feb 19 19:11:13 EST: %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed for user rd: ^G from console - login Feb 19 19:12:40 v6-n9508-agg1 : 2014 Feb 19 19:12:17 EST: last message repeated 1 time Feb 19 19:13:34 v6-n9508-agg1 : 2014 Feb 19 19:13:11 EST: %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed for user User Access Verification from console - login Feb 19 19:14:48 v6-n9508-agg1 : 2014 Feb 19 19:14:24 EST: %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed for user d: ^G from console - login Feb 19 19:15:23 v6-n9508-agg1 : 2014 Feb 19 19:14:59 EST: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on 192.168.12.220@pts/1 Feb 19 19:15:43 v6-n9508-agg1 : 2014 Feb 19 19:15:20 EST: %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed for user User Access Verification from console - login Feb 19 19:16:57 v6-n9508-agg1 : 2014 Feb 19 19:16:34 EST: %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed for user rd: ^G from console - login Feb 19 19:18:00 v6-n9508-agg1 : 2014 Feb 19 19:17:37 EST: last message repeated 1 time Feb 19 19:19:04 v6-n9508-agg1 : 2014 Feb 19 19:18:41 EST: %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed for user from console - login Note how hostname shows up in the syslog file. But this is only because rsyslogd will resolve incoming msgs to hostname from packet source ipaddr, whether in DNS or /etc/hosts. In our case, it's in /etc/hosts: adrichar@avr-lubuntu[~/svnwork/Configs/v6]% grep "n9" /etc/hosts 192.168.10.191 v6-n9508-agg1 192.168.10.192 v6-n9508-agg2 192.168.10.193 v6-n9396-p1a 192.168.10.194 v6-n9396-p1b
With rsyslogd I have configured /etc/hosts as a temporary workaround. rsyslogd will resolve ipaddrs from packet headers. But this isn't a necessarily a scalable solution, nor does it work on tools that take msgs off the wire.
This enhancement is fixed on N9K in 7.0(3)I3(1) release onwards.
Click on a version to see all relevant bugs
Cisco Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
