OPERATIONAL DEFECT DATABASE
...
...
You will see the below log in NSX Edge's syslog. 2023-10-31T13:00:34.914Z NSX 3186 - [nsx@6876 comp="nsx-edge" subcomp="nsx-sha" username="root" level="WARNING" s2comp="metric-collector"] Failed to collect metrics, Exception:badly formed hexadecimal UUID string, trace:Traceback (most recent call last):#012 File "/opt/vmware/nsx-netopa/lib/python/sha/core/collector/_metric_collector.py", line 157, in _collect#012 data = metric_usage()#012 File "/opt/vmware/nsx-netopa/lib/python/sha/contrib/metric/utils/_time_out_cache.py", line 15, in wrapper#012 record[1] = fn(*args, **kwargs)#012 File "/opt/vmware/nsx-netopa/lib/python/sha/contrib/metric/audit_log_health_for_remote_error_status.py", line 93, in usage#012 key = uuid.UUID(cur_logging_server[_EXPORTER_NAME])#012 File "/opt/vmware/nsx-netopa/libexec/python-3.8.3/lib64/python3.8/uuid.py", line 169, in __init__#012 raise ValueError('badly formed hexadecimal UUID string')#012ValueError: badly formed hexadecimal UUID string Lab replication and findings :**************************************NSX version : 3.2.1.2.0.205412191. CLI configuration :edge01> get logging-serversThu Nov 09 2023 UTC 06:23:27.083You create a logging-server using Edge CLI.edge01> set logging-server 1.1.1.1:514 proto tcp level warn WARNING - You are configuring tcp-based log forwarding. This will send sensitive information unencrypted over the network. The Splunk App for NSX-T only accepts TLS connections.edge01> get logging-serversThu Nov 09 2023 UTC 06:24:28.8851.1.1.1:514 proto tcp level warning exporter_name 3b2439a2-58c8-40f5-9fb3-2a456db44b53Logging server 1.1.1.1 gets configured successfully with an exporter name as UUID string 3b2439a2-58c8-40f5-9fb3-2a456db44b53.From /etc/rsyslog.conf file :-$ActionQueueType LinkedList # nsx exporter: 3b2439a2-58c8-40f5-9fb3-2a456db44b53*.warning @@1.1.1.1:514;RFC5424fmt # nsx exporter: 3b2439a2-58c8-40f5-9fb3-2a456db44b53GET : https://192.168.120.1/api/v1/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters{ "_schema": "NodeSyslogExporterPropertiesListResult", "_self": { "href": "/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters", "rel": "self" }, "result_count": 3, "results": [ { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/6bd7551b-74b8-4218-897b-ad7c78f092f7", "rel": "self" }, "exporter_name": "6bd7551b-74b8-4218-897b-ad7c78f092f7", "level": "WARNING", "port": 514, "protocol": "TCP", "server": "2.2.2.2" }, { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/3b2439a2-58c8-40f5-9fb3-2a456db44b53", "rel": "self" }, "exporter_name": "3b2439a2-58c8-40f5-9fb3-2a456db44b53", "level": "WARNING", "port": 514, "protocol": "TCP", "server": "1.1.1.1" }, { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/925eb71f-1040-48de-b66f-78e5f5033d24", "rel": "self" }, "exporter_name": "925eb71f-1040-48de-b66f-78e5f5033d24", "level": "INFO", "port": 514, "protocol": "UDP", "server": "192.168.120.200" } ]}2. API configuration :POST : https://192.168.120.1/api/v1/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters{ "exporter_name": "Amit-exporter", "facilities": ["KERN", "USER"], "level": "INFO", "msgids": ["tcpin", "tcpout"], "port": 514, "protocol": "TCP", "server": "5.5.5.5"}GET : https://192.168.120.1/api/v1/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters{ "_schema": "NodeSyslogExporterPropertiesListResult", "_self": { "href": "/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters", "rel": "self" }, "result_count": 4, "results": [ { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/6bd7551b-74b8-4218-897b-ad7c78f092f7", "rel": "self" }, "exporter_name": "6bd7551b-74b8-4218-897b-ad7c78f092f7", "level": "WARNING", "port": 514, "protocol": "TCP", "server": "2.2.2.2" }, { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/7e5a3cbb-6f06-44de-b90a-a4d29e04b2bd", "rel": "self" }, "exporter_name": "7e5a3cbb-6f06-44de-b90a-a4d29e04b2bd", "level": "WARNING", "port": 514, "protocol": "TCP", "server": "1.1.1.1" }, { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/925eb71f-1040-48de-b66f-78e5f5033d24", "rel": "self" }, "exporter_name": "925eb71f-1040-48de-b66f-78e5f5033d24", "level": "INFO", "port": 514, "protocol": "UDP", "server": "192.168.120.200" }, { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/Amit-exporter", "rel": "self" }, "exporter_name": "Amit-exporter", "facilities": [ "KERN", "USER" ], "level": "INFO", "msgids": [ "tcpin", "tcpout" ], "port": 514, "protocol": "TCP", "server": "5.5.5.5" } ]}From /etc/rsyslog.conf file :- $ActionQueueType LinkedList # nsx exporter: Amit-exporterif $msgid == 'tcpin' or $msgid == 'tcpout' then { # nsx exporter: Amit-exporterkern,user.info @@5.5.5.5:514;RFC5424fmt # nsx exporter: Amit-exporter} # nsx exporter: Amit-exporterroot@edge01:~#As we can see that if you configure exporter using API and exporter name as a string, then in /etc/rsyslog.conf, you will see nsx exporter name as string instead of UUID.
If an exporter is configured using API POST /node/services/syslog/exporters, a non-uuid string as exporter name gets created. When NSX does a audit_log_health.remote_logging_server_error check for all the rsyslog exporters, the events would be logged as it expects an UUID instead of a string as exporter name. UUID as an exporter name is only possible if exporter configuration is made with CLI instead of an API which generates a string as exporter name.
There is no impact.
A mechanism for more robust handling of exporters is introduced in NSX 4.2.0 version.
Use CLI to configure exporter.edge01> set logging-server 6.6.6.6:514 proto tcp level info WARNING - You are configuring tcp-based log forwarding. This will send sensitive information unencrypted over the network. The Splunk App for NSX-T only accepts TLS connections.NSX auto generates UUID when an exporter was configured using CLI.$ActionQueueType LinkedList # nsx exporter: 76738e49-93e2-4e88-9e6d-c6177e67cf05*.info @@6.6.6.6:514;RFC5424fmt # nsx exporter: 76738e49-93e2-4e88-9e6d-c6177e67cf05
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
