Symptoms

Dell Technologies Avamar Release Notes v7.4 and v7.5 mention the following: Vulnerability scanningAs part of every Avamar release, the product is scanned for vulnerabilities using at least two common vulnerability assessment tools. This release was scanned with Foundstone and Nessus. The Avamar solution has also been scanned by various customers by using tools such as eEye Retina without issue. However, it is possible that the usage of other port/vulnerability scanners might cause disruption to normal operation of the Avamar server. Therefore, it might be necessary to disable scanning of the Avamar server if problems occur. For Avamar Gen4T hardware, the vulnerability scanning results in the following two scenarios: 1. "BMC Internal HW Watchdog Trip" events seen in: The Avamar Administrator UI The messages file: grep -i "BMC.*Watchdog" /var/log/messages May 23 04:41:57 avamar-gen4t ipmiutil: igetevent-gen4t: 006d 05/16/17 05:01:54 MAJ BMC BMC Internal HW Watchdog Trip #e1 Cold Reset Mode (asserted) 6f [00 ff ff] May 23 04:41:58 avamar-gen4t ipmiutil: igetevent-gen4t: 0082 05/23/17 04:41:43 MAJ BMC BMC Internal HW Watchdog Trip #e1 Cold Reset Mode (asserted) 6f [00 ff ff] The "System event log" (SEL) List A dialhome may also be generated. A BMC Watchdog event generally means that the BMC became unresponsive so the system resets in an attempt to clear the condition. 2. The Avamar Administrator UI repeatedly displays old hardware messages whenever the Avamar hardware monitoring services are restarted due to the vulnerability scanning. The following events are seen in the logs: grep -i "BIOS Processor Errors.*Empty CPU Socket" /data01/cur/err.log Sep 21 03:14:01 avamar-gen4t ipmiutil: igetevent-gen4t: 0044 06/09/17 21:15:40 MAJ BIOS Processor Errors #05 CPU Socket Socket:1 70 [82 01 ff] - ELOG(27) Empty CPU Socket: 00000001 Oct 12 00:19:45 avamar-gen4t ipmiutil: igetevent-gen4t: 0044 06/09/17 21:15:40 MAJ BIOS Processor Errors #05 CPU Socket Socket:1 70 [82 01 ff] - ELOG(27) Empty CPU Socket: 00000001 The current events on the logs are reporting old events that took place some time ago. grep -i "sshd.*invalid user" /var/log/messages ... Sep 21 02:50:04 avamar-gen4t sshd[4857]: Invalid user NoSuchUser from XX.XX.XX.XXX Sep 21 02:50:04 avamar-gen4t sshd[4857]: Failed none for invalid user NoSuchUser from XX.XX.XX.XXX port XXXXX ssh2 Sep 21 02:51:02 avamar-gen4t sshd[6012]: Invalid user NoSuchUser from XX.XX.XX.XXX Sep 21 02:51:02 avamar-gen4t sshd[6012]: Failed none for invalid user NoSuchUser from XX.XX.XX.XXX port XXXXX ssh2 ... Oct 11 22:03:09 avamar-gen4t sshd[25493]: Invalid user NoSuchUser from XX.XX.XX.XXX Oct 11 22:03:09 avamar-gen4t sshd[25493]: Failed none for invalid user NoSuchUser from XX.XX.XX.XXX port XXXXX ssh2 Oct 11 22:04:54 avamar-gen4t sshd[27676]: Invalid user sysadmin from XX.XX.XX.XXX Oct 11 22:04:54 avamar-gen4t sshd[27676]: Failed none for invalid user sysadmin from XX.XX.XX.XXX port XXXXX ssh2 ... The BMC Watchdog events can be triggered by a crash of the monitoring services (hwfaultd, supervisord, ipmitool and so forth) caused by the Security Scanner: grep -i "ERROR.*ipmitool" /var/log/messages May 23 23:49:36 avamar-gen4t bmcshutdownd: ERROR: Could not get response from ipmitool for raw command 0x04 0x2D 0xF4

Cause

Vulnerability scanning causes Avamar Gen4T hardware monitoring services to crash and report false positive messages in the logs.

Resolution

1. It is recommended that Avamar grids be excluded, by way of an allow-list, from vulnerability, security, and port scanning running on final servers or endpoints. This ensures that false positive messages are not generated. 2. The "BMC Main SP1 Partition" component must be running version 23.00 or above: a. Log in to the Avamar Utility Node as admin. b. Elevate to root privilege. c. Load the root ssh keys. See Avamar: How to Log in to an Avamar Server and Load Various Keys for more information about logging on and loading keys if required. d. Run the following command: mapall --noerror --all+ --user=root showfwvers ... BMC Main SP1 Partition | 23.00 The updated BMC firmware improves the behavior of Avamar when the BMC IP address is scanned. Any node running the old firmware version must have Hotfix 336357 installed to update this component. The most up-to-date hotfix (as of September 2022) for Avamar 19.3 and higher is HF336357 and includes version 24.90. It can be obtained using one of the links below: https://dl.dell.com/downloads/XVCXK_Avamar-Gen4T-18.11-firmware-update-(Hotfix-336357).zip Searching for 336357 on the Dell support site Drivers & Downloads Note: Avamar grids running lower than v19.3 should be upgraded ASAP.

Support Cases