OPERATIONAL DEFECT DATABASE
...
...
Endpoint Detection and Response (EDR) tools such as Carbon Black are commonly used to detect and investigate suspicious activities on virtual machines(VM). When a threat is detected, an EDR tool may quarantine a VM. This action can interfere with the Horizon workflows resulting in inaccurate information being displayed in the Horizon console. This can lead an EDR admin to take incorrect actions which could potentially cause a security incident.This article describes the events, logs and status of instant clone virtual machines in the Horizon console when an endpoint is put in quarantine by an EDR tool. This will help EDR admins to set the correct policies when a threat is detected and avoid spreading the threat to other VMs.
What happens when an instant clone VM is quarantined by an EDR tool? When a threat is detected the EDR tool puts a VM in quarantine. When a VM is quarantined, the Horizon agent communication from a VM to the Horizon connection server is cut off. Status of the VM – Initially ‘Connected’ and then ‘Agent unreachable’ after the handshake timeout is reached between the agent and connection server. The below screenshot of the Horizon console shows where admins can see the status of the VMs You can check the status of a VM from the following locations in the console - Inventory -> Desktops -> MachinesInventory -> Machines -> Select a vCenter At this time, If there is any active end-user session from the Horizon client, it will get terminated. The end-user will no longer be able to connect to the VM from the Horizon client. If a Horizon Admin tries to force the session to log off from the console, it will throw an error - ‘Failed to log off session.’ Status of the VM - ‘Agent unreachable’The event log shows ‘Audit Failure’ with the message ‘User X failed to log off Machine Y’You can view the event log from Console -> Events and sessions from Console -> Sessions Because the VM is not logged off, it will not be refreshed. What happens when a VM is un-quarantined by an EDR tool?When a VM is un-quarantined, the Horizon agent communication with the connection server resumes. Status of the VM – ‘Disconnected.’The user can now log in to this VM. VM is not ‘refreshed’ unless it is forced to or a user initiates a log-off. Admins can view this session from Console -> Sessions How an incorrect interpretation of the Horizon Agent status can lead to a security incident: When a VM is quarantined, it gets disconnected from the connection server. At this time, the status of the VM changes to 'Agent Unreachable'. Because the VM is not reachable, any attempts to log off the user(From Horizon Client or Conole) will not be successful. Hence, the VM will not be refreshed to the latest golden image and snapshot.Potentially an EDR admin can incorrectly interpret 'Agent unreachable' as the successful user log-off. This assumption can lead to an incorrect conclusion that a VM is refreshed and it can be un-quarantined.However, as the VM is not refreshed, the threat may still be active and spread to other VMs when a user logs in to an unquarantined VM. Recommended approach when a threat is detected by an EDR tool on a Horizon Machine: When a threat is detected, an EDR tool puts a VM in quarantine. EDR admins should note that: The quarantined VM will not refreshed This VM will show Initially as ‘Connected’ and then ‘Agent unreachable’ after the handshake timeout in Horizon Console. Any attempt to force the log off (from Horizon client or Console) will not be successful. At this point, EDR admins should NOT take the VM out of quarantine. Here, the admin should decide what they want to do with the compromised VM. The options are: Force a refresh of the VM using the ‘Remove’ or ‘Refresh’ operation. This will re-create the VM in a pristine state from the golden image and snapshot. However, Admins will no longer be able to investigate this VM. Continue to quarantine the VM for investigation (take a snapshot or make a copy of the VM) and then Remove/Refresh to get a “pristine” VM Taking the VM out of quarantine without forcing the remove/refresh is NOT recommended. As the EDT tool puts the VM in quarantine and not the user, this user will still be able to log in to a different VM in the pool(When using floating pools). This new session is visible from the Horizon console. The status of the new VM shows as ‘Connected’ in the console, while the status of the quarantined VM will still show as ‘Agent unreachable.’ If the user terminates the Horizon client from their end, the sessions log-in Horizon Console shows an entry for the user log-off (‘User X has logged out’) and not for the VM. This should not be treated as a log-off from the machine. The quarantined VM could still be compromised and should not be taken out of quarantine.
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
