OPERATIONAL DEFECT DATABASE
...
...
It is critical to understand the difference between password validation and providing software with the capability to utilize credentials on behalf of the user. When software only needs to validate that a user has entered a correct password, the software only needs to store a hashed value of the known good password. When the user provides their password to authenticate, the provided password can then be hashed and compared to the known password hash to validate. Because hashing is a one-way process, it cannot be used when software must be able to utilize a password to take authorized actions on behalf of the user (e.g., entering a password into a website, automatically logging into an RDP session, or automatically connecting to a remote server without prompting the user to reenter credentials). Task automation requires OS and 3rd party software to store saved passwords in their configuration. While saved credentials are always encrypted to prevent extraction from a copy of the configuration, the actual software must be able to decrypt the passwords locally in order to be able to use them. Because the software has the functionality to decrypt the password, anyone with sufficient access rights to the OS the software is running on can extract saved credentials in plain text form by directly calling the same functions used by the software for this task. This is why tools are widely available to extract saved credentials from web browsers, applications, and even the saved credentials used to access a file share from Windows. Since backup software must be able to take automated actions and connect to protected workloads periodically and automatically to retrieve changed data, ALL backup software or backup appliances must store encrypted credentials or access keys to those workloads. No backup solution is different in that sense because, without storing credentials or access keys in a form that allows them to be extracted to plain text, such software would have to prompt users to enter a password interactively every time a workload needs to be accessed over the network for backup or restore operations. However, as discussed previously, since backup software has built-in functions to decrypt the saved credentials and extract their values in plain text, so can anyone with sufficient access rights to the OS the software is running on, be it installable software or a backup appliance. Some backup vendors practice security via obscurity and choose not to disclose the following important information about their stored credentials, even though it is as easy to extract credentials from any backup software or backup appliances once administrative access is attained, for example, through a vulnerability enabling local privilege escalation. Veeam believes strongly in full transparency and makes this information available despite its competition using this article to falsely claim that Veeam is “insecure,” even if no backup solution is any different in how it handles and accesses stored credentials. We believe full disclosure is important as it draws our customers’ attention to this security challenge that ANY enterprise management system that performs automated tasks on remote systems has, and we hope that it will prompt our customers to tighten or completely restrict direct access to their backup server OS – just as they would do for an Active Directory domain controller, for example. Veeam Backup & Replication encrypts saved credentials using native Microsoft Data Protection API, a certified cryptographic solution built-in to all Windows OSes, by leveraging the unique MachineKey of the Windows OS where the software is installed to ensure the saved credentials can only be decrypted locally on the backup server – but never from a copy of configuration database. The password encryption is further secured by utilizing an encryption salt that is unique to each backup server and is stored in a registry key only accessible by an account with local administrator privileges. These measures ensure that saved credentials can only be decrypted locally on the backup server and only by users with the highest privileges. It is therefore recommended to restrict direct access to the backup server OS, limit the number of accounts with administrative privileges to the minimum required, and patch the backup server OS vulnerabilities regularly to prevent local privilege escalation attacks. For information about Veeam Backup Server security, review the following:Veeam Backup & Replication Best Practice Guide - Security Domains Summary and Key Points: All backup software and backup appliances store saved credentials or access keys to remote systems in configuration in an encrypted form. All backup software implements a function to extract plain text credentials as needed when performing backup or restore. These same functions can also be used directly by any privileged user on the system to extract plain text credentials. The ability to recover the saved credentials or access keys from the backup software configuration is not a vulnerability. It is inherent to the nature of any enterprise management software that must perform automated actions on remote systems on behalf of a user, as such actions require authentication with plain text credentials or access keys. Veeam Backup & Replication are encrypted using Microsoft Data Protection API and are stored in the configuration database in a way that enables backup software to automatically extract them when performing backup or restore activity. Credentials can only be extracted by executing code on the machine where Veeam Backup & Replication is installed. Credentials cannot be recovered from a copy of the database, as the decryption process must be run locally on the machine that initially encrypted the credentials. Credentials can only be extracted by a privileged process because the decryption process additionally requires a unique salt stored in the registry on the machine where Veeam Backup & Replication is installed. The registry value has custom permissions assigned, which makes it readable only by privileged processes. Unauthorized credentials extraction can, therefore, be prevented by revoking administrative privileges from backup server users and restricting physical backup server access to prevent malicious actors from obtaining access to remaining accounts with administrative privileges. Veeam provides this information in the spirit of transparency to help its customers better understand potential attacks against backup servers and the importance of securing backup server access. Competition may point you to this support article and falsely claim that it proves how Veeam is “insecure.” However, no competition is any different in how it handles and accesses saved credentials. Saved credentials or access keys can be easily extracted in plain text from any backup software or backup appliance once privileged local access to the base OS is obtained.
This article documents how to recover account credentials stored within the Veeam Backup & Replication configuration database. The procedure detailed in this article can only be used to recover account credentials and not to recover backup encryption passwords.
It is beyond the scope of Veeam Support to assist with any of the operations outlined in this document. No support will be provided for recovering credentials from the Veeam Configuration Database.
Credentials Manager Cleanup In some environments, duplicate accounts may be listed within the Credentials Manager. For more information, review: KB3224: How to Clean Up the Credentials Manager in Veeam Backup & Replication.
It is critical to understand the difference between password validation and providing software with the capability to utilize credentials on behalf of the user. When software only needs to validate that a user has entered a correct password, the software only needs to store a hashed value of the known good password. When the user provides their password to authenticate, the provided password can then be hashed and compared to the known password hash to validate. Because hashing is a one-way process, it cannot be used when software must be able to utilize a password to take authorized actions on behalf of the user (e.g., entering a password into a website, automatically logging into an RDP session, or automatically connecting to a remote server without prompting the user to reenter credentials). Instead, Task automation requires OS and 3rd party software to store saved passwords in their configuration. While saved credentials are always encrypted to prevent extraction from a copy of the configuration, the actual software must be able to decrypt the passwords locally in order to be able to use them. Because the software has the functionality to decrypt the password, anyone with sufficient access right to the OS the software is running on can extract saved credentials in plain text form by directly calling the same functions used by the software for this task. This is why tools are widely available to extract saved credentials from web browsers, applications, and even the saved credentials used to access a file share from Windows. Since backup software must be able to take automated actions and connect to protected workloads periodically and automatically to retrieve changed data, ALL backup software or backup appliances must store encrypted credentials or access keys to those workloads. No backup solution is different in that sense because without storing credentials or access keys in a form that allows them to be extracted to plain text, such software would have to prompt users to enter a password interactively every time a workload needs to be accessed over the network for backup or restore operation. However, as discussed previously, since backup software has built-in functions to decrypt the saved credentials and extract their values in plain text, so can anyone with sufficient access right to the OS the software is running on, be it installable software or a backup appliance. Some backup vendors practice security via obscurity and choose not to disclose the following important information about their stored credentials, even though it is as easy to extract credentials from any backup software or backup appliances once administrative access is attained, for example, through a vulnerability enabling local privilege escalation. Veeam believes strongly in full transparency and makes this information available despite its competition using this article to falsely claim that Veeam is “insecure,” even if no backup solution is any different in how it handles and accesses stored credentials. We believe full disclosure is important as it draws our customers’ attention to this security challenge that ANY enterprise management system that performs automated tasks on remote systems has, and we hope that it will prompt our customers to tighten or completely restrict direct access to their backup server OS – just as they would do for an Active Directory domain controller, for example. Veeam Backup & Replication encrypts saved credentials using native Microsoft Data Protection API, a certified cryptographic solution built-in to all Windows OSes, by leveraging the unique MachineKey of the Windows OS where the software is installed to ensure the saved credentials can only be decrypted locally on the backup server – but never from a copy of configuration database. This ensures that saved credentials can only be decrypted locally on the backup server. It is therefore recommended that all security best practices be closely followed and access to the Veeam Backup Server be limited. For information about Veeam Backup Server security, review the following:Veeam Backup & Replication Best Practice Guide - Security Domains Summary and Key Points: All backup software and backup appliances store saved credentials or access keys to remote systems in configuration in an encrypted form. All backup software implements a function to extract plain text credentials as needed when performing backup or restore. This same function can also be used directly by any privileged user on the system to extract plain text credentials. The ability to recover the saved credentials or access keys from the backup software configuration is not a vulnerability. It is inherent to the nature of any enterprise management software that must perform automated actions on remote systems on behalf of a user, as such actions require authentication with plain text credentials or access keys. Veeam Backup & Replication are encrypted using Microsoft Data Protection API and are stored in the configuration database in a way that enables backup software to automatically extract them when performing backup or restore activity. Credentials can only be extracted by executing code on the machine where Veeam Backup & Replication is installed. Credentials cannot be recovered from a copy of the database as the decryption process must be run locally on the machine that initially encrypted the credentials. Unauthorized credentials extraction can, therefore, be prevented by any one of the following measures: Restricting remote connections to the backup server OS through RDP and other remote protocols to prevent malicious actors from being able to run code on the machine. Veeam provides this information in the spirit of transparency to help its customers better understand potential attacks against the backup server and the importance of securing backup server access. Competition may point you to this support article and falsely claim that it proves how Veeam is “insecure.” However, no competition is any different in how it handles and accesses saved credentials. Saved credentials or access keys can be easily extracted in plain text from any backup software or backup appliance once privileged local access to the base OS is obtained.
This article documents how to recover account credentials stored within the Veeam Backup & Replication configuration database. The procedure detailed in this article can only be used to recover account credentials and not to recover backup encryption passwords.
It is beyond the scope of Veeam Support to assist with any of the operations outlined in this document. No support will be provided for recovering credentials from the Veeam Configuration Database.
Credentials Manager Cleanup In some environments, duplicate accounts may be listed within the Credentials Manager. For more information, review: KB3224: How to Clean Up the Credentials Manager in Veeam Backup & Replication.
Veeam Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
