Symptoms

IDPA DP4400 - v2.2 deployment fails at 97 Percent after installing the pre-deplpyment patch: Idpa_pre_update_2.2.0.614446.tar.gz Additional Details:IDPA DP4400 Installation fails at 97% with multiple errors inc. "Error occurred while creating ESX user idpauser." with all other component deployments completed Diagnostic report for configuration errors and warningsIntegrated Data Protection Appliance Integration and Validation FAILED: Installing available upgrade patch. Error: Failed to run script file /data01/tmp/patch/validate.sh. FAILED: Failed to create ESX user idpauser. Error: Error occurred while creating ESX user idpauser. FAILED: Failed to change ESXi password. Error: Unable to update ESXi root password 10.x.y.z (where 10.x.y.z is the customers esxi host public facing IP address) Status of all other components installations completed at 100% Integrated Data Protection Appliance Configuration progress 97% Error:1 Warning:0 Diagnostic report Integrated Data Protection Appliance Integration and Validation 83% Started vCenter server deployment. vCenter server deployed successfully. Waiting for services on vCenter server. Finished vCenter server deployment. Started vCenter server configuration. Assigning Administrator privileges to root user. Setting root users password to never expire. Assigning license to vCenter server. Creating datacenter DPappliance and adding ESXi host under this datacenter. Finished vCenter server configuration. Configuring Integrated Data Protection Appliance. Setting timezone on Integrated Data Protection Appliance. Setting timezone on vCenter server. Configure Secure OpenLDAP on ACM. Deploying Data Protection vApp. Creating Data Protection vApp. Moving vApps into Data Protection vApp. Configuring Data Protection vApp. Installing available upgrade patch. FAILED: Installing available upgrade patch. Creating log bundle. Protection Storage 100% Protection Storage deployed successfully. Waiting for vCenter server. Deploying Protection Storage virtual app. Creating Protection Storage virtual app. Moving Protection Storage virtual machines into virtual app. Configuring virtual app for Protection Storage. Powering on Protection Storage virtual app. Assigning network to Protection Storage. Configuring Protection Storage. Configuring network settings. Configuring license on Protection Storage. Configuring passphrase. Configuring storage settings. Configuring cloud tier. Configuring cache tier. Configuring active tier. Creating filesys. Configuring group settings. Configuring DDBoost settings. Configuring NFS settings. Configuring SNMP settings. Configuring Protection Storage support settings. Configuring time zone settings. Configuring mail server settings. Configuring EMC connect settings. Changing default password of Protection Storage. Marking configuration setup complete. Rebooting Protection Storage. Completed. Backup Server 100% Deploying AVE. Configuring AVE data disks. Deploying Avamar virtual app. Creating virtual app for Avamar. Moving Avamar virtual machines into virtual app. Configuring virtual app for Avamar. Powering on Avamar virtual app. Configuring Avamar server. Installing ave-config package. Project ave-config: region started with task 1. Configuring license on Avamar. Downloading mcserver.xml file. Stopping MCS service. Updating settings in mcserver.xml file. Uploading mcserver.xml file. Starting MCS service. Checking MC connection with Avamar. Starting scheduler service. Starting maintenance windows scheduler service. Deploying Avamar proxy. Network identified for proxy. vCenter server added into Avamar. Proxy virtual machine deployed. Avamar virtual app created. Proxy virtual machine moved into Avamar virtual app. Adding Protection Storage as backend storage for Avamar. Completed. Skipping installation of OS rollup package on Avamar. Already done. IDPA System Manager 100% Deploying IDPA System Manager. Creating virtual app for IDPA System Manager. Moving IDPA System Manager Virtual Machine into virtual app. Powering on IDPA System Manager virtual app. Setting IDPA System Manager user interface title. Restarting msm-elg service on IDPA System manager. Integrating IDPA System Manager with Protection Storage. Integrating IDPA System Manager with Backup Server. Integrating IDPA System Manager with Search. Integrating IDPA System Manager with Reporting and Analytics. Completed. Reporting and Analytics 100% Deploying Data Protection Advisor Datastore server. Deploying Data Protection Advisor Application server. Deploying Data Protection Advisor virtual app. Creating virtual app for Data Protection Advisor. Moving Data Protection Advisor virtual app into DataProtection virtual app. Configuring virtual app for Data Protection Advisor. Powering on Data Protection Advisor virtual app. Validating if all virtual machines in Data Protection Advisor virtual app are up. Configuring Data Protection Advisor Datastore server. [Datastore Server] Setting timezone on Data Protection Advisor Datastore server. [Datastore Server] Binding Data Protection Advisor Datastore server. [Datastore Server] Binding Data Protection Advisor Application server. [Datastore Server] Tune Datastore Service on Datastore Server. [Datastore Server] Recreating Data Protection Advisor Datastore. [Datastore Server] Starting Data Protection Advisor Datastore service. [Datastore Server] Rebuilding Data Protection Advisor Datastore server tablespaces. [Datastore Server] Set Datastore Database password on Datastore Server. [Datastore Server] Setting password of superuser for Datastore Server Database. [Datastore Server] Setting agentpassword on Datastore Server. [Datastore Server] Restarting Datastore server and Data Collection Agent services. Configuring Data Protection Advisor Application server. [Application Server] Setting time zone on Data Protection Advisor Application server. [Application Server] Binding Data Protection Advisor Datastore server. [Application Server] Set Datastore Database password on Application Server. [Application Server] Tune Application Service on Application Server. [Application Server] Restarting Application Server and Data Collection Agent services. [Application Server] Setting administrator password for Data Protection Advisor Application server. [Application Server] Setting agentpassword on Application Server. [Application Server] Applying license on Data Protection Advisor Application server. Integrating Protection Storage and Avamar with Data Protection Advisor. Configuring Data Protection Advisor for monitoring Protection Storage. Setting Data Protection Advisor Data Collection Agent as default agent. Setting Protection Storage SNMP credential. Setting DELL|EMC Data Protection Data Domain SSH credential. Discovering EMC Data Domain as protection storage. Configuring Data Protection Advisor for monitoring EMC Avamar. Setting EMC Avamar viewuser credentials in Data Protection Advisor. Discovering EMC Avamar as backup application. Completed. Search 100% [ Search Index Master ] Deploying Search Index Master node. [ Search Index Master ] Expanding data disk of Search Index Master VM. [ Search Index Master ] Powering on Search Index Master node. [ Search Index Master ] Verifying connection with the Search Index Master node. [ Search Index Master ] Checking whether Search Master Configuration is Completed. [ Search Index Master ] Waiting for services on the Search Index Master node. Creating Search vApp. Moving Search VMs to vApp. Reconfiguring Search vApp. Powering on Search vApp. [ Search Index Master ] Configuring Search Index Master node. [ Search Index Master ] Setting the time zone on the Search Index Master node. Integrating Search Index Master. Integrating Search Index Master with Avamar. Adding Avamar app admin role for LDAP group dp_admin on Search Index Master. Adding Avamar search admin role for LDAP group dp_admin on Search Index Master. Completed.

Cause

The root cause of the installation failing at 97%, despite all other components succeeding at 100%, was identified as a result of the pre-deployment patch being copied, unpacked, and installed from: /data01/upgrade/ location on the ACM: customer-acm1:/data01/upgrade # ls -ltr * -rwx------ 1 root root 1552957440 Apr 30 09:55 Idpa_hf_hotfix.tgz -rw------- 1 root root 1618483200 Apr 30 14:46 Idpa_pre_update_2.2.0.614446.tar.gz Idpa_pre_update_2.2.0.614446: total 1580544 -rw-r--r-- 1 root root 65501867 Jan 28 05:27 dataprotection-2.2.0-614446.x86_64.rpm -rwxr-xr-x 1 root root 8747 Jan 28 16:32 install.sh -r--r--r-- 1 root root 757 Jan 28 16:32 applianceVersion.xml -rwxr-xr-x 1 root root 1552957440 Jan 28 16:32 Idpa_hf_hotfix.tgz customer-acm1:/data01/upgrade # The following error in the server.log files helped to identify the root cause: FAILED: Installing available upgrade patch. Error: Failed to run script file /data01/tmp/patch/validate.sh. Additional Details from the server.log: 2019-05-01 19:41:50,691 INFO [pool-121-thread-6]-util.CommandExecutor: Executing command : /bin/sh -c tar -xvf '/data01/upgrade/Idpa_pre_update_2.2.0.614446.tar.gz' -C '/data01/tmp/patch/' 2019-05-01 19:41:53,025 INFO [http-nio-8543-exec-10]-filters.BasicAuthFilter: Client IP:10.193.8.102 2019-05-01 19:41:53,025 INFO [http-nio-8543-exec-10]-filters.BasicAuthFilter: Validating token ... 2019-05-01 19:41:53,025 INFO [http-nio-8543-exec-10]-authentication.Tokenization: Token:10.193.8.102:20190501_181829is present in map 2019-05-01 19:41:53,025 INFO [http-nio-8543-exec-10]-restadapter.ConfigStatusService: Received request for retrieving appliance configuration status. 2019-05-01 19:41:53,025 INFO [http-nio-8543-exec-10]-configure.ConfigStatusCollector: Getting appliance configuration status. 2019-05-01 19:41:53,032 INFO [http-nio-8543-exec-5]-filters.BasicAuthFilter: Client IP:10.193.8.102 2019-05-01 19:41:53,032 INFO [http-nio-8543-exec-5]-filters.BasicAuthFilter: Validating token ... 2019-05-01 19:41:53,032 INFO [http-nio-8543-exec-5]-authentication.Tokenization: Token:10.193.8.102:20190501_181829is present in map 2019-05-01 19:41:53,033 INFO [http-nio-8543-exec-5]-restadapter.ConfigStatusService: Received request for retrieving appliance configuration status. 2019-05-01 19:41:53,033 INFO [http-nio-8543-exec-5]-configure.ConfigStatusCollector: Getting appliance configuration status. 2019-05-01 19:41:54,764 WARN [Timer-1]-customizemanager.CustomUserDeletor: deleteCustomUsers -->Skipping deletion operation of custom support users 2019-05-01 19:42:02,088 INFO [pool-121-thread-6]-util.CommandExecutor: Execution of command /bin/sh -c tar -xvf '/data01/upgrade/Idpa_pre_update_2.2.0.614446.tar.gz' -C '/data01/tmp/patch/' completed. Exit Value = 0 2019-05-01 19:42:02,088 INFO [pool-121-thread-6]-util.CommandExecutor: Output: 2019-05-01 19:42:02,088 INFO [pool-121-thread-6]-util.CommandExecutor: Idpa_pre_update_2.2.0.614446/ Idpa_pre_update_2.2.0.614446/install.sh Idpa_pre_update_2.2.0.614446/applianceVersion.xml Idpa_pre_update_2.2.0.614446/Idpa_hf_hotfix.tgz Idpa_pre_update_2.2.0.614446/dataprotection-2.2.0-614446.x86_64.rpm 2019-05-01 19:42:02,088 INFO [pool-121-thread-6]-util.CommandExecutor: Error: 2019-05-01 19:42:02,089 INFO [pool-121-thread-6]-util.CommandExecutor: 2019-05-01 19:42:02,089 INFO [pool-121-thread-6]-updatemanager.UpdateManagerImpl: extractTarball --> Patch file extracted successfully. 2019-05-01 19:42:02,089 INFO [pool-121-thread-6]-updatemanager.UpdateManagerImpl: validatePatch --> Trying to execute the verification script:bash /data01/tmp/patch/validate.sh 2019-05-01 19:42:02,089 INFO [pool-121-thread-6]-updatemanager.UpdateManagerImpl: Setting flag to skip version check during upgrade 2019-05-01 19:42:02,089 INFO [pool-121-thread-6]-updatemanager.UpdateManagerImpl: Created file: /data01/tmp/patch/skipVersionCheckDuringUpgrade.txt 2019-05-01 19:42:02,089 INFO [pool-121-thread-6]-util.CommandExecutor: Executing command : bash /data01/tmp/patch/validate.sh 2019-05-01 19:42:02,092 INFO [pool-121-thread-6]-util.CommandExecutor: Execution of command bash /data01/tmp/patch/validate.sh completed. Exit Value = 127 2019-05-01 19:42:02,093 INFO [pool-121-thread-6]-util.CommandExecutor: Output: 2019-05-01 19:42:02,093 INFO [pool-121-thread-6]-util.CommandExecutor: 2019-05-01 19:42:02,093 INFO [pool-121-thread-6]-util.CommandExecutor: Error: 2019-05-01 19:42:02,093 INFO [pool-121-thread-6]-util.CommandExecutor: bash: /data01/tmp/patch/validate.sh: No such file or directory 2019-05-01 19:42:02,093 INFO [pool-121-thread-6]-updatemanager.UpdateManagerImpl: Deleted file: /data01/tmp/patch/skipVersionCheckDuringUpgrade.txt 2019-05-01 19:42:02,093 ERROR [pool-121-thread-6]-updatemanager.UpdateManagerImpl: validatePatch --> Verification script failed. 2019-05-01 19:42:02,093 ERROR [pool-121-thread-6]-idpavapp.IntegrateIdpaVappTask: installPatchesAndHotfixes--> Exception in installPatchesAndHotfixes com.emc.vcedpa.common.exception.ApplianceException: Failed to run script file /data01/tmp/patch/validate.sh. at com.emc.vcedpa.updatemanager.UpdateManagerImpl.validatePatch(UpdateManagerImpl.java:614) at com.emc.vcedpa.idpavapp.IntegrateIdpaVappTask.installPatchesAndHotfixes(IntegrateIdpaVappTask.java:493) at com.emc.vcedpa.idpavapp.IntegrateIdpaVappTask.run(IntegrateIdpaVappTask.java:200) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Resolution

Move or delete anything that exists under the /data01/upgrade/ directory so that the directory is empty and then rollback the installation from: https://<ACM-IP-ADDRESS>:8543Once the pre-install patch is installed, it is no longer needed and can be deleted. If for any reason the files must be retained, then: ssh into ACM as user root.Create a directory: mkdir /tmp/pre-deploy-patchMove /data01/upgrade/* /tmp/pre-deploy-patchConfirm the /data01/upgrade is empty: ls -ltr /data01/upgrade/Then rollback the installation from: https://<ACM-IP-ADDRESS>:8543

Support Cases