Symptoms

Upgrade flags are created on ECS for every GA version, even if the specific ECS VDC has skipped these versions during its own upgrade path. The upgrade completion flag is set after ALL VDCs in an environment are upgraded during upgrade tear down. Remote Proactive team issues the tear down operation and confirms that the upgrade flags are set. Upgrade flags tell the system that the upgrade is complete for the federation and that It can start using its new features in the new code.However, sometimes tear down can fail to set upgrade completion flags automatically. Multiple issues may lead to this situation such as inability to access the management API on the remote VDCs, which then requires manual intervention.xDoctor error: Timestamp = 2017-10-04_191656 Category = UPGRADE Source = CF_CLIENT Severity = WARNING Node = 1XX.2XX.1.1 Message = One or more upgrade flags not set Extra = {'1XX.2XX.1.4': ['com.emc.ecs.upgrade.2_2_1_upgrade_complete'], '1XX.2XX.1.3': ['com.emc.ecs.upgrade.2_2_1_upgrade_complete'], '1XX.2XX.1.2': ['com.emc.ecs.upgrade.2_2_1_upgrade_complete'], '1XX.2XX.1.1': ['com.emc.ecs.upgrade.2_2_1_upgrade_complete']} Service console tear down fails to set upgrade flags: Example of tear down failing to set and check flags on both VDC1 after an upgrade from 3.2.2.1 to 3.4.0.1. 20200315 12:59:57.488: | Check Upgrade Completion flags version across the cluster: 3.4.0.1.113843.05a36e2 Checking flags on VDC vdc1... Unable to check flags on VDC vdc1 - cannot get CF client Checking flags on VDC vdc2... [VDC vdc2] 'com.emc.ecs.upgrade.3_2_1_upgrade_complete' is 'true' [VDC vdc2] 'com.emc.ecs.upgrade.3_1_upgrade_complete' is 'true' [VDC vdc2] 'com.emc.ecs.upgrade.3_0_upgrade_complete' is 'true' [VDC vdc2] 'com.emc.ecs.timeFormat.rfc822_date_time_format' is 'false' [ERROR] CF flag 'com.emc.ecs.timeFormat.rfc822_date_time_format' check failed on VDC 'vdc2': the value is 'false' [VDC vdc2] 'com.emc.ecs.upgrade.3_3_upgrade_complete' is 'false' [ERROR] CF flag 'com.emc.ecs.upgrade.3_3_upgrade_complete' check failed on VDC 'vdc2': the value is 'false' [VDC vdc2] 'com.emc.ecs.upgrade.3_4_upgrade_complete' is 'false' [ERROR] CF flag 'com.emc.ecs.upgrade.3_4_upgrade_complete' check failed on VDC 'vdc2': the value is 'false' [VDC vdc2] 'com.emc.ecs.upgrade.2_2_upgrade_complete' is 'true' [VDC vdc2] 'com.emc.ecs.upgrade.3_2_upgrade_complete' is 'true' [VDC vdc2] 'com.emc.ecs.upgrade.3_4_0_1_upgrade_complete' is 'false' [ERROR] CF flag 'com.emc.ecs.upgrade.3_4_0_1_upgrade_complete' check failed on VDC 'vdc2': the value is 'false' [VDC vdc2] 'com.emc.ecs.upgrade.2_2_1_upgrade_complete' is 'true' [VDC vdc2] 'com.emc.ecs.upgrade.3_2_2_upgrade_complete' is 'false' [ERROR] CF flag 'com.emc.ecs.upgrade.3_2_2_upgrade_complete' check failed on VDC 'vdc2': the value is 'false' 20200315 13:00:45.926: | | FAIL (48 sec) [ERROR] 'Check Upgrade Completion flags' failed: Check Completion flags after upgrade failed on VDCs vdc1,vdc2 The below command should be run on all VDCs to manually check status of all flags: # svc_param list | grep upgrade_complete

Cause

Human error. Tear down unable to set upgrade flags due to connectivity issues between VDCs or a non-standard emcservice or emcmonitor password.

Resolution

It is critical that the upgrade complete flag is set not only for the current release, but also for all past releases. Do NOT set the upgrade complete flag for new ECS release until all zones are upgrade complete.The xDoctor is expected and can be ignored, if an upgrade activity is ongoing. If the upgrade complete flag is set prematurely, the system falls into a hybrid state and business units and tenants may lose access to their data. Open an SR with the Dell ECS support team upon meeting the following criteria. Reference this KB article, along with information to be collected from all VDCs provided below. ECS support team performs the steps to manually set the upgrade flags. Run the below on all VDCs to collect the status of the flags: Identify your scope of work (all VDCs in the federation). #svc_vdc list -d Secure the shell to each VDC and confirm that all nodes are on the same code version by running svc_version. If any version differs between nodes, they are marked by (*). Output from all VDCs should match. admin@ecs:/root> svc_version svc_version v1.1.0 (svc_tools v1.6.8) Started 2020-04-08 15:47:03 ECS Version: 3.3.0.2 GA Object Version 3.3.0.2-109511.7d24713 OS Version 3.3.0.1-327.2442c46.73 Fabric Version 3.3.0.0-3673.c1f0c9c Fabric-agent Version 3.3.0.0-3673.c1f0c9c Syslog Version 3.3.0.0-3673.c1f0c9c Zookeeper Version 3.4.9.0-85.c194184 Registry Version 2.3.1.0-64.007e532 Utilities Version 3.3.0.0-3673.c1f0c9c Service Console Version 4.4.0.0-20412.8865279e5* xDoctor Version 4.8-57 svc_tools Version 1.6.8 *Versions differ between nodes | Config Changes Mismatched Invalid Patch(es) installed | Detected Patch(es) Patch(es) -------------------------------------------------------------------------------- | -------------------------------------------------- Confirm code version history. This helps you understand the upgrade path that the system was going through. admin@ecs:/root> svc_version -history svc_version v1.1.0 (svc_tools v1.6.8) Started 2020-04-08 15:37:21 Timestamp ECS Release Version Node(s) 2016-12-22 21:36:04,984 3.0.0.0-85807.98632a9 3.0.0.0-85807.98632a9 r1n1 - r1n8 (nodes added) 2017-02-16 00:11:20,860 3.0HF1 3.0.0.0-86239.1c9e5ec r1n1 - r1n8 2017-10-27 02:52:49,872 3.0HF2 3.0.0.0-86889.0a0ee19 r1n1 - r1n8 2018-05-04 01:54:50,770 3.1 HF3 3.1.0.3.95716.cf3f257 r1n1 - r1n8 2019-03-23 08:04:25,324 3.2.2.0 GA 3.2.2.0-102274.fe37f82 r1n1 - r1n8 2019-03-23 12:44:39,487 3.2.2.1 IP 22.1 (22 hdfs) 3.2.2.1-103201.6be28a6.22.1.bugf... r1n1 - r1n8 List the flags on all VDCs. Confirm we can run svc_param on all nodes. The below example is for a direct upgrade from 3.2.1 to 3.3. Flags were created for 3.2.1, 3.2.2, and 3.3 but were not set. admin@ecs-vdc1:/root> svc_param list | grep upgrade_complete com.emc.ecs.upgrade.2_1_upgrade_complete should be enabled once all zones completed upgrade to 2.1 true com.emc.ecs.upgrade.2_2_1_upgrade_complete should be enabled once all zones completed upgrade to 2.2.1 true com.emc.ecs.upgrade.2_2_upgrade_complete should be enabled once all zones completed upgrade to 2.2 true com.emc.ecs.upgrade.3_0_upgrade_complete should be enabled once all zones completed upgrade to 3.0 true com.emc.ecs.upgrade.3_1_upgrade_complete should be enabled once all zones completed upgrade to 3.1 true com.emc.ecs.upgrade.3_2_1_upgrade_complete should be enabled once all zones completed upgrade to 3.2.1 false* com.emc.ecs.upgrade.3_2_2_upgrade_complete should be enabled once all zones completed upgrade to 3.2.2 false* com.emc.ecs.upgrade.3_2_upgrade_complete should be enabled once all zones completed upgrade to 3.2 true com.emc.ecs.upgrade.3_3_upgrade_complete should be enabled once all zones completed upgrade to 3.3 false* admin@ecs-vdc2:/root> svc_param list | grep upgrade_complete com.emc.ecs.upgrade.2_1_upgrade_complete should be enabled once all zones completed upgrade to 2.1 true com.emc.ecs.upgrade.2_2_1_upgrade_complete should be enabled once all zones completed upgrade to 2.2.1 true com.emc.ecs.upgrade.2_2_upgrade_complete should be enabled once all zones completed upgrade to 2.2 true com.emc.ecs.upgrade.3_0_upgrade_complete should be enabled once all zones completed upgrade to 3.0 true com.emc.ecs.upgrade.3_1_upgrade_complete should be enabled once all zones completed upgrade to 3.1 true com.emc.ecs.upgrade.3_2_1_upgrade_complete should be enabled once all zones completed upgrade to 3.2.1 false* com.emc.ecs.upgrade.3_2_2_upgrade_complete should be enabled once all zones completed upgrade to 3.2.2 false* com.emc.ecs.upgrade.3_2_upgrade_complete should be enabled once all zones completed upgrade to 3.2 true com.emc.ecs.upgrade.3_3_upgrade_complete should be enabled once all zones completed upgrade to 3.3 false Criteria before opening an SR: Receive an xDoctor alert that flags are not set; or Remote Proactive team upgrade and tear down failed. Confirm that the upgrades are all done on all VDCs.All nodes are upgraded and are running on the same code level.Run svc_param commands on the ECS without errors.Confirmed that we are using standard emcservice and emcmonitor user passwords (The svc_param tool use these users to set the flags.)Schedule a change window to set the upgrade flags.An SR should be opened to Dell ECS support team; reference this KB article, KB 491224, along with previously mentioned checks in this section. ECS support team helps with performing the steps to manually set the upgrade flags.

Support Cases