OPERATIONAL DEFECT DATABASE
...
...
What are the CLI steps for SyncIQ failover and failback? The UI process has a step-by-step guide, is there was a similar guide for the CLI?
Detailed steps on doing FOFB
CLI guide for failover and failback: PowerScale OneFS 9.5.0.0 CLI Administration Guide Page 271 for 9.5 PowerScale OneFS 9.7.0.0 CLI Administration Guide Page 288 for 9.7 PowerScale OneFS 9.8.0.0 CLI Administration Guide Page 295 for 9.8 PowerScale OneFS 9.9.0.0 CLI Administration Guide Page 296 for 9.9 PowerScale OneFS 9.10.0.0 CLI Administration Guide Page 299 for 9.10 While the above documentation offers some good information, the steps below are more detailed when performing a failover and failback using CLI. The steps below use SyncIQ terminology for these two terms: SOURCE Cluster = PRIMARY TARGET Cluster = SECONDARY FAILOVER: At the PRIMARY cluster, consider running domainmark job days or weeks in advance if this is the first failover attempt for the cluster. If the dataset is large, it helps save time in speeding up the domainmark job phase. Note: A new "Accelerated Failback" option removes this step. This step must ONLY be done once. Once marked, future domainmark jobs are (see step 7 below) a no-op. # isi job jobs start domainmark --root=<path> --dm-type=synciq This marks every LIN with the appropriate protective domain ID ahead of time rather than having the failover job perform it all (see step 7). The domainmark job can take a long time depending on the size of the dataset. Stop all writing to the PRIMARY policy's path. Note: Writes on the PRIMARY policies path that occurs from this step forward are not be retained resulting in possible DL . Confirm that all writes to that path ON THE PRIMARY have stopped. At the PRIMARY cluster, backup the policy schedules and then disable all schedules by setting policies to manual. To save a backup copy of the schedules: # cat /ifs/.ifsvar/modules/tsm/config/siq-policies.gc|egrep 'common.name|schedule ' >> /ifs/.ifsvar/modules/tsm/config/policy-schedules.txt Then disable all schedules by setting policies to manual. Note: A sync job and failover job cannot run simultaneously by design and causes the failover attempt to fail. To avoid this condition, set all policies to manual. # isi sync policies modify --policy=[POLICY] --schedule="" At the PRIMARY cluster, run one last sync job and confirm it has been successfully completed. Note: This step is only recommended if testing FOFB functionality. DO NOT perform this step if the PRIMARY cluster has already encountered a failure event and the SECONDARY cluster has already been set to allow writes. # isi sync jobs start [POLICY] Run this command to confirm successful completion: # isi sync reports list --reports-per-policy=1 *Confirm the End time and State=finished At the PRIMARY cluster, run one last sync job. # isi sync jobs start [POLICY] At the SECONDARY cluster, perform 'Allow Writes' action and verify that the local job finishes that action. # isi sync recovery allow-write --policy-name=[POLICY] # isi sync target list Name Source Target Path Last Job State FOFB State ----------------------------------------------------------------------------------- qtestsync primary_clust /ifs/data/siq_quota_test finished writes_enabled ----------------------------------------------------------------------------------- Total: 1 NOTE: Reverse SmartLock directory settings as needed on both clusters. https://infohub.delltechnologies.com/en-us/l/dell-powerscale-smartlock-best-practices/synciq/ Redirect clients (SMB, NFS, HTTP, FTP, and so forth) to the SECONDARY cluster Note: The specifics of this step are outside the scope of this article and require creating SMB shares, Active Directory Domain join, Machine accounts, SPN, NFS exports, redirecting SmartConnect DNS and adding Authentication Providers. Create a recovery snapshot on both clusters before proceeding with the resync-prep ON SOURCE # isi snapshot snapshots create --path=[SOURCE_PATH] --name=SIQ-recovery-policy-[POLICY_NAME] --expires=2W ON TARGET # isi snapshot snapshots create --path=[TARGET_PATH] --name=SIQ-recovery-policy-[POLICY_NAME] --expires=2W At the PRIMARY cluster, perform the failover job with prepare resync and confirm that the resync_prep_finalize phase is completed. # isi sync recovery resync-prep --policy-name=[POLICY]# isi sync reports list --policy-name=qtestsync --sort job_id Policy Name Job ID Start Time End Time Action State --------------------------------------------------------------------------------------------- qtestsync 1 2015-02-11T08:31:27 2015-02-11T08:31:34 run finished qtestsync 2 2015-02-11T08:41:19 2015-02-11T08:41:31 resync_prep finished qtestsync 3 2015-02-11T08:41:31 2015-02-11T08:41:34 resync_prep_domain_mark finished qtestsync 4 2015-02-11T08:41:34 2015-02-11T08:41:42 resync_prep_restore finished qtestsync 5 2015-02-11T08:41:42 2015-02-11T08:41:45 resync_prep_finalize finished FAILBACK Important Note: Prior to running this step, check SECONDARY cluster for the existence of a source_record xml for the same policy ID as on the original policy. For example, for source policy ID 7da67596f099b75ad687a05f6b11781d run on target: ls -l /ifs/.ifsvar/modules/tsm/config/source_records/7da67596f099b75ad687a05f6b11781d* The new [POLICY]_mirror policy on the SECONDARY cluster can be run to start syncing back to the PRIMARY. # isi sync jobs start --policy-name=[POLICY]_mirror Stop all writing to the SECONDARY policy's path. Note: Writes on the SECONDARY policies path that occurs from this step forward are not retained resulting in possible DL. Confirm that all writes to that path ON THE SECONDARY have stopped. Disable all schedules by setting the policies to manual. # isi sync policies modify --policy=[POLICY]_mirror --schedule="" At the SECONDARY cluster, run one last sync job # isi sync jobs start --policy-name=[POLICY]_mirror At the PRIMARY cluster, perform 'Allow Writes' action and verify that the local job finishes that action. # isi sync recovery allow-write --policy-name=[POLICY]_mirror # isi sync target list Name Source Target Path Last Job State FOFB State ----------------------------------------------------------------------------------- qtestsync_mirror secondary_clust /ifs/data/siq_quota_test finished writes_enabled ----------------------------------------------------------------------------------- Total: 1 NOTE: Reverse SmartLock directory settings as needed on both clusters. https://infohub.delltechnologies.com/en-us/l/dell-powerscale-smartlock-best-practices/synciq/ Redirect clients (SMB, NFS, HTTP, FTP, and so forth) to the PRIMARY cluster Note: The specifics of this step are outside the scope of this article and require creating SMB shares, NFS exports, and redirecting SmartConnect DNS. Create a recovery snapshot on both clusters before proceeding with the resync-prep ON SOURCE # isi snapshot snapshots create --path=[SOURCE_PATH] --name=SIQ-recovery-policy-[POLICY_NAME] --expires=2W ON TARGET # isi snapshot snapshots create --path=[TARGET_PATH] --name=SIQ-recovery-policy-[POLICY_NAME] --expires=2W At the SECONDARY cluster, perform the failback job with prepare resync and confirm that the resync_prep_finalize succeeded # isi sync recovery resync-prep --policy-name=[POLICY]_mirror# isi sync reports list --policy-name=[POLICY]_mirror --sort job_id --reports-per-policy=5 Policy Name Job ID Start Time End Time Action State --------------------------------------------------------------------------------------------- qtestsync_mirror 1 2015-02-12T08:31:27 2015-02-12T08:31:34 run finished qtestsync_mirror 2 2015-02-12T08:41:19 2015-02-12T08:41:31 resync_prep finished qtestsync_mirror 3 2015-02-12T08:41:31 2015-02-12T08:41:34 resync_prep_domain_mark finished qtestsync_mirror 4 2015-02-12T08:41:34 2015-02-12T08:41:42 resync_prep_restore finished qtestsync_mirror 5 2015-02-12T08:41:42 2015-02-12T08:41:45 resync_prep_finalize finished The SECONDARY is now READ-ONLY and the SECONDARY [POLICY]_mirror policy is disabled. Note: Do not delete any mirror policies. The original policies on the PRIMARY are now enabled. Use the backup file from FAILOVER step 3 to restore your policy schedules. On the PRIMARY: View the saved copy of the policy schedules: # cat /ifs/.ifsvar/modules/tsm/config/policy-schedules.txt Restore the policy schedules: # isi sync policies modify --policy=[POLICY] --schedule=[schedule] On the original Secondary, the SIQ-<mirrorpolID>-latest snapshot will be left over after a successful failback. Manually clean up SIQ-<mirrorpolID>-latest snapshot to avoid COW writes to any existing snapshots on Secondary. # isi snapshot snapshots list ID Name Path ----------------------------------------------------------------------- 16 SIQ-recovery-policy-Test /ifs/data/failovertest 18 SIQ-005056ac0655f7f5e267a71dae70c997-latest /ifs/data/failovertest <-- pol_mirror-latest 24 SIQ-ps9715x1-Test-2025-03-25_19-20-52 /ifs/data/failovertest ----------------------------------------------------------------------- Total: 3# isi snapshot snapshots delete --id=<id>
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
