Symptoms

What are the CLI steps for SyncIQ failover and failback?The UI process has a step-by-step guide, is there was a similar guide for the CLI?

Cause

Detailed steps on doing FOFB

Resolution

CLI guide for failover and failback: PowerScaleOneFS 9.4.0.0 CLI Administration Guide Page 258 for 9.4PowerScale OneFS 9.5.0.0 CLI Administration Guide Page 270 for 9.5PowerScale OneFS 9.7.0.0 CLI Administration Guide Page 298 for 9.7 While the above documentation offers some good information, the steps below are more detailed when performing a failover and failback using CLI. The steps below use SyncIQ terminology for these two terms: SOURCE Cluster = PRIMARYTARGET Cluster = SECONDARY FAILOVER: At the PRIMARY cluster, consider running domainmark job days or weeks in advance if this is the first failover attempt for the cluster. If the dataset is large, it helps save time in speeding up the domainmark job phase. Note: A new "Accelerated Failback" option removes this step. This step must ONLY be done once. Once marked, future DomainMark jobs are (see step 7 below) a no-op. # isi job jobs start domainmark --root= --dm-type=synciq This marks every LIN with the appropriate protective domain ID ahead of time rather than having the failover job perform it all (see step 7). The domainmark job can take a long time depending on the size of the dataset. Stop all writing to the PRIMARY policy's path. Note: Writes on the PRIMARY policies path that occurs from this step forward are not be retained resulting in possible DL. Confirm with the customer that all writes to that path ON THE PRIMARY have stopped. At the PRIMARY cluster, backup the policy schedules and then disable all schedules by setting policies to manual. To save a backup copy of the schedules: # cat /ifs/.ifsvar/modules/tsm/config/siq-policies.gc|egrep 'common.name|schedule ' >> /ifs/.ifsvar/modules/tsm/config/policy-schedules.txt Then disable all schedules by setting policies to manual. Note: A sync job and failover job cannot run simultaneously by design and causes the failover attempt to fail. To avoid this condition, set all policies to manual. # isi sync policies modify --policy=[POLICY] --schedule="" At the PRIMARY cluster, run one last sync job and confirm it has been successfully completed. Note: This step is only recommended if you are testing FOFB functionality. DO NOT perform this step if the PRIMARY cluster has already encountered a failure event and the SECONDARY cluster has already been set to allow writes. # isi sync jobs start [POLICY] Run this command to confirm successful completion: # isi sync reports list --reports-per-policy=1 *Confirm the End time and State=finished At the PRIMARY cluster, run one last sync job. # isi sync jobs start [POLICY] At the SECONDARY cluster, perform 'Allow Writes' action and verify that the local job finishes that action. # isi sync recovery allow-write --policy-name=[POLICY] # isi sync target list Name Source Target Path Last Job State FOFB State ----------------------------------------------------------------------------------- qtestsync primary_clust /ifs/data/siq_quota_test finished writes_enabled ----------------------------------------------------------------------------------- Total: 1 Redirect clients (SMB, NFS, HTTP, FTP, and so forth) to the SECONDARY cluster Note: The specifics of this step are outside the scope of this article and require creating SMB shares, Active Directory Domain join, Machine accounts, SPN, NFS exports, redirecting SmartConnect DNS and adding Authentication Providers. Create a recovery snapshot on both clusters before proceeding with the resync-prep ON SOURCE # isi snapshot snapshots create --path=[SOURCE_PATH] --name=SIQ-recovery-policy-[POLICY_NAME] --expires=2W ON TARGET # isi snapshot snapshots create --path=[TARGET_PATH] --name=SIQ-recovery-policy-[POLICY_NAME] --expires=2W At the PRIMARY cluster, perform the failover job with prepare re-sync and confirm that the resync_prep_finalize phase is completed. # isi sync recovery resync-prep --policy-name=[POLICY] # isi sync reports list --policy-name=qtestsync --sort job_id Policy Name Job ID Start Time End Time Action State --------------------------------------------------------------------------------------------- qtestsync 1 2015-02-11T08:31:27 2015-02-11T08:31:34 run finished qtestsync 2 2015-02-11T08:41:19 2015-02-11T08:41:31 resync_prep finished qtestsync 3 2015-02-11T08:41:31 2015-02-11T08:41:34 resync_prep_domain_mark finished qtestsync 4 2015-02-11T08:41:34 2015-02-11T08:41:42 resync_prep_restore finished qtestsync 5 2015-02-11T08:41:42 2015-02-11T08:41:45 resync_prep_finalize finished FAILBACK Important Note: Prior to running this step, check SECONDARY cluster for the existence of a source_record xml for the same policy ID as on the original policy. For example, for source policy ID 7da67596f099b75ad687a05f6b11781d run on target: ls -l /ifs/.ifsvar/modules/tsm/config/source_records/7da67596f099b75ad687a05f6b11781d* The new [POLICY]_mirror policy on the SECONDARY cluster can be run to start syncing back to the PRIMARY. # isi sync jobs start --policy-name=[POLICY]_mirror Stop all writing to the SECONDARY policy's path. Note: Writes on the SECONDARY policies path that occurs from this step forward are not retained resulting in possible DL. Confirm with the customer that all writes to that path ON THE SECONDARY have stopped. Disable all schedules by setting the policies to manual. # isi sync policies modify --policy=[POLICY]_mirror --schedule="" At the SECONDARY cluster, run one last sync job # isi sync jobs start --policy-name=[POLICY]_mirror At the PRIMARY cluster, perform 'Allow Writes' action and verify that the local job finishes that action. # isi sync recovery allow-write --policy-name=[POLICY]_mirror # isi sync target list Name Source Target Path Last Job State FOFB State ----------------------------------------------------------------------------------- qtestsync_mirror secondary_clust /ifs/data/siq_quota_test finished writes_enabled ----------------------------------------------------------------------------------- Total: 1 Redirect clients (SMB, NFS, HTTP, FTP, and so forth) to the PRIMARY cluster Note: The specifics of this step are outside the scope of this article and require creating SMB shares, NFS exports, and redirecting SmartConnect DNS. Create a recovery snapshot on both clusters before proceeding with the resync-prep ON SOURCE # isi snapshot snapshots create --path=[SOURCE_PATH] --name=SIQ-recovery-policy-[POLICY_NAME] --expires=2W ON TARGET # isi snapshot snapshots create --path=[TARGET_PATH] --name=SIQ-recovery-policy-[POLICY_NAME] --expires=2W At the SECONDARY cluster, the perform the failback job with prepare re-sync and confirm that the resync_prep_finalize succeeded # isi sync recovery resync-prep --policy-name=[POLICY]_mirror # isi sync reports list --policy-name=[POLICY]_mirror --sort job_id --reports-per-policy=5 Policy Name Job ID Start Time End Time Action State --------------------------------------------------------------------------------------------- qtestsync_mirror 1 2015-02-12T08:31:27 2015-02-12T08:31:34 run finished qtestsync_mirror 2 2015-02-12T08:41:19 2015-02-12T08:41:31 resync_prep finished qtestsync_mirror 3 2015-02-12T08:41:31 2015-02-12T08:41:34 resync_prep_domain_mark finished qtestsync_mirror 4 2015-02-12T08:41:34 2015-02-12T08:41:42 resync_prep_restore finished qtestsync_mirror 5 2015-02-12T08:41:42 2015-02-12T08:41:45 resync_prep_finalize finished The SECONDARY is now READ-ONLY and the SECONDARY [POLICY]_mirror policy is disabled. Note: Do not delete any mirror policies. The original policies on the PRIMARY are now enabled. Use the backup file from FAILOVER step 3 to restore your policy schedules. On the PRIMARY: View the saved copy of the policy schedules: # cat /ifs/.ifsvar/modules/tsm/config/policy-schedules.txt Restore the policy schedules: # isi sync policies modify --policy=[POLICY] --schedule=[schedule]

Support Cases