OPERATIONAL DEFECT DATABASE
...
...
As occurs with VMFS and NFS datastores, VMware vSAN prevents multiple virtual machines from opening the same virtual disk (VMDK) in read-write mode. This is to protect the data stored on the virtual disk from corruption caused by multiple writers on the non-cluster-aware filesystems used by most guest OSes. To enable in-guest systems that leverage cluster-aware filesystems that have distributed write (multi-writer) capability, we must explicitly enable multi-writer support for all applicable virtual machines and VMDKs. This article outlines the process to create shareable VMDKs that reside on VMware vSAN and to enable multi-writer mode. This will permit multiple VMs to access the same VMDKs in read-write mode to enable the use of in-guest shared-storage clustering solutions such as Oracle RAC. Warning: Do not enable multi-writer mode for any VM/VMDK combination unless the guests are capable of safely arbitrating and coordinating multiple systems accessing the same storage. Enabling multi-writer mode for disks that do not use in-guest cluster-aware filesystems will result in data corruption.VMDK Clustering allows multiple virtual machines from the same or different ESXi host to simultaneously access a virtual disk on a vSAN datastore. VMware vSAN offers two types of VMDK clustering - multiwriter and “clustered VMDK”.Multiwriter allows admins to make a VMDK accessible from more than one VM simultaneously. However, it does not provide any reservations or locking mechanism for coordinating access to the VMDK for consistency. One of the most popular applications that use multiwriter is Oracle RAC. Oracle RAC manages its own locking mechanism. Several other application clustering solutions have modes for managing the locking / block reservation system. Ensure that your application is managing its own locking and reservations.“Clustered VMDK” uses a shared virtual SCSI bus. With Clustered VMDKs, vSAN implements SCSI3-PR for Microsoft Windows Server Failover Clusters (WSFC) specifically. Not all of the SCSI3-PR reservation types are supported, only those needed for MS WSFC. If you are trying to use any other application with SCSI3-PR on vSAN, it is not currently supported by VMware.Note: Enabling multi-writer mode removes support for some virtual machine operations and vSphere features. Please refer to this matrix for operation/feature supportability when multi-writer mode is enabled:Supported and unsupported actions or features Actions or Features Supported Unsupported Notes Power on, off, restart virtual machine ✓ Suspend VM × Hot add virtual disks ✓ Only to existing adapters Hot remove devices ✓ Hot extend virtual disk × Connect and disconnect devices ✓ Snapshots × Virtual backup solutions leverage snapshots through the vStorage APIs; for example, VMware Data Recovery, vSphere Data Protection. These are also not supported. Snapshots of VMs with independent-persistent disks ✓ Only the shared disks need to be in independent-persistent mode Cloning × Storage vMotion × Changed Block Tracking (CBT) × vSphere Flash Read Cache (vFRC) × Stale writes can lead to data loss and/or corruption vMotion ✓ Supported for ORAC only and limited to 8 ESX/ESXi hosts Virtual Use case: Oracle RAC, leveraged such that writes originating from two or more different virtual machines do not cause data loss.This configuration was tested and supported starting from vSphere 6.x version.Virtualized Oracle RAC is supported with vSAN iSCSI Target from vSAN 6.5 and later. For Oracle RAC VMs deployed on vSAN clusters, it's recommended to use vSAN native solution. For Oracle RAC VMs running on non-vSAN clusters, iSCSI initiators can be configured in Oracle RAC VMs to consume storage from vSAN iSCSI target Limitations and Requirements: As VMware vSAN does not support Raw Device Mappings (RDMs), this document applies only to virtual disks resident on the vSAN datastore.vSAN supports SCSI-3 persistent reservations only for Microsoft Windows Server Failover Clusters (WSFC) as of vSAN version 6.7 U3 and newer. Previous versions of vSAN do not support SCSI-3 persistent reservations.Prior to vSAN 6.7 Patch 01, using the multi-writer mode, the virtual disk must be eager-zeroed thick (EZT). When creating EZT disk on vSAN, the disk is not zeroed automatically. If zeroing is required, you must use the vmkfstools -w command to zero out all blocks. For more information, see A virtual machine fails to power on with the error: Thin/TBZ disks cannot be opened in multiwriter mode. VMware ESX cannot open the virtual disk for clustering. (1033570).Starting VMware vSAN 6.7 Patch 01, Oracle RAC on vSAN does NOT require the shared VMDKs to be Eager Zero Thick provisioned (OSR=100) for multi-writer mode to be enabled. Shared VMDKs can be thin provisioned (OSR=0) for multi-writer mode to be enabled. For existing Oracle RAC deployments migrating to this vSAN 6.7 Patch 01 vSAN version or higher, one can use SPBM (Storage Policy Based Management) to change the existing storage policy from OSR = 100 to OSR =0 (or vice versa if required). This is an online change and does not require any VM downtime.Hot adding a virtual disk removes Multi-Writer Flag. For more information, see Hot adding a virtual disk in ESXi 5.5 removes the multi-writer flag (KB 2078540)Multi-writer disk sharing is limited by concurrent lock support by the underlying file system. Up to 8 devices may share the lock concurrently. From vSphere 6.7 Update 1 onward, this limitation is removed. See detail about Generation Based Locking in below note. Note: Up to 8 Oracle RAC guests may share a disk object concurrently using the Multi-Writer Flag. VMware recommends using Anti-affinity rules to prevent multiple Oracle RAC guests participating in the same RAC cluster from residing on the same ESXi host. However, from vSphere 6.7 Update 1 on wards, the virtual disks sharing support in multi-writer has been extended to more than 8 hosts. This applies to VMFS5 / VMFS6 as well. In order to enable this feature, you need to enable /VMFS3/GBLAllowMW advanced config option as shown below:Login to ESXi host and execute the below command:#esxcli system settings advanced set -i 1 -o /VMFS3/GBLAllowMWIn a shared environment such as Raw Device Mapping (RDM) disks, you cannot use Multi-Writer Locks for virtual machines on more than 8 ESXi hosts. If you migrate a virtual machine to a ninth host, it might fail to power on with the error message Could not open xxxx.vmdk or one of the snapshots it depends on. (Too many users). This fix makes the advanced configuration option /VMFS3/GBLAllowMW visible. You can manually enable or disable multi-writer locks for more than 8 hosts by using generation-based locking. This issue is resolved in this release. For more details, see VMware knowledge base article 1034165. Physical Use case: vSAN 6.5 and later provides support for vSAN iSCSI Targets feature that presents vSAN based storage as iSCSI LUNs to iSCSI initiators.If iSCSI initiators are configured in Oracle RAC physical machines, they can share iSCSI LUNs presented from vSAN 6.5 and later.Refer vSAN 6.5 documentation for instructions on configuring vSAN iSCSI Targets. Enabling or disabling simultaneous write protection provided by VMFS using the multi-writer flag (1034165): https://kb.vmware.com/s/article/1034165
Overview The process of configuring an Oracle RAC cluster on a vSAN datastore needs to be done once per RAC cluster at creation time. This requires these steps: Creating a VM Storage Policy that will be applied to the virtual disks that will be used as the cluster’s shared storage.Before vSAN 6.7 Patch 01, creating the shared virtual disks in eager-zeroed mode was needed. Starting VMware vSAN 6.7 Patch 01, shared disks do not need to be eager zero thick (EZT) for multi-writer attribute.Attaching the shared disks to one or more VMs.Enabling multi-writer mode for the VMs and Disks.Applying the VM Storage Policy to the shared disks. Create a VM Storage Policy Depending on your Virtual Machine design specifications, you will need to define the VM Storage Policy that will be applied to the RAC shared disks. Example Storage Policy for vSAN 6.7 Patch 01 and newer: Shared vmdks with multi-writer attribute can be thin provisioned ( OSR=0%)Example Storage Policy prior to vSAN 6.7 P01 : Shared vmdks with multiwriter attribute must be thick provisioned (OSR = 100%) Notes: In this example we named the VM Storage Policy "RAC"Object Space Reservation: 100% which pre-allocates all object’s components on disk. This is only required for previous version of vSAN 6.7 Patch 01Starting from vSAN 6.7 Patch 01, Object Space Reservation (OSR) can be set to 0% for shared disks Number of failures to tolerate: 1 is the default which will provide a mirror replica on another vSAN node. This provides RAID 1 protection against one host/hardware component failure. As the dialog shows, this will consume double the storage space on the vSAN datastore.Number of disk stripes per object: The default vSAN policy is 1. For this policy select the desired stripe width. This is similar to RAID 0. In this example we are using the value of 2. For more information about storage policy configuration options, please see the VMware vSAN documentation. Add a storage controller to the Oracle RAC VMs Note: Create controllers of the same type and in the same position (SCSI address) on each Oracle RAC virtual machine. Right-click the VM in the vSphere Web Client and select Edit Settings.At the bottom of the subsequent window, select SCSI Controller from the New device: drop-down menu and then click Add.Expand the New SCSI Controller entry.Leave the Bus Sharing mode as None, and choose the desired type (LSI Parallel or LSI SAS or Paravirtual). Note: The Paravirtual controller type is generally recommended for RAC shared disks. Click OK Repeat this process on the remaining RAC VMs. For VMware vSAN 6.7 P01 and newer : Prior VMware vSAN 6.7 P01 Oracle RAC on vSAN requires the shared VMDKs to be Eager Zero Thick provisioned (OSR=100) for multi-writer mode to be enabled. Starting VMware vSAN 6.7 P01 Oracle RAC on vSAN does NOT require the shared VMDKs to be Eager Zero Thick provisioned (OSR=100) for multi-writer mode to be enabled. Creating shared disks on the first virtual machine To create shared disks on the first virtual machine using the vSphere Web Client, follow the steps as shown in the section for vSphere version 6.5.The only change, starting from vSAN 6.7 Patch 01, is shared vmdk’s does not need to be eager zero thick (EZT). You can select the storage policy with OSR set to Thin Provisioning.The rest of the steps are the same as described in the section for “vSphere version 6.5 or newer but below vSAN 6.7 Patch 01”. For all the vSAN 6.5 and higher, but lower than vSAN 6.7 P01 : Creating shared disks on the first virtual machineStarting with vSphere version 6.5, the vSphere Web Client has the option to create eager-zeroed thick (EZT) disks on the vSAN datastore. Note: The virtual disks should be added to the same SCSI positions on each virtual machine. If a disk is in position 1:0 on one virtual machine, it should be in position 1:0 on all virtual machines in the Oracle RAC.Using the vSphere Web ClientTo create shared disks on the first virtual machine using the vSphere Web Client: Right-click the appropriate virtual machine and select Edit Settings.Select New Hard Disk from the New Device drop-down menu and then click Add.Expand the New Hard Disk entry and select Thick provision eager zeroed option in the Disk Provisioning dropdown menu.In the Sharing dropdown menu, select the Multi-writer option.Change the Disk Mode to Independent-Persistent. Note: While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the cannot snapshot shared disk error when a virtual machine snapshot is taken. The use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software. Modify the Virtual Device Node, as appropriate.Click OK to save the changes.(Optional) As EZT disks created on vSAN are not zeroed automatically, you must use the command “vmkfstools -w <path-to-vmdk>” to zero out all blocks if zeroing is required. Be aware of the additional IO workload on vSAN during zeroing.Repeat this process for the remaining shared disks. Adding shared disks to one or more virtual machinesTo add shared disks to one or more virtual machines using the vSphere Web Client: Right-click the appropriate virtual machine and select Edit Settings.Select Existing Hard Disk from the New Device drop-down menu and then click Add.Navigate to the applicable directory and select the disk.Click OK.Expand the New Hard Disk entry and modify the Virtual Device Node, as appropriate.In the Sharing dropdown menu, select the Multi-writer option.Change the Disk Mode to Independent-Persistent. Note: While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the “cannot snapshot shared disk” error when a VM snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software. Click OK to save the changes.Repeat this process for the remaining virtual machines and shared disks. For vSphere version 6.0 Update 1 or newer, but below version 6.5Creating Eager-Zeroed Disks (PowerCLI) In vSphere version 6.0 U1 or later, but below version 6.5 (Example: 6.0 U3), the vSphere Web Client cannot create eager-zeroed disks on the vSAN datastore. While the vSAN Datastore supports eager-zeroed disks, this functionality is not currently exposed in the vSphere Web Client. To accommodate this limitation in the current release, we must use either PowerCLI or the ESXi command line to create the eager-zeroed disks. This section outlines the PowerCLI method to create disks, and is the preferred method. Launch PowerCLI and connect to vCenter with Connect-VIServer. For more guidance with PowerCLI conventions, login, etc., please see the VMware vSphere PowerCLI Documentation. Create the new hard disk with the New-HardDisk cmdlet in the form of: New-HardDisk -VM VM_name -CapacityGB size -Datastore vSAN_datastore_name -StorageFormat EagerZeroedThick -Controller controller_name -Persistence IndependentPersistent For example, to create a 10GB disk on the RAC_0 VM, attached to SCSI Controller 1 on the datastore vsanDatastore, run this command: New-HardDisk -VM “RAC_0” -CapacityGB 10 -Datastore “vsanDatastore” -StorageFormat EagerZeroedThick -Controller “SCSI Controller 1” -Persistence IndependentPersistent The name of the storage controller will match the name of the device we added to the VMs in the previous section.(Optional) As EZT disks created on vSAN are not zeroed automatically, you must use the command vmkfstools -w <path-to-vmdk> to zero out all blocks if zeroing is required. Be aware of the additional IO workload on vSAN during zeroing.Repeat this process for any remaining shared disks you wish to create. Creating Eager-Zeroed Disks (ESXi command line) In vSphere version 6.0 U1 or later, but below version 6.5 (Example: 6.0 U3), the vSphere Web Client cannot create eager-zeroed disks on the vSAN datastore. While the vSAN Datastore supports eager-zeroed disks, this functionality is not currently exposed in the vSphere Web Client. To accommodate this limitation in the current release, we must use either PowerCLI or the ESXi command line to create the eager-zeroed disks. This section outlines the ESXi command line method to create disks.Note: You must enable the local ESXi shell or SSH access to the host and log in as privileged (root) user to complete the below procedure. Navigate to the directory of the first VM in the Oracle RAC cluster: cd /vmfs/volumes/vsan_datastore/VM_Name For example: cd /vmfs/volumes/vsanDatastore/RAC_0 Create an Eager Zeroed Thick virtual disk to be shared using vmkfstools: vmkfstools -c size -W vsan -d eagerzeroedthick `pwd`/vmdk_ile_name For example: vmkfstools -c 12G –W vsan –d eagerzeroedthick `pwd`/RAC_0_1.vmdk(Optional) As EZT disks created on vSAN are not zeroed automatically, you must use the command “vmkfstools -w <path-to-vmdk>” to zero out all blocks if zeroing is required. Be aware of the additional IO workload on vSAN during zeroing vmkfstools -w `pwd`/vmdk_file_nameFor Example:vmkfstools -w `pwd`/RAC_0_1.vmdk Repeat step 2 for as many shared disks as need to be created. Adding shared disks to one or more virtual machines After the eager-zeroed disks are created, you must add them to the remaining RAC VMs using either the vSphere Web Client or PowerCLI. Note: The virtual disks should be added to the same SCSI positions on each virtual machine. If a disk is in position 1:0 on one virtual machine, it should be in position 1:0 on all virtual machines in the RAC cluster. Using the vSphere Web Client To add shared disks to one or more virtual machines using the vSphere Web Client: Right-click the appropriate VM and select Edit Settings.Select Existing Hard Disk from the New Device drop-down menu and then click Add.Navigate to the applicable directory and select the disk.Click OK.Expand the New Hard Disk entry and modify the Virtual Device Node, as appropriate.In the Sharing dropdown menu, select the Multi-writer option.Change the Disk Mode to Independent-Persistent. Note: While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the “cannot snapshot shared disk” error when a VM snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software. Click OK to save the changes. Repeat this process for the remaining virtual machines and shared disks. Using PowerCLI Launch PowerCLI and connect to vCenter Server with Connect-VIServer. Add the existing disk with the New-HardDisk cmdlet in the form of: New-HardDisk -VM VM_Name -DiskPath “[datastore_name] folder/disk_file” -Controller controller_name -Persistence IndependentPersistent For example, to add the disk RAC_0_1.vmdk on the datastore vsanDatastore in the folder RAC_0 to the VM RAC_1 and attach it to SCSI Controller 1, run this command: New-HardDisk -VM RAC_1 -DiskPath “[vsanDatastore] RAC_0/RAC_0_1.vmdk” -Controller “SCSI Controller 1” -Persistence IndependentPersistent Note: While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the cannot snapshot shared disk error when a virtual machine snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software. Repeat this process for the remaining virtual machines and shared disks. For vSphere version prior to 6.0 Update 1 Enabling multi-writer mode for the VMs and Disks using the ESXi shell The addition of the multi-writer flag is not possible via the vSphere Web Client version prior to 6.0 Update 1. If vSphere 6.0 Update 1 or newer is not installed, use the ESXi shell to enable multi-writer mode on the applicable virtual machines and disks. Note: As this involves modifying and loading virtual machine configurations, it is recommended that all RAC virtual machines be registered to the same ESXi host for the purposes of making these changes so we don’t have to log in to multiple hosts. The VMs can be distributed throughout the vSphere cluster after making these changes. To enable SCSI bus sharing for a particular disks, run: echo ‘scsi:.sharing = “multi-writer”’ >> path_to_VMX_file For example, to enable multi-writer for SCSI device 1:0 for the VM RAC_0 on the datastore vsanDatastore, run this command: echo ‘scsi1:0.sharing = “multi-writer”’ >> /vmfs/volumes/vsanDatastore/RAC_0/RAC_0.vmx Complete step 2 for all applicable disks and VMs, modifying the SCSI position as appropriate. For example, to share four disks, add all of these entries to the applicable VMX files: scsi1:0.sharing = "multi-writer" scsi1:1.sharing = "multi-writer" scsi1:2.sharing = "multi-writer" scsi1:3.sharing = "multi-writer" Refresh the VM configuration to apply the changes. Identify the VMs to refresh using this command: vim-cmd vmsvc/getallvms |grep -i VM_name For example, to list all VMs with RAC in the name registered to the host, run this command: vim-cmd vmsvc/getallvms |grep -i RAC You see output with the VM registration that includes the VMID in the left-most column. The VMID is a simple integer number.Refresh the configuration: vim-cmd vmsvc/reload VMID For example, if the VMID of one of the RAC VMs is 24, run this command: vim-cmd vmsvc/reload 24 Reload all applicable VMs. Applying the VM Storage Policies to the shared disks After the shared disks are created and added to all virtual machines using one of the three methods above, you must apply the storage policy created for the RAC shared disks. The policy must be applied to all applicable disks on all RAC virtual machines. In vSphere Web Client, right-click the VM and click Edit Settings.Expand the appropriate hard disk entry.Choose the desired VM storage policy from the dropdown list.Verify that the disk mode is Independent – Persistent. Note: While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the cannot snapshot shared disk error when a VM snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software. Click OK. Repeat this process for all applicable virtual machines and disks. Note: If you want to apply a different storage policy to the RAC shared disks at a later date, the policy change must be executed against all shared disks and on all virtual machines sharing that disk.
For more information on a similar process performed on VMFS datastores, see: Enabling or disabling simultaneous write protection provided by VMFS using the multi-writer flagOracle on VMware vSphere Essential Database Deployment TipsOracle Databases on VMware RAC Deployment GuideOracle Databases on VMware: Best Practices Guide 在 vSphere 6.x vSAN 数据存储上使用 Oracle RACvSphere 6.x vSAN データストアでの Oracle RAC の使用
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
