OPERATIONAL DEFECT DATABASE
...
...
This article provides answers to frequently asked questions about vStorage APIs for Array Integration (VAAI).
vStorage APIs for Array Integration is a feature introduced in ESXi/ESX 4.1 that provides hardware acceleration functionality. It enables your host to offload specific virtual machine and storage management operations to compliant storage hardware. With the storage hardware assistance, your host performs these operations faster and consumes less CPU, memory, and storage fabric bandwidth.For more information about vStorage APIs for Array Integration and when it is used, see the Storage Hardware Acceleration section in the Configuration Guide for your version of ESXi/ESX.What does VAAI require?VAAI requires: ESXi/ESX 4.1 or later Standard, Enterprise, Enterprise Plus or Remote Office Branch Office licensing for ESXi/ESX hosts Storage arrays that support VAAI storage-based hardware accelerationNotes: ESXi/ESX 4.1 does not support hardware acceleration with NAS storage devices. Support for NAS storage devices was introduced in ESXi 5.x. How do I know if my storage array supports VAAI?To determine if your storage array supports VAAI, see the Hardware Compatibility List or consult your storage vendor. To enable the hardware acceleration on the storage array, check with your storage vendor. Some storage arrays require explicit activation of hardware acceleration support.Note: In VMware HCL version 2, there are options under the Features Category and Features fields where arrays can be searched against to verify if they are VAAI-certified. What are the fundamental operations used by VAAI and what are their corresponding Advanced Settings?VAAI uses these fundamental operations: Atomic Test & Set (ATS), which is used during creation and locking of files on the VMFS volume Clone Blocks/Full Copy/XCOPY, which is used to copy or migrate data within the same physical array Zero Blocks/Write Same, which is used to zero-out disk regions Thin Provisioning in ESXi 5.x and later hosts, which allows the ESXi host to tell the array when the space previously occupied by a virtual machine (whether it is deleted or migrated to another datastore) can be reclaimed on thin provisioned LUNs. Block Delete in ESXi 5.x and later hosts, which allows for space to be reclaimed using the SCSI UNMAP feature. For more information on Block Delete/SCSI UNMAP, see Using vmkfstools to reclaim VMFS deleted blocks on thin-provisioned LUNs (2014849). The fundamental operations are controlled by these advanced settings: Advanced Parameter nameDescriptionHardwareAcceleratedLockingAtomic Test & Set (ATS), which is used during creation of files on the VMFS volumeHardwareAcceleratedMoveClone Blocks/Full Copy/XCOPY, which is used to copy dataHardwareAcceleratedInitZero Blocks/Write Same, which is used to zero-out disk regionsNote: For Thin Provisioning, enabling/disabling occurs on the array and not on the ESXi host.What improvements have been made to VAAI for ESXi 5.x?To encourage customers to use larger and fewer datastores, support for Thin Provisioning VAAI primitive has been added in ESXi 5.0.In ESXi 5.x, support for NAS Hardware Acceleration is included with support for these primitives: Full File Clone – Like the Full Copy VAAI primitive provided for block arrays, this Full File Clone primitive enables virtual disks to be cloned by the NAS device. Native Snapshot Support – Allows creation of virtual machine snapshots to be offloaded to the array. Extended Statistics – Enables visibility to space usage on NAS datastores and is useful for Thin Provisioning. Reserve Space – Enables creation of thick virtual disk files on NAS.Note: Previously, the only supported VMDK type that could be created on NAS was thin. Support for Block Delete to reclaim space use on thin provisioned LUNs on support arrays is also included in ESXi 5.x. For more information on Block Delete/SCSI UNMAP, see Using vmkfstools to reclaim VMFS deleted blocks on thin-provisioned LUNs (2014849).How do I know if VAAI is enabled?To determine if VAAI is enabled using the vSphere Client: In the vSphere Client inventory panel, click the host. Click the Configuration tab, then click Advanced Settings under Software. Check that these options are set to 1 (enabled):DataMover.HardwareAcceleratedMoveDataMover.HardwareAcceleratedInitVMFS3.HardwareAcceleratedLockingNote: These options are enabled by default. On ESXi/ESX 4.x hosts, to determine if VAAI is enabled using the service console in ESX or the vCLI in ESXi, use the esxcfg-advcfg command to check if the options are set to 1 (enabled):# esxcfg-advcfg -g /DataMover/HardwareAcceleratedMove# esxcfg-advcfg -g /DataMover/HardwareAcceleratedInit# esxcfg-advcfg -g /VMFS3/HardwareAcceleratedLockingOn ESXi 5.x hosts, to determine if VAAI is enabled, run these commands and check if Int Value is set to 1 (enabled):# esxcli system settings advanced list -o /DataMover/HardwareAcceleratedMove# esxcli system settings advanced list -o /DataMover/HardwareAcceleratedInit# esxcli system settings advanced list -o /VMFS3/HardwareAcceleratedLockingYou see output similar to:Path: /VMFS3/HardwareAcceleratedLockingType: integerInt Value: 1 <-- set to 1 if enabledDefault Int Value: 1Min Value: 0Max Value: 1String Value:Default String Value:Valid Characters:Description: Enable hardware accelerated VMFS locking (requires compliant hardware)Note: For Thin Provisioning, enabling/disabling occurs on the array and not on the ESXi host.When enabling or disabling VAAI settings, is a reboot required?No, a host reboot is not required. The changes made may or may not be effective immediately depending on use case. For details please see below table:. Use Case Effective ESXi Host reboot Needed Volume mount/unmount Needed Remark Disabling Hardware Accelerated Move (XCOPY) in ESXi Immediate No No Disabling Hardware Accelerated Init (WRITESAME) in ESXi Immediate No No Disabling ATS Usage at ESXi host level - For upgraded VMFS datastores a.k.a Non ATS-Only VMFS datastores Immediate No No Disabling ATS Usage at ESXi host level - for ATS-Only VMFS datastores No No Yes Not effective unless the datastore is unmounted and mounted back.If you reboot the ESXi host to achieve the mount/unmount you will potentially end up in a mixture i.e. ATS+SCSI locking till all the host are rebooted which is not recommended Disabling ATS usage per VMFS datastore - For upgraded VMFS datastores a.k.a Non ATS-Only VMFS datastores NA NA NA There is no option/need to disable ATS on such datastore. Disabling ATS usage per VMFS datastore - For ATS-Only VMFS datastores No No Yes Datastore must be unmounted before executing command to disable ATS usage. Disabling ATS Usage only for VMFS heartbeats Immediate No No What happens if I have VAAI enabled on the host but some of my disk arrays do not support it?When storage devices do not support or provide only partial support for the host operations, the host reverts to its native methods to perform the unsupported operations.Are there any caveats that I should be aware of in ESXi 5.x?Use of the Atomic Test-and-Set (ATS) primitive depends on the type of filesystem: On VAAI HardwareNew VMFS-5Upgraded VMFS-5VMFS-3Single-extent datastoreATS only [1]ATS, but fall back to SCSI-2 reservationsATS, but fall back to SCSI-2 reservationsMulti-extent datastoreOnly allow spanning on ATS hardware [2]ATS, except when locks on non-headATS, except when locks on non-headNotes: If a new VMFS-5 is created on a non-ATS storage device, SCSI-2 reservations will be used. When creating a multi-extent datastore where ATS is used, the vCenter Server will filter out non-ATS devices, so that only devices that support the ATS primitive can be used. Are there any caveats that I should be aware of in ESXi/ESX 4.x?There are a few reasons why a hardware-accelerated operation may fail.For any primitive that the array does not implement, the array returns an error, which triggers the ESX data mover to attempt the operation using software data movement. In the case of ATS, it reverts to using SCSI Reservations.VAAI hardware offload cannot be used when: The source and destination VMFS volumes have different block sizes The source file type is RDM and the destination file type is non-RDM (regular file) The source VMDK type is eagerzeroedthick and the destination VMDK type is thin The source or destination VMDK is any kind of sparse or hosted format Cloning a virtual machine that has snapshots because this process involves consolidating the snapshots into the virtual disks of the target virtual machine. The logical address and/or transfer length in the requested operation is not aligned to the minimum alignment required by the storage device (all datastores created with the vSphere Client are aligned automatically) The VMFS datastore has multiple LUNs/extents spread across different arrays Notes: When using enhanced vMotion (move both VM and datastore simultaneously), the source and destination hosts must both see the source and destination datastores, or the file transfer defaults to NFC over the the management network, instead of VAAI. Hardware cloning between arrays (even if within the same VMFS datastore) does not work. For more information on supportability with Horizon View, see View Composer API for Array Integration (VCAI) support in VMware Horizon View (2061611). What do I need to know about the Hardware Acceleration Support Status?If you go to Host > Configuration > Storage, you can see the Hardware Acceleration Status in the panel on the right side.For each storage device and datastore, the vSphere Client displays the hardware acceleration support status in the Hardware Acceleration column of the Devices view and the Datastores view.The status values are Unknown, Supported, and Not Supported. The initial value is Unknown. The status changes to Supported after the host successfully performs the offload basic operations. If the offload operation fails, the status changes to Not Supported.To determine if your storage device supports VAAI, test the Full Copy VAAI primitive: Using the vSphere Client, browse the datastore and locate a virtual disk (VMDK) of at least 4 MB that is not in use. Copy the virtual disk to a new file. Check the Hardware Acceleration status to verify that it changes from Unknown to either Supported or Not Supported. Note: VAAI primitives can also be tested by creating a virtual machine with at least one new virtual disk, or cloning a virtual machine.Can I check the VAAI status from the command line? On ESXi 5.xTo check the VAAI status, run this command:# esxcli storage core device vaai status getYou see output similar to:mpx.vmhba1:C0:T1:L0VAAI Plugin Name:ATS Status: unsupportedClone Status: unsupportedZero Status: unsupportedDelete Status: unsupportedmpx.vmhba1:C0:T0:L0VAAI Plugin Name:ATS Status: unsupportedClone Status: unsupportedZero Status: unsupportedDelete Status: unsupported On ESXi/ESX 4.xTo check the VAAI status, run this command:# esxcfg-scsidevs -l | egrep "Display Name:|VAAI Status:"You see output similar to:Display Name: Local VMware Disk (mpx.vmhba2:C0:T0:L0)VAAI Status: unsupportedDisplay Name: Local USB CD-ROM (mpx.vmhba32:C0:T0:L0)VAAI Status: unknownDisplay Name: Local TEAC CD-ROM (mpx.vmhba3:C0:T0:L0)VAAI Status: unknownDisplay Name: DGC Fibre Channel Disk (naa.6006016030801c00f8506073d0d7de11)VAAI Status: unknownDisplay Name: DGC Fibre Channel Disk (naa.6006016030801c00f9506073d0d7de11)VAAI Status: unknown Are changes to VAAI settings logged?Changes to VAAI advanced settings (for example, enabling or disabling VAAI) are recorded in the VMkernel logs at /var/log/vmkernel (for ESXi 5.0 vmkernel.log) or /var/log/messages.After enabling VAAI using the advanced settings, the VMkernel log contains entries similar to:cpuN:1234)Config: 297: "HardwareAcceleratedMove" = 1, Old Value: 0, (Status: 0x0)cpuN:1234)Config: 297: "HardwareAcceleratedInit" = 1, Old Value: 0, (Status: 0x0)cpuN:1234)Config: 297: "HardwareAcceleratedLocking" = 1, Old Value: 0, (Status: 0x0)After disabling VAAI using the advanced settings, the VMkernel log contains entries similar to:cpuN:1234)Config: 297: "HardwareAcceleratedMove" = 0, Old Value: 1, (Status: 0x0)cpuN:1234)Config: 297: "HardwareAcceleratedInit" = 0, Old Value: 1, (Status: 0x0)cpuN:1234)Config: 297: "HardwareAcceleratedLocking" = 0, Old Value: 1, (Status: 0x0) VAAI For NAS Storage In ESXi 5.x, NAS Hardware Acceleration was introduced with support for these primitives: · Full File Clone – Like the Full Copy VAAI primitive provided for block arrays, this Full File Clone primitive enables virtual disks to be cloned by the NAS device. · Native Snapshot Support – Allows creation of virtual machine snapshots to be offloaded to the array. · Extended Statistics – Enables visibility to space usage on NAS datastores and is useful for Thin Provisioning. · Reserve Space – Enables creation of thick virtual disk files on NAS. With NAS storage devices, the hardware acceleration integration is implemented through vendor-specific NAS plug-ins. These plug-ins are typically created by vendors and are distributed as VIB packages through a website. No claim rules are required for the NAS plug-ins to function. Installing the NAS Plugin 1. Place the host into maintenance mode 2. Set the host acceptance level: # esxcli software acceptance set–-level=value The host acceptance level must be the same or less restrictive than the acceptance level of any VIB you want to add to the host. The value can be one of the following: · VMwareCertified · VMwareAccepted · PartnerSupported · CommunitySupported 3. Install the VIB package: # esxcli software vib install –-viburl=URL The URL specifies the URL to the VIB package to install. http:, https:, ftp:, and file: are supported. 4. Restart the host for the installation to take effect Verify Hardware Acceleration Status for NAS You can verify the NAS hardware acceleration status using the following command: # esxcli storage nfs list The Hardware Acceleration column in the output shows the status Disabling VAAI when using NAS storage To disable NAS VAAI functionality the plugin must be removed using the below commands. 1. Log into ESXi using SSH/DCUI and execute command:# esxcli software vib remove –-vibname=nameFor Example: # esxcli software vib remove –-vibname=NetAppNasPlugin 2. Verify that the plugin has been removed using command:esxcli software vib list 3. Reboot the ESXi host to take effect. Note: The advanced configuration options for Block VAAI mentioned above (DataMover.HardwareAcceleratedMove, DataMover.HardwareAcceleratedInit, VMFS3.HardwareAcceleratedLocking) do not affect NAS VAAI functionality
For more information, see the vSphere Storage Guide for your release of vSphere: vSphere 5.5 Storage Guide vSphere 5.1 Storage Guide vSphere 5.0 Storage Guide vSphere 4.1 SAN Configuration Guide This is a list of commonly used SCSI opcodes related to VAAI operations: 0x93WRITE SAME(16) 0x41WRITE SAME(10) 0x42UNMAP0x89SCSI COMPARE and WRITE - ATS0x83EXTENDED COPY Occasionally, failures during VAAI operations would be logged in the vmkernel.log file under /var/log as shown in this example:Cmd(0x41248092e240) 0x42, CmdSN 0x13bb23 to dev "naa.xxxxxxxx" failed H:0x0 D:0x2How to reclaim VMFS deleted blocks on thin-provisioned LUNsvStorage APIs for Array Integration 常见问题esxtop コマンドを使用して、 VAAI の監視および状態確認をする方法
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
