Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
This article provides information about Storage DRS.
What is Storage DRS? Storage DRS is an intelligent vCenter Server feature for efficiently managing VMFS and NFS storage, similar to DRS which optimizes the performance and resources of your vSphere cluster. What are the key metrics Storage DRS revolves around? Storage DRS revolves around two storage metrics, Space and IO. Does Storage DRS fully support VMFS and NFS storage? Yes, Storage DRS fully supports VMFS and NFS datastores. However, it does not allow adding NFS datastores and VMFS datastores into same datastore cluster. What are the core Storage DRS features? Resource aggregation: It enables to group multiple datastores in to a single flexible pool of storage called a Datastore Cluster (aka Storage DRS POD).Initial placement: This feature takes care of disk placement for operations such as Create Virtual Machine, add disk, clone and relocate.Load balancing based on Space and IO: Storage DRS dynamically balances the Storage DRS cluster imbalance based on Space and IO threshold set. Default space threshold per datastore is 80% and default IO latency threshold is 15ms.Datastore maintenance mode: This feature helps when an admin wants to do maintenance activities on storage. Similar to host maintenance mode, Storage DRS will Storage vMotion all the virtual machine files.Inter/Intra VM Affinity rules: As name states, we can have affinity/anti-affinity rules between virtual machines or VMDKs. What are the requirements of the Storage DRS cluster? VMware vCenter server 5.0 or laterVMware ESXi 5.0 or laterVMware vSphere compute/hosts cluster (recommended)VMware vSphere enterprise plus licenseShared VMFS or NFS datastore volumesShared datastore volumes accessible by at least one ESXi host inside the cluster. VMware recommends to have full cluster connectivity, however, it is not enforcedDatastore inside Storage DRS cluster must be visible in only one data center What are the features/solution Storage DRS interop (fully supported) with? Storage DRS interop with Site recovery Manager (SRM)Storage DRS interop with vSphere ReplicationStorage DRS interop with SPBM (Storage policy based management). For more information, see Storage DRS integration with storage profiles (2142765). Deep integration with vSphere APIs for Storage Awareness (VASA). Storage DRS now understands storage array advanced features such as deduplication, auto-tiering, snapshotting, replication, thin-provisioning.Storage DRS interop with Content LibraryStorage DRS has interop with various sub-features For example: Storage DRS interop with RDM, thin provisioning disks Storage DRS works with solutions like vCD, vRA, Horizon view etc. For more information, see vSphere 5.1 Storage DRS Interoperability. How do you enable/configure Storage DRS? For more information, see the Enable and Disable Storage DRS section in the vSphere 6.5 Resource Management Guide. What are the Storage DRS workflows users can perform? For more information, see the Storage DRS Workflows section in the vCenter Server and Host Management. What virtual machine operations come under initial placement for Storage DRS? Initial placement operations considered are: Create a virtual machineRelocate a virtual machineAdd new vmdk to the virtual machineClone a virtual machine Does Storage DRS has affinity rules like DRS? Yes, Storage DRS has affinity rules the way DRS has. VMDK keep together: This is the default rule. It keeps all the virtual machine VMDKs on the same datastore (i.e. Under virtual machine working directory).VMDK-VMDK anti-affinity (intra VM anti-affinity): Users can configure this rule if they want to keep each VMDK (from particular virtual machine) on the separate datastore.VM-VM anti-affinity (InterVM anti-affinity): Users can configure this rule if they want to keep virtual machines on separate datastores. Does Storage DRS leverage storage vMotion functionality? Yes, Storage DRS leverages Storage vMotion for moving virtual machine files from a datastore to recommended datastore. What automation levels does Storage DRS support? Storage DRS supports two automation levels unlike DRS (where partially automated mode is also there). Storage DRS can be in either fully automated mode or manual mode. What are the constraints on Storage DRS? VMFS, NFS cannot be part of the same datastore clusterMax 64 datastores per datastore clusterMax 256 PODs per vCenter serverMax 9000 VMDKs per POD What are some of the major best practices while configuring a Storage DRS cluster? Group disk with similar characteristics (RAID-1 with RAID-1, replicated with replicated, 15k RPM with 15k RPM) i.e. Identical storage profiles.Cluster-wide consistent datastore connectivity. This means every host in the cluster should be able to see all datastores participating in the storage DRS cluster. This improves DRS and Storage DRS performance.Do not mix virtualized and non-virtualised IO workloadPair I/O latency Threshold with disk Type. i.e. SSD disk : 10-15 ms. FC/SAS: 20-40 ms, SATA disks: 30-50 ms. Does Storage DRS violate space thresholds? Yes, Storage DRS may violate space thresholds when there is no datastore in the cluster, which is below the space threshold. Storage Space threshold is a soft constraint used by Storage DRS for balancing and defragmentation. It is not hard limit. Storage DRS tries to keep virtual machines on datastores based on space threshold but Storage DRS does not guarantee that you will have always some amount of free space in datastores. Storage DRS affinity rules also can lead to threshold violation. Is there a threshold priority? For example: If both I/O Threshold and Space Threshold cannot be satisfied on 1 datastore, which Threshold would Storage DRS drop first to try and place the virtual machine? When Storage DRS runs, it is possible that both space and I/O thresholds are violated which causes the load-balancing algorithm to run. Storage DRS tries to correct both of them, however, correction is not guaranteed to be successful. Should we have many small LUNs or some large size LUNs for Storage DRS? In fact, it depends. You could create 2x 64TB LUNs, 4x 32TB LUNs, 16x 8TB LUNs or 32x 4TB LUNs. When there are more datastores, SDRS will have more option to find right datastore to fit the virtual machine to placed or moved. However, it does not mean that we should always go for many small datastores. It is better to try to find the the sweet spot for your environment by taking failure domain (backup restore time), IOps, queues (SIOC) and load balancing options for Storage DRS in to account. For more information, see Should I use many small LUNs or a couple large LUNs for Storage DRS? More datastores in cluster > Better space and I/O balanceLarger datastore size > better space balance What are the various disk types Storage DRS supports? Storage DRS supports following disk types: Thick provisionedThin provisionedIndependent diskvSphere Linked clones Can I take control of virtual machine initial placement by manually selecting the datastore? Yes, users can take control of initial placement. If user picks the datastore, Storage DRS gets disabled on that virtual machine and Storage DRS will not consider this virtual machine for moves but space utilized by this virtual machine would be considered by Storage DRS for making needed recommendations. Does Storage DRS prefer moving Powered-off VMs to Powered ON? Yes, Storage DRS prefers moving powered-off virtual machines to Powered on virtual machines to reduce the storage vMotion overhead. In case of moving powered-off virtual machine, there is no need to track the VMDK block changes. How does the initial placement of virtual machine with multiple disks work: calculation on the virtual machine, or on the individual disks? Disks are considered individually but depending on virtual machines disk affinity. They can be on a same datastore or placed on different datastores. However, disks are considered individually. Does Storage DRS consider VM swap files? Initial placement algorithm does not consider a swap file. Storage DRS initial Placement algorithm does not take virtual machine swap file capacity into account. However, subsequent rebalance calculations are based on space usage of all datastores. Therefore, if a virtual machine is powered on and has a swap file, it is counted toward the total space usage. Swap file size is dependent on virtual machine RAM and reserved RAM. If reserved RAM is equal to RAM assigned to virtual machine, there will be no swap file for that virtual machine. Also, there is a way to dedicate one of the datastores as swap file datastore where all the swap files from all the virtual machines will be stored. Storage DRS uses the construct DrmDisk as the smallest entity it can migrate. This mean that Storage DRS creates a DrmDisk for each VMDK belonging to the virtual machine. The interesting part is how it handles the collection of system files and swap file belonging to the virtual machine. Storage DRS creates a single DrmDisk representing all the system files. If, however, an alternate swap file location is specified, the vSwap file is represented, as a separate DrmDisk and Storage DRS will be disabled on this swap DrmDisk. For example: Virtual machine with 2 VMDKs and no alternate swap file location specified, Storage DRS creates 3 DrmDisks as follows: Above technical details show that swap file is considered for load balancing when a virtual machine is in powered on state, and when swap file is located in the same directory as other disks of the virtual machine. A separate DrmDisk for each VM Disk fileA DrmDisk for system files (VMX, Swap, logs etc) Which virtual machine files does Storage DRS consider in both Initial Placement and Subsequent Rebalance Calculations? Storage DRS has a concept of system-files even during initial-placement. system-files includes virtual machine configuration file such as VMX, snapshot files. Initial placement and rebalance, both take all the virtual machine system files/snapshot files into consideration. How are simultaneous initial placement requests are handled? We do not support real simultaneous initial placement requests. Recommenddatastores() API accepts one virtual machine as the input parameter. When calling the API for placement, you cannot specify datastore in the Input spec. Multiple virtual machine provisioning can behave differently less deterministically because of other Storage DRS calculation factors (I/O load, space load, growth rate of the disk (in case of thin provisioned type disk)), also because of particular provisioning workflow, exact timing when Storage DRS recommendation is called and when datastore space is really consumed. Recall that datastore reported free capacity is one of the main factors for next Storage DRS recommendations. How SDRS treats soft constraints with some of examples? Soft-constraints or soft-rules can be dropped by SDRS when it is absolutely required. In case of initial placement, SDRS will try to find an ideal datastore, which can satisfy all the soft constraints without compromising on the load balancing. If there is no ideal match available for initial placement, SDRS will start dropping some soft constraints. We have multiple categories of soft-rules. Examples: In this way, SDRS has various other soft constraints. If ideal placement is not possible due to hard rules (affinity-rule and anti-affinity rules, datastore maintenance, severe imbalance), SDRS will start to drop constraints in order of severity and re-run the algorithm to find a better match. SDRS will try to correct soft constraints/rule violation in the next SDRS run. If a user is using Site Recovery Manager and has placed disks on a datastore which is part of consistency group, SDRS would remove that disk to the datastore, which is part of the same consistency group.This use-case is related to storage-profiles. If a user wants to place VMDK on Storage-Profile1, SDRS append to place it on datastore which can satisfy the Storage-Profile1.We do have Space threshold (default value is 80%) constraint, SDRS tries its best to honor this constraint. Similarly, we have one soft constraint for correlated datastore where SDRS will avoid recommending any moves. How Storage DRS behaves with fully connected datastores? Storage DRS prefers datastores connected to all the hosts inside DRS cluster i.e. full connectivity before considering partially connected datastores. How Storage DRS gives preference to IO and Space metrics? When space is running low, it will try to balance space more than IO (vice-versa). Can I use Storage DRS just for space and disable IO metrics? Yes, user can leverage just datastore space management. While creating Storage DRS, using web client, we have an option to disable Storage DRS IO metric. With I/O thresholds turned off, is it expected that the decision is based only on free space i.e Should we always pick the datastore with most free space or do we account for other things? Yes, rebalance and initial placement decision would be based on free space, affinity /anti-affinity rules configured & growth rate of the VMDKs etc. It is not required to pick the datastore with most free space always. When selecting a datastore, Initial placement takes both DRS and Storage DRS threshold metrics into account. It will select the host with the least utilization and highest connectivity (datastore and DRS cluster connectivity) to place the virtual machine. If I start multiple virtual machine deployment (either cloneVM or createVM operation) from vRA, how does Storage DRS process each request? Storage DRS uses RecommendDatastores() API for initial placement request. This API processes one virtual machine at a time. For any given cluster, this API call will be processed sequentially; regardless it is for cloning a virtual machine, or creating a virtual machine, or other type of operation. Storage DRS is an intelligent engine, which prepare placement recommendations for initial placement and recommendations for continuous load balancing also (Based on space and I/O load). That means other software component (C# Client, Web Client, PowerCLI, vRealize Automation, vCloud Director, etc) are responsible for initial placement provisioning and Storage DRS gives them recommendations where is the best place to put a new storage objects (vmdk file or VM system files) at that moment. How thin provisioned type VMDKs are considered by Storage DRS? VMFS datastore accurately reports committed, uncommitted and unshared blocks. By default, NFS datastore is always thin provisioning, as we do not know how NFS server is allocating blocks. Thin-provisioned disks and thick provisioned disks use same calculated space and IO metrics. One aspect, which SDRS uses is, while load balancing, SDRS looks at growth rate of the VMDKs. How is independent disk type supported by Storage DRS? VMs with independent disks are supported when using Storage DRS, however, SDRS will only make recommendations for the independent disks initial placement, including relocation or clone placement. It should be noted that for an independent disk, Storage DRS load balancing is disabled by default. SDRS load balancer can make migration recommendations for other disks, but for independent disks, SDRS by default does not make migration recommendations - vpxd.log will show them as disabled. Does Storage DRS do datastore cluster defragmentation? If enough free space is available in the datastore cluster but not enough space is available per datastore, the datastore is considered as fragmented. In this case, Storage DRS will do defragmentation to free-up the space required for new virtual machine placement. Are space & IO threshold set per datastore or datastore cluster? Storage DRS thresholds are configured on Storage DRS cluster level but these threshold values are effective on each datastore inside the cluster. If there are multiple destination datastores nearing the threshold, how Storage DRS considers the right datastore? It uses space utilization ratio difference threshold to determine which datastore to consider as destination for virtual machine migration. This threshold is the advanced option for SDRS with default value set to 5%. For example, If there is datastore utilized upto 83%, SDRS will not move any virtual machine disks from this datastore to a 78% utilized datastore. Does Storage DRS consider datastores with dynamic growth rate? Storage DRS attempts to avoid migrating virtual machines with data-growth rates that may cause the destination datastore to exceed the space utilization threshold in the near future. Which latency does Storage DRS consider and how does Storage DRS utilize it? VMObservedLatency is what Storage DRS considers as of vSphere 5.1. VMObservedLatency: It measures the I/O round trip from the time when VMkernel receives the I/O by the virtual machine monitor (VMM) to the datastore, and all the way back to the VMM. The hosts collect the VMObservedLatency values of the virtual machines and send them periodically to vSphere Storage DRS, which by default stores the statistics as percentile values aggregated over a period of a day. These data points are sorted in ascending order; if the 90th percentile value exceeds the latency threshold, vSphere Storage DRS detects the datastore as overloaded. The 90th percentile value resembles the lowest edge of the busiest period. Overall, the datastore must experience a sustained load above the threshold for more than 1.6 hours a day. This can be either a workload that is sustained for more than 1.6 hours or one with sustained loads with periodic intervals. vSphere Storage DRS marks a datastore that incurs a threshold violation as a source for migrations. What is the exact purpose of Storage DRS latency and SIOC latency? The main goal of the SIOC latency threshold is to give fair access to the datastore, throttling virtual machine outstanding I/O to the datastore across multiple hosts to keep the measured latency below the threshold. The Storage DRS latency is a threshold to trigger virtual machine migrations. For example, if the VMObservedLatency of the virtual machines on a particular datastore is higher than the Storage DRS threshold, then Storage DRS will consider that datastore as a source to storage vMotion virtual machines from that datastore. Is Storage DRS I/O metric and SIOC the same thing? No, SIOC is not the same as Storage DRSI/O Metric. SIOC can be used without Storage DRS enabled. The goal of Storage DRS I/O load balancing is to fix long-term prolonged I/O imbalances, VMware vSphere Storage I/O Control addresses short-term burst and loads. For more information on SIOC (Storage IO control), see vSphere Enhanced Application Performance and Availability. Does Storage DRS recommend dependent / prerequisite? Yes, Storage DRS can recommend dependent moves. These moves are called as prerequisite moves. Storage DRS can recommend prerequisite moves when there is a situation where Storage DRS cannot place the VMDK unless some existing VMDK is moved to other suitable datastore. What is depth of recursion in Storage DRS? What is the default value? Depth of recursion is how many level of prerequisite moves Storage DRS can recommend. In addition, it is a set of Storage DRS recommendations to make the place for the virtual machine created or moved. Default value is 1 (2 steps i.e. 0,1). Maximum value is 5. Do I need to use Storage DRS I/O metrics for load balancing? It depends on your physical storage system (disk array). If you have disk array with modern storage architecture, then you may have all datastores (LUNs, volumes) on single physical disk pool. In that case, it is not required to load balance (do storage vMotion in case of I/O contention) between datastores because it will always end up in same physical spindles and will generate additional storage workload. The same is true for initial placement. If you have your datastores on different physical spindles then it can help. This is typically useful on storage systems using RAID groups, which is not very common now a day. What is the role of datastore correlation detector in Storage DRS? Different datastores exposed via a single storage array may share the same set of underlying physical disks and other resources. For instance, in case of EMC ClaRiiON array, one can create RAID groups using a set of disks, with a certain RAID level and carve out multiple LUNs from a single RAID group. These LUNs are essentially sharing the same set of underlying disks for RAID and it is not required to move a VMDK from one to another for IO load balancing. Storage DRS will try to find such correlation using Storage DRS injector. When two datastores are marked as performance-correlated, Storage DRS does not generate IO load balancing recommendations between those two datastores. However, Storage DRS can still generate recommendations to move virtual machines between two correlated datastores to address out of space situations or to correct rule violations. How Storage DRS does leverage SIOC injector? For more information, see the SDRS and Auto-Tiering solutions – The Injector. What are the some useful advanced options Storage DRS has to modify the default behavior? Note: The default behavior works best for most of the cases. Use advanced options only when it is required. Below are some of the Storage DRS advanced options user can configure on Storage DRS cluster. EnforceStorageProfiles To configure Storage DRS interop with SPBM, below options need to be set: 0 – disabled (default)1 – soft enforcement2 – hard enforcement PercentIdleMBInSpaceDemand The PercentIdleMBInSpaceDemand setting defines the percentage of IdleMB that is added to the allocated space of a VMDK during free space calculation of the datastore. IdleMB = Provisioned space minus allocated space. NB this setting is case-sensitive and prior to vSphere 6.7 is written "PercentIdleMBinSpaceDemand". The default value is set to 25%. This value can range from 0 to 100. For more information, see Avoiding VMDK level over-commitment while using Thin disks and Storage DRS. EnforceCorrelationForAffinity Use datastore correlation while enforcing/fixing anti-affinity rules: 0 – disabled (default)1 – soft enforcement2 – hard enforcement How Storage DRS works with VMs having RDM files? For more information, see the Storage DRS generates only one datastore as initial placement recommendation in clusters with physical RDM (2148523). How Storage DRS works with storage IOPS reservation? Let us understand with one example as follows. Consider a Storage Cluster having two shared datastores with the following configuration DSRemaining IO CapacityFree SpaceDS15003 GBDS210002 GB Create a one GB virtual machine with an IO Reservation of 700 on the POD. Storage DRS will recommend DS2 which can meet the virtual machines IOPS requirements. Prior to this feature Storage DRS would have recommended DS1 which had more free space. Note: Similar to above scenario, there are some more cases with respect to IO reservation SDRS handles. Additional Storage DRS white-papers/video resources? Storage DRS: Automated Management of Storage Devices In a Virtualized DatacenterUnderstanding VMware vSphere 5.1 Storage DRSVMware vSphere Storage DRS™ InteroperabilityWhat Is VMware vSphere Storage DRS? (vSOM)Creating a Datastore Cluster with Storage DRS (vSOM)
Storage DRS integration with storage profilesStorage DRS generates only one datastore as initial placement recommendation in clusters with physical RDM