Loading...
Loading...
Introduction When a drive failure occurs on a cluster, the cluster must have enough free space available to smartfail the drive and reprotect its data. The Virtual Hot Spare (VHS) feature allows you to set aside space for this purpose. Dell Technologies recommends that customers always keep VHS enabled, except in specific circumstances, and only disable it under the direction of Dell Technical Support. If VHS is disabled on your cluster, before you enable it, verify that it has enough space available to safely enable VHS. For more information about VHS, see the OneFS Web Administration Guide or OneFS CLI Administration Guide for your version of OneFS. This document covers: Why keep VHS enabled? Why VHS might be disabled? How OneFS calculates VHS space. Procedures Phase 1: Decide how space to reserve for VHS Phase 2: Verify that the cluster has enough free space to safely enable VHS Phase 3: Enable VHS Why keep VHS enabled? You should keep VHS enabled because: VHS helps the cluster maintain enough space to reprotect data if there are drive failures. VHS provides a buffer of space to help prevent the cluster from filling completely. If the cluster becomes full, serious problems, including the inability to write data, delete data, or run commands may occur. In severe cases, the cluster might stop working completely. CAUTION! Enabling VHS does not guarantee that the cluster does not fill up completely, nor properly reprotect data. You must monitor the cluster and pay attention to warning messages stating the cluster is nearing capacity (95% or more full). Replace all failed devices promptly, once the smartfail process competes. Although users and clients cannot write data to the VHS reserved space, the cluster can use that space for system maintenance jobs. FlexProtect and Snapshot may write to the VHS if there is no other space available on the cluster. If these warnings are ignored and the cluster is approaching full capacity, the cluster could still become full. It is critical to follow cluster best practices to ensure that the cluster does not become too full. See Best Practices Guide for Maintaining Enough Free Space on Isilon Clusters and Pools for more information. Why VHS might be disabled? The VHS feature was added in OneFS 6.0, and it is enabled by default on initial installations of OneFS version 6.0 and later versions. However, if the VHS is off and disabled before an upgrade, it remains disabled after the upgrade. How OneFS calculates the VHS space. When you configure the size of the VHS, you must choose to configure at least one virtual drive worth of space. You can configure up to four virtual drives. VHS uses the size of the largest hard disk drive in the pool as the basis for calculating the VHS size. VHS is calculated using the base-2 size of the drive (not the base-10 size). For example, the amount used in the calculation for a 1 TB drive is not 1 TB, but 931 GB. The amount of space reserved per drive is approximately 2.2 times the size of the largest drive in the pool. For example, if the largest drive in the pool is 1 TB (931 GB in base 2), then the space reserved per drive for VHS in that pool is approximately 2 TB. The reason the reserved amount is greater than the space of a single drive is because extra space is used to reprotect the data if the drive fails. The drive's absence may cause stripes to be written with a smaller width, which reduces storage efficiency and increases the amount of space required by VHS to reprotect a single drive's data. In addition, some extra space is required for system services during the reprotection. To calculate the VHS reserve per pool: Multiply the number of drives configured for VHS by the size of the largest Hard Disk Drive (HDD) in the pool, then multiply that by 2.2. (Number of drives configured for VHS) x (Size of largest HDD in pool) x 2.2 If you do not know the size of the largest Hard Drive in the pool, contact Technical Support for assistance. To calculate the total VHS space reserved per cluster , add the total VHS space reserved for all the pools in the cluster. The space reserved for each pool might be different depending on the size of the largest hard disk drive in that pool. Once VHS is configured, you can run the isi status | grep "VHS Size" command from the command-line to display the amount of space reserved for VHS.
To enable VHS that was disabled, or if VHS is enabled and you want to increase the amount of space reserved by VHS, follow these instructions. Before you enable or enlarge VHS, you must first decide on the amount of disk space you want to reserve. Verify that each node pool or disk pool has enough space to accommodate the reserved space. Phase 1: Decide on the amount of disk space to reserve for VHS. Phase 2: Verify that the cluster has enough free space to safely enable VHS Phase 3: Enable VHS
Procedures Phase 1 : Decide on the amount of disk space to reserve for VHS. The recommended minimum size for VHS is 10% of the cluster's storage capacity. However, when deciding the total amount of space to configure for VHS, there is no specific formula that applies to all workflows. You should consider the following: How many drives are in each node pool or disk pool? How quickly can you replace failed drives? In clusters with large pools, it is recommended that you configure multiple drives' worth of VHS space. This protects against the increased risk of running out of space due to simultaneous drive failures. In clusters where immediate drive replacement is not feasible, recommend that multiple drives' worth of VHS space be configured. This might be especially significant if the cluster is in a remote location. If a drive cannot be replaced it in the time, configure more drives' worth of VHS to protect against additional drives failures. Phase 2 : Verify that the cluster has enough free space to safely enable VHS. Do one of the following: From the OneFS web administration interface: Go to File System > Storage Pools > Summary . In the Local Storage Usage section, look at the H DD % Used for each node pool. From the OneFS command-line interface: Open an SSH connection on any node in the cluster and log in using the root account. Then run the following command: isi status -p -q The output contains a section that looks similar to the following. The names of the node pools are listed in the Name column. The HDD Storage Used/Size column shows the percentage of hard drive space in use in each node pool. Throughput (bps) HDD Storage SSD Storage Name Health| In Out Total| Used / Size |Used / Size -------------------+-----+-----+-----+-----+-----------------+----------------- x200_36tb_6gb | OK | 0| 408K| 408K| 275G/ 92T(< 1%)| (No SSDs) s200_7.2tb_24gb | OK | 342| 626| 968| 5.8T/ 19T( 31%)| (No SSDs) x200_6tb_6gb | OK | 604K| 16| 604K| 5.1T/ 21T( 25%)| (No SSDs) s200_6.9tb_200gb-ss|U----| 0| 0| 0| 3.2T/ 12T( 27%)| 6.3G/ 367G( 2%) d_48gb | | | | | | Check the HDD% Used column (web interface) or HDD Storage Used/Size column (command-line interface) for each node pool, and proceed as follows: For node pools containing nodes with…. Do this…. Up to 12 drives If EVERY node pool is less than 80% full, there is enough space to safely enable VHS for one hard disk drive worth of space. If ANY node pool is 80% or more full, or if you want to configure more than one drive worth of space for VHS, contact Technical Support. 18 to 59 drives If EVERY node pool is less than 90% full, you have enough space to safely enable VHS for one hard disk drive's worth of space. If ANY node pool is 90% or more full, or if you want to configure more than one drive's worth of space for VHS, contact Technical Support for assistance. Phase 3 : Enable VHS. In the OneFS Web administration interface, go to File System > Storage Pools > SmartPools Settings . Under Virtual hot spare (VHS) , select BOTH these check boxes: Ignore reserved space when calculating available free space. This setting subtracts the space reserved for virtual hot spare when calculating available free space. This shows the amount of true free space left on the cluster when you view the cluster status in the web interface, or when you run commands such as isi status on the command-line interface. NOTE If this setting is enabled and Deny data writes to reserved disk space is disabled, it is possible for the file system utilization to be reported at more than 100% Deny data writes to reserved disk space. This setting prevents client, or user write operations from using the VHS reserved disk space. CAUTION! Even with Deny data writes to reserved disk space selected, the cluster can still use the VHS reserved space for maintenance jobs, such as FlexProtect and Snapshot jobs, if there is no other space available on the cluster. Under VHS Space Reserved , use the drop-down list to set the At least X virtual drive(s) to the number of drives that you want to reserve. You must select at least 1 (default). The maximum is 4 . Under VHS space to reserve , use the drop-down list to set the At least X % of total storage to 0 (default). CAUTION! It is recommended not to set this value to anything other than zero. Contact Dell Technical Support if you have questions. Click Save Changes . After the change is committed, you should immediately see the space reserved for VHS reflected in the Web User-Interface page and in the command-line interface wherever the VHS space, cluster capacity, node pool capacity, and percentage of space used are displayed. Go to File System > Storage Pools > Summary . In the Local Storage Usage section, look at the HDD % Used for each node pool. Verify that the percentage for each node pool is less than 95%. CAUTION! If any node pools show an HDD % Used rate of 95% or more, immediately create more space in the pool by removing or redistributing data, or by adding nodes. If this is not possible, contact Dell Technical Support.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.