Loading...
Loading...
VPLEX storage-volume is in a 'critical-failure' state, and the underlying volume is healthy on the Array side. Sample output from cli command " ll /clusters/cluster-1/storage-elements/storage-volumes/<storage-volume name> " VPlexcli:/> ll /clusters/cluster-1/storage-elements/storage-volumes/Symm1234_0000 /clusters/cluster-1/storage-elements/storage-volumes/ Symm1234_0000 : Name Value ----------------------------- ------------------------------------------------ application-consistent false block-count 65798400 block-size 4K capacity 251G description - free-chunks [] health-indications [hardware dead] health-state critical-failure <<-- io-status dead itls 0x5000144200000010/0x5000097000000000/80, 0x5000144200000010/0x5000097000000000/80, 0x5000144200000011/0x5000097000000000/80, 0x5000144200000011/0x5000097000000000/80, 0x5000144200000110/0x5000097000000000/80, 0x5000144200000110/0x5000097000000000/80, 0x5000144200000111/0x5000097000000000/80, 0x5000144200000111/0x5000097000000000/80, 0x5000144200000110/0x5000097000000000/80, 0x5000144200000110/0x5000097000000000/80, ... (16 total) largest-free-chunk 0B locality - operational-status error provision-type legacy storage-array-name EMC-SYMMETRIX-XXXXXXXXX storage-volumetype normal system-id VPD83T3:60000097000000000000000000000000 thin-capable false thin-rebuild true total-free-space 0B underlying-storage-block-size 512 use unusable used-by [extent_ Symm1234_0000_1 ] vendor-specific-name EMC Storage Volume in critical-failure state VPlexcli:/> storage-volume summary SUMMARY (cluster-1) StorageVolume Name IO Status Operational Status Health State ---------------------------------------- --------- ------------------ ---------------- Symm1234_0000 dead error critical-failure <<-- Storage Array in ok state VPlexcli:/> ll **/storage-arrays /clusters/cluster-1/storage-elements/storage-arrays: Name Connectivity Auto Ports Logical --------------------------- Status Switch ------------------- Unit --------------------------- ------------ ------ ------------------- Count --------------------------- ------------ ------ ------------------- ------- EMC-SYMMETRIX-XXXXXXXXX ok true 0x5000097000000000, X 0x5000097000000000, 0x5000097000000000, 0x5000097000000000, A sample excerpt from the Dial Home that highlights the logical unit that died: <EventData><![CDATA[ disk VPD83T3:60000970000000000000000000000000: disk died, no mirror exists If this storage-volume is not part of a distributed device, then the host loses access to this Virtual Volume causing a Data Unavailability (DU) situation. VPLEX firmware logs show streaming 'scsi/27' events with SCSI Sense Code key 0x7 asc 0x27 ascq 0x7 (07/27/07) which translates to SPACE ALLOCATION FAILED WRITE PROTECT. VPLEX also records a scsi/169 event in firmware.log which means " The reported thin-enabled storage-volume has run out of space. " 128.221.252.37/cpu0/log:5988:W/"00601636f400114833-1":113877:<2>2016/07/27 22:07:22.05: scsi/169 Space allocation failed on scsi vol VPD83T3:6000009700000000000000000000000 Sample scsi/27 instances from firmware.log 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298782:<6>2016/11/11 06:50:57.94: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298783:<6>2016/11/11 06:50:58.00: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298784:<6>2016/11/11 06:50:58.16: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298785:<6>2016/11/11 06:50:58.30: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0 After 21 retries are performed, the storage-volume is marked dead due to continuous write failures. 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298797:<4>2016/11/11 06:51:01.42: amf/45 disk VPD83T3:60000097000000000000000000000000: write failure: marking this in-use disk dead 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298798:<2>2016/11/11 06:51:01.42: amf/97 disk VPD83T3:60000097000000000000000000000000: disk died, no mirror exists
Space allocation has failed on the thin-enabled storage-volumes. There are no more available storage blocks on the backend array to map to the address to which the VPLEX has issued a write command.
The storage-array that is sending the scsi check condition, 7/27/07, to VPLEX must be investigated by the respective array vendor. This issue is triggered by the array not being able to service the write I/O request due to an "SPACE ALLOCATION FAILED WRITE PROTECT" issue on the storage-array. The following cli grep command may be run on the VPLEX Management-Server to get a list of logical-units impacted by the 7/27/07 check condition: grep "scsi/27" firmware.log* | grep 'key 0x#' | grep 'asc 0x##' | grep 'ascq 0x#' | awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nr Example : service@ManagementServer:/var/log/VPlex/cli> grep "scsi/27" firmware.log* | grep 'key 0x7' | grep 'asc 0x27' | grep 'ascq 0x7' | awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nr 193474 scsi/27 VPD83T3:60000097000000000000000000000000 key 0x7 asc 0x27 ascq 0x7 42254 scsi/27 VPD83T3:60000097000000000000000000000001 key 0x7 asc 0x27 ascq 0x7 32408 scsi/27 VPD83T3:60000097000000000000000000000002 key 0x7 asc 0x27 ascq 0x7 26581 scsi/27 VPD83T3:60000097000000000000000000000003 key 0x7 asc 0x27 ascq 0x7 21643 scsi/27 VPD83T3:60000097000000000000000000000004 key 0x7 asc 0x27 ascq 0x7 19409 scsi/27 VPD83T3:60000097000000000000000000000005 key 0x7 asc 0x27 ascq 0x7 17651 scsi/27 VPD83T3:60000097000000000000000000000006 key 0x7 asc 0x27 ascq 0x7 If this is a non-Dell array, have the end user engage the respective array vendor. In order to resolve the issue that exists on the storage-array.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.