Symptoms

VPLEX storage-volume is in a 'critical-failure' state and the underlying volume is healthy on the Array side. Sample output from cli command "ll /clusters/cluster-1/storage-elements/storage-volumes/" VPlexcli:/> ll /clusters/cluster-1/storage-elements/storage-volumes/Symm1234_0000 /clusters/cluster-1/storage-elements/storage-volumes/Symm1234_0000:Name Value----------------------------- ------------------------------------------------application-consistent falseblock-count 65798400block-size 4Kcapacity 251Gdescription -free-chunks []health-indications [hardware dead]health-state critical-failure io-status deaditls 0x5000144200000010/0x5000097000000000/80, 0x5000144200000010/0x5000097000000000/80, 0x5000144200000011/0x5000097000000000/80, 0x5000144200000011/0x5000097000000000/80, 0x5000144200000110/0x5000097000000000/80, 0x5000144200000110/0x5000097000000000/80, 0x5000144200000111/0x5000097000000000/80, 0x5000144200000111/0x5000097000000000/80, 0x5000144200000110/0x5000097000000000/80, 0x5000144200000110/0x5000097000000000/80, ... (16 total)largest-free-chunk 0Blocality -operational-status errorprovision-type legacystorage-array-name EMC-SYMMETRIX-XXXXXXXXXstorage-volumetype normalsystem-id VPD83T3:60000097000000000000000000000000thin-capable falsethin-rebuild truetotal-free-space 0Bunderlying-storage-block-size 512use unusableused-by [extent_Symm1234_0000_1]vendor-specific-name EMC Storage Volume in critical-failure state VPlexcli:/> storage-volume summarySUMMARY (cluster-1)StorageVolume Name IO Status Operational Status Health State---------------------------------------- --------- ------------------ ----------------Symm1234_0000 dead error critical-failure Storage Array in ok state VPlexcli:/> ll **/storage-arrays/clusters/cluster-1/storage-elements/storage-arrays:Name Connectivity Auto Ports Logical--------------------------- Status Switch ------------------- Unit--------------------------- ------------ ------ ------------------- Count--------------------------- ------------ ------ ------------------- -------EMC-SYMMETRIX-XXXXXXXXX ok true 0x5000097000000000, X 0x5000097000000000, 0x5000097000000000, 0x5000097000000000, Sample excerpt from the Dial Home that highlights the logical unit that died: disk VPD83T3:60000970000000000000000000000000: disk died, no mirror existsIf this storage-volume is not part of a distributed device then the host will lose access to this Virtual Volume causing a Data Unavailability (DU) situation.VPLEX firmware logs show streaming 'scsi/27' events with SCSI Sense Code key 0x7 asc 0x27 ascq 0x7 (07/27/07) which translates to SPACE ALLOCATION FAILED WRITE PROTECT.VPLEX also records a scsi/169 event in firmware.log which means "The reported thin-enabled storage-volume has run out of space." 128.221.252.37/cpu0/log:5988:W/"00601636f400114833-1":113877:2016/07/27 22:07:22.05: scsi/169 Space allocation failed on scsi vol VPD83T3:6000009700000000000000000000000 Sample scsi/27 instances from firmware.log 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298782:2016/11/11 06:50:57.94: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298783:2016/11/11 06:50:58.00: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298784:2016/11/11 06:50:58.16: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298785:2016/11/11 06:50:58.30: scsi/27 tgt VPD83T3:60000097000000000000000000000000 cmd 0x2a status 0x2 valid 0 resp 0x70 seg 0x0 bits 0x0 key 0x7 info 0x0 alen 10 csi 0x0 asc 0x27 ascq 0x7 fru 0x0 sks 0x0 After 21 retries are performed the storage-volume is marked dead due to continuous write failures. 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298797:2016/11/11 06:51:01.42: amf/45 disk VPD83T3:60000097000000000000000000000000: write failure: marking this in-use disk dead 128.221.253.37/cpu0/log:5988:W/"0060166fcbfe192045-1":3298798:2016/11/11 06:51:01.42: amf/97 disk VPD83T3:60000097000000000000000000000000: disk died, no mirror exists

Cause

Space allocation has failed on the thin-enabled storage-volume(s). There are no more available storage blocks on the backend array to map to the address to which the VPLEX has issued a write command.

Resolution

The storage-array that is sending the scsi check condition, 7/27/07, to VPLEX needs to be investigated by the respective array vendor. This issue is triggered by the array not being able to service the write I/O request due to an "SPACE ALLOCATION FAILED WRITE PROTECT" issue on the storage-array.The following cli grep command may be run on the VPLEX Management-Server to get a list of logical-units impacted by the 7/27/07 check condition:grep "scsi/27" firmware.log* | grep 'key 0x#' | grep 'asc 0x##' | grep 'ascq 0x#' | awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nrExample: service@ManagementServer:/var/log/VPlex/cli> grep "scsi/27" firmware.log* | grep 'key 0x7' | grep 'asc 0x27' | grep 'ascq 0x7' | awk '{print $3, $5, $18, $19, $26,$27,$28,$29}'| sort | uniq -c | sort -nr 193474 scsi/27 VPD83T3:60000097000000000000000000000000 key 0x7 asc 0x27 ascq 0x7 42254 scsi/27 VPD83T3:60000097000000000000000000000001 key 0x7 asc 0x27 ascq 0x7 32408 scsi/27 VPD83T3:60000097000000000000000000000002 key 0x7 asc 0x27 ascq 0x7 26581 scsi/27 VPD83T3:60000097000000000000000000000003 key 0x7 asc 0x27 ascq 0x7 21643 scsi/27 VPD83T3:60000097000000000000000000000004 key 0x7 asc 0x27 ascq 0x7 19409 scsi/27 VPD83T3:60000097000000000000000000000005 key 0x7 asc 0x27 ascq 0x7 17651 scsi/27 VPD83T3:60000097000000000000000000000006 key 0x7 asc 0x27 ascq 0x7 If this is a non-EMC array please have the end user engage the respective array vendor in order to resolve the issue that exists on the storage-array.

Support Cases