Symptoms

Clip deletion fails with Error Number: -10204 System Error: 18 Error: FP_OPERATION_NOT_ALLOWED blobsvc.log shows ERROR_OBJECT_UNDER_RETENTION.Example how to search across all nodes when logged into one node over ssh:svc_log -f "ERROR_OBJECT_UNDER_RETENTION" -sr blobsvc -n all -sf -snExample output: admin@ecsnode1:~> svc_log -f "ERROR_OBJECT_UNDER_RETENTION" -sr blobsvc -n all -sf -sn svc_log v1.0.28 (svc_tools v2.10.0) Started 2023-02-27 14:32:28 Running on nodes: <All nodes> Time range: 2022-06-29 14:32:28 - 2022-06-30 14:32:28 Filter string(s): 'ERROR_OBJECT_UNDER_RETENTION' Show filename(s): True Show nodename(s): True Search reclaim logs (if any): False 169.254.1.2 2022-06-30T14:05:04,806 [NIOCOMMUNICATOR-0-1-197] ERROR ObjectDirectoryOperation.java (line 2670) update object failed, request objectId: 4da5a4c7b4b81df5c1bf2600c6f34bb8d381d21bc065e9888c80b4f447ea76c9, update record: UpdateMetadataRecord: [System metadata: [mtime:1656597904788, size:15564, keypoolname:BUCKET, namespace:NAMESPACE, fs-ctime:1656597904788, createtime:1656597904788, keypoolid:NAMESPACE.BUCKET, user-metadata-size:0, keypool-hash-id:a34b72fddc6824dcbfbc1f4e086585befbd991655dcb07e9975b6218be8f3151, retention:CgcImJio/sAG]; hasMoreSysMd:false]; Head System metadata: [totalsize:10807, reason:, OT:Reflection, incomingip:10.1.2.3, retention.period:223603199, cas.mtime:1656597904788, creation.date:2013.10.01 05:44:44 GMT, creation.profile:BUCKET, clip.naming.scheme:MG, principal:BUCKET, RC:user, RD:false, deletedsize:26371, modification.date:2013.10.01 05:44:44 GMT, numtags:1, sdk.version:3.2.661, name:PAM_01, numfiles:1, WT:1656597904000, modification.profile:BUCKET, RP:false]; HasMoreHeadSysMd:false]; User metadata: []; hasMoreUserMd:false]; Data Indices: [; hasMoreDataIndices:false]; ], uid BUCKET, groups [BUCKET], exception com.emc.storageos.data.object.exception.ObjectControllerException: Operation 'DELETE' is not allowed for the object: 4BM4JCUCKHMT0e4KKLM0G2RJLB7G4185P8O41S0F7RDD8IN263100, 'Retention period' is set 169.254.1.2 2022-06-30T14:05:04,807 [1794721639,unknown,host.url.com/22/DELETE_CLIP,host.url.com/22/DELETE_CLIP,1-0a9c0869:17f1b64c025:854:a888b] ERROR ObjectControllerExceptionHelper.java (line 388) Method setHeadSystemMetadataForObject ObjectInfo: [id=4da5a4c7b4b81df5c1bf2600c6f34bb8d381d21bc065e9888c80b4f447ea76c9, versionId=-3, CoS=urn:storageos:VirtualArray:ed9791b3-33ad-4a3e-8810-953f358ecf83.urn:storageos:ReplicationGroupInfo:acba7e3c-c417-40c0-96cd-0b29ddcd8487:global, FAMode=disabled, PreviousFAMode=, FAStartVersion=0, FAEndVersion=0, ObjectOwnerZone=null, objectOwnerRecordListnull, keyName=4BM4JCUCKHMT0e4KKLM0G2RJLB7G4185P8O41S0F7RDD8IN263100] hit error, failed due to exception com.emc.storageos.data.object.exception.ObjectControllerException: directory server 10.1.2.123 returns error ERROR_OBJECT_UNDER_RETENTION, objectId 4da5a4c7b4b81df5c1bf2600c6f34bb8d381d21bc065e9888c80b4f447ea76c9, requestId 0a9c0869:17f1b64c025:854:a888d 169.254.1.2 2022-06-30T14:05:04,807 [1794721639,unknown,host.url.com/22/DELETE_CLIP,host.url.com/22/DELETE_CLIP,1] ERROR UpdateMetadata.java (line 36) Failed to set metadata: error is UNDER_RETENTION com.emc.vipr.cashead.api.ObjectUnderRetentionException: Unable to set object metadata for: 4BM4JCUCKHMT0e4KKLM0G2RJLB7G4185P8O41S0F7RDD8IN263100 admin@ecsnode1:~> But the retention has passed: CASScript>crv <meta name="retention.period" value="223603199"/> --> 7.09041092719 years <meta name="sdk.version" value="3.2.661"/> <meta name="creation.date" value="2013.10.01 05:44:44 GMT"/> <meta name="modification.date" value="2013.10.01 05:44:44 GMT"/> Clip Properties: Name: PAM_01 Creation Date: 2013.10.01 05:44:44 GMT Size: 26751 Number of Tags: 1 Number of Blobs: 1 Retention Class: Retention Seconds: 223603199 Modified: False EBR Enabled : False Retention Hold: False

Cause

The migrated clip cannot be deleted once retention expires. This is due to a software issue in an earlier ECS version that has been used for transformation migration from Centera.

Resolution

Open a Case with Dell Support and see knowledge article 000210103. ECS: CAS Clip deletion fails with error -10204 (FP_OPERATION_NOT_ALLOWED). 

Support Cases