Symptoms

DELL ECS may be used as a cloud provider with DD Cloud Tier option (Long-Term Retention, LTR). For details about DD LTR see Dell KB article 23144, Data Domain Restorer (DDR) and Long Term Retention (LTR) to the Cloud: Frequently Asked Questions (FAQ/FAQs)ECS and public cloud providers (such as AWS, Azure, and GCP) work exactly in the same way except for minor implementation details. Cloud Tier cleans work differently in ECS due to ECS cloud being local, owned by the customer, and hence: Latency and performance are faster than for public cloud providersUsers do not incur additional expense when consuming bandwidth or running operations on ECSECS is developed by DELL and has extensions to the S3 protocol standard. This allows it to run Cloud Tier cleans more efficiently than for public cloud providers ECS has an S3 extension implemented called "Copy Range." This may be seen in other places such as MULTI COPY.This is a set of extended headers, not present in the S3 standard, and hence only supported with ECS. It allows for the ECS to create a new object by pulling byte ranges from existing objects. All the work is done through ECS and not the client.This nonstandard operation only supported by ECS is used by the DD only during the Cloud Tier clean "copy" phase. It allows the DD to run Garbage Collection (GC) on ECS-based cloud units in the same way as it runs on Active tier (which is more effective in reclaiming unused space).It is not used when writing new data to cloud during data-movement (it uses PUT), nor when recalling or reading data from cloud unit (when it uses GET).

Cause

Copy Range/MULTI COPY, being an extension to the S3 protocol, it is not a new method.Copy Range/MULTI COPY is a set of new headers that is added to PUT calls. ECS knows how to interpret this, but not other cloud implementations.In the DDs, there are places where MULTI COPY is reported, instead of PUT. This is to separate the regular PUT (which are used to write data to the cloud during data-movement) from those being generated during CT GC (which would be the so-called MULTI COPY, but they are PUT under the hood).As an example, we may see that the following values were reported for a customer, 7 days apart: GENERATED_ON=Tue Jul 18 06:09:51 CEST 2023 || GENERATED_ON=Tue Jul 25 06:09:52 CEST 2023 || Cloud op stats for bucket:xxxxxxxxxxxxxxxx-yyyyyyyyyyyyyyyy-d0 || Cloud op stats for bucket:xxxxxxxxxxxxxxxx-yyyyyyyyyyyyyyyy-d0 Number of PUT object requests : 1881 || Number of PUT object requests : 559289 + 557,408 PUT Number of GET object requests : 51 || Number of GET object requests : 51 + 0 GET Number of DELETE object requests : 1026 || Number of DELETE object requests : 25149154 + 25,148,128 DELETE Number of MULTI COPY object requests : 0 || Number of MULTI COPY object requests : 18453552 + 18,453,552 MULTI COPY Number of LIST object requests : 105847 || Number of LIST object requests : 105847 + 0 LIST The DD would report amounts of operations completed for PUT, GET, DELETE, MULTI COPY, and LIST.MULTI COPY is not a valid S3 method, not even for ECS. MULTI COPY is really PUT with some special headers added in by the DD when running the operation with ECS.When checking on the ECS side, customers and ECS Support do not see "MULTI COPY" operations at all. They only see, if applicable, PUT, GET, DELETE, and LIST. The DD reports PUT separately when doing data-movement to the cloud. The DD reports MULTI COPY (which are PUT generated during CT GC, to create new objects from byte ranges read from existing objects in cloud).For this reason, customers and ECS Support see PUT being run from the DD during DD CT GC. This could be misunderstood as new data (data-movement from the DD) but it could be due to ongoing DD CT GC (which does MULTI COPY, that is, PUT with special parameters).Each MULTI COPY operation recorded in the DD stats results in one and only one PUT operation recorded on the ECS side.During DD CT GC, the process enters the phase of "background deletions." This is when the DD CT GC has finalized all the copy forwards using MULTI COPY. It now requests ECS to delete all source objects we read byte ranges from.In the stats above, the amount of DELETE during the week outnumbers the combined number of new objects (PUT, for new data, and MULTI COPY, for copy forwards during CT GC).This results in a net gain of space in the cloud tierIn this example, 557,408 + 18,453,552 - 25,148,128 = -6,137,168, that is more than 20 TiB post comp with the usual object size for ECS-based cloud units.

Resolution

This is expected behavior.Some PUT operations performed by the DD to ECS are a special type, called "MULTI COPY" on the DD side, but they appear as regular PUT on the ECS side.MULTI COPY is only generated from the DD during CT GC. They create a single new object in the cloud by reading byte ranges from existing objects in ECS. All this happens within ECS itself.This has the following consequences: During DD CT GC, ECS at some point sees lots of PUT operations (which are the DD MULTI COPY). This is perfectly expected, and causes a transient increase in storage usage on ECSAfter DD CT GC has completed copy forwards, it issues DELETE to ECS, and begins to free up space. ECS freeing up space on its own storage may be delayed due to how ECS GC worksIt may be difficult to tell the PUT for new data apart from PUT for DD CT GC. Check with ECS Support if assistance is needed with this.MULTI COPY is an extension of the S3 standard and only happens with ECS-based cloud unitsMULTI COPY asking ECS to read from existing objects to create a new object, it may result in failures or corruption alerts being posted on the DD. The expectation is that all source objects exist in ECS. If not, DD CT GC stops and reports alerts for DD CT GC failure, integrity check failure, or corruption. In this case, contact DELL Data Domain Support.Even if MULTI COPY extensions are implemented in ECS, they are not supported for IAM users. Only (regular) object users as created from ECS can use the MULTI COPY extensions. If the DD cloud profile is configured with an IAM user credentials, or a regular object user is changed to an IAM user later one, everything would work fine on the DD with ECS except for clean. This would eventually fail with the following alert: Jul 19 11:24:59 dd.example.com ddfs[10217]: INFO: Event posted: m0-4577 (210011e1:553652705): EVT-CLOUD-00001: Unable to access provider for cloud unit ECS_CLOUD_UNIT.EVT-OBJ::CloudUnit=ECS_CLOUD_UNIT EVT-INFO::Cause=IAM user is not supported for Copy Range API

Support Cases