Symptoms

During filesystem shutdown, any errors from the cloud provider will cause the filesystem to abort inflight requests. This is by design since retrying a single cloud request can take up to 5 minutes and there is a time limit for the filesystem to shutdown. Following an example of an error in ddfs.inf0 and the subsequent abort of the request: search for "Request interrupted before retrying". The "Active" cloud unit will be marked unavailable and the filesystem will exit with a unclean shutdown.04/12 12:06:29.060 (tid 0x1618b430): ERROR: CAL cl_async_channel_postprocess_enqueue:672 - Request op:6 uri:https://ddvestorageaccount.blob.core.windows.net/ddvecontainer//d1/ca70f3a8/0000000000137383/0000000000000000 timeout reached after nretries:0 04/12 12:06:29.060 (tid 0x7f8b05316600): ERROR: CAL cal_libcloud_iodone_cb:2310 - CAL I/O operation:CAL_READ_OP for object:/d1/ca70f3a8/0000000000137383/0000000000000000 returned error:Request interrupted before retrying 04/12 12:06:29.061 (tid 0x7f8b05316600): INFO: CAL cal_process_io_error:1293 - Marking cloud unit:Active as UNAVAILABLE

Cause

During filesystem shutdown, any errors from the cloud provider will cause the filesystem to exit in an unclean state.

Resolution

Re-try system upgrade after confirming file system is enabled and running again. # filesys statusThe filesystem is enabled and running.Note: To reduce the chances of hitting this problem, customers should stop all the activities (ingest, restore, replication, GC) on DDVE before the system upgrade is performed. Contact Dell EMC Data Domain support if the system continues to report upgrade failures with file system shutdown error.

Support Cases