Loading...
Loading...
Upgrading a single node or DD HA system to DDOS 7.6 , 7.7 or later releases may take longer than expected. Specifically and most notable, once the first part of the upgrade has already completed, and the DD is already booting into the new code, the upgrade seems to not complete and if querying the status, it stays as shown below for a very long time (even hours): # system upgrade status Current Upgrade Status: DD OS upgrade In Progress Node 0: phase 4/4 (Finalize 51%) After a variable amount of time (which depends on the particular DD and can be up to a few hours) the upgrade process completes normally without error and bring the FS process up.
In DDOS 7.6 and 7.7 we have introduced an automatic format upgrade for all existing MTrees' "attribute btrees". Attribute btrees are important pieces of metadata information useful for speeding up backups and replication, which are stored alongside file data on a per-MTree basis, but in the past they have been the source of some issues, as they grew bigger and more complex than initially though, resulting in performance loss and even filesystem instability, hence the need for a new format to be converted to in DDOS 7.6 and later releases.
Unless in the presence of other problems, the upgrade process will eventually complete. Attribute btree upgrades will take a time which is proportional to the number of combined "pages" in the btrees to be upgraded, which will be a function of the total amount of files in the corresponding MTrees. In other words: Systems with few files (thousands) will see a negligible delay during the upgrade no matter what. Systems with many files (millions) may see upgrade completion delayed. Only those MTrees which have "attribute btrees" or which are still in an old format need be converted, so having a system with many files does not immediately imply the upgrade will take longer than expected. We have seen systems being "stuck" in the mentioned post-reboot upgrade phase for up to 4 hours due to a particular combination of file amounts and other issues. If making sure the upgrade completes on time is of critical importance to you, and your system has a large amount of files (more than 1 million), please contact your contracted support provider and provide this KB article as a reference, to have us determine if this system may be subject to a longer than usual upgrade, to provide any possible workarounds. A quick way to check the number of files in your system is by looking into a recent ASUP and searching for the "File Distribution" information. If already upgrading and the process seems to not progress past the "Node 0: phase 4/4 (Finalize 51%)" status, you may check progress is really being made, by checking the "dmck.info" log file for activiy, and more specifically, for messages like this: # log view debug/dmck.info 11/15 14:43:21.718734 [7ef9046f44b0] DM_ATTR_BTREE: __dmbt_copy_walk_intr_page_and_upgrade(): Converted 136194 keys of 1361948. Processed 68206 pages. 11/15 14:52:07.986345 [7ef9046f44b0] DM_ATTR_BTREE: __dmbt_copy_walk_intr_page_and_upgrade(): Converted 272388 keys of 1361948. Processed 134221 pages. 11/15 15:00:18.632498 [7ef9046f44b0] DM_ATTR_BTREE: __dmbt_copy_walk_intr_page_and_upgrade(): Converted 408582 keys of 1361948. Processed 199840 pages. 11/15 15:08:42.900000 [7ef9046f44b0] DM_ATTR_BTREE: __dmbt_copy_walk_intr_page_and_upgrade(): Converted 544777 keys of 1361948. Processed 265325 pages. ... Process should be complete once all the keys (1361948) have been converted. If progress is not seen in the logs, please contact your contracted support provider.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.