Symptoms

As part of OneFS upgrades, a necessary and mandatory migration of our prior version of inode design to a new version must happen; there is no way to defer or prevent this migration. This will touch every single file on your OneFS cluster, including user space (/ifs) and all system type files.This procedure begins after the OneFS cluster upgrade is "committed" in the upgrade process.The large scale of this job in some scenarios and deployments may cause transient performance issues. After committing a OneFS upgrade a system job of type upgrade that upgrades the file system after a software version upgrade, will be automatically started and must be allowed to run until completion. If there are inode structure changes between the OneFS releases of the upgrade, the Upgrade job will perform inode structure changes on every file. When this occurs the running Upgrade job may cause temporary performance issues due to the increased IOPS required to perform inode structure changes. This may be mitigated by changing the policy for the Upgrade job to Low from the current default value of Medium.Which Systems May Be Affected? Product (and version)All OneFS / PowerScale clusters for Isilon that upgrade to a version with inode changes.Running this Core Software (Operating System (OS) or Operating Environment (OE)) OneFS releases with inode structure changes: • OneFS 7 • OneFS 8 • OneFS 9.2 When this condition is trueIf the issue will occur, it will occur in the hours or days after the general upgrade is complete, and while this finale migration job runs.And when this condition is true The likelihood of this happening is not certain and depends on many transient factors like overall cluster health, type of hardware, general impact to the system from operator and end-user policies and workflows, the number of file objects under /ifs and other factors.

Cause

During this, literally all files on the storage platform will be migrated rapidly to the new inode version. The general lab-observed pace of updates across hardware platforms, and using the default job policy, can go as fast as tens of millions of overall file objects per hour. Please note that "files" in this context does not refer just to the "user uploaded" files as you will see for example under your Quotas and reporting. This includes all system-level files and objects as well. Once the migration is underway, as seen as a job in the CLI named "Upgrade" that runs under the default Medium policy, there is a possibility that an Isilon cluster may see temporary and transient performance issues while this upgrade occurs. From the limited number of reports of this we have observed, it appears that the Upgrade inode migration can take on a scale of hours to days based on a Medium policy.

Resolution

This may be mitigated by changing the policy for the Upgrade job from the current default value of Medium to Low.If performance impacts are seen, to mitigate those impacts change the policy of the Upgrade job from Medium to Low. Changing the policy to Low reduces the overall impact that the job has while it runs, but does increase how long the job takes to complete. To change the policy of a currently running Upgrade job to Low, run the following command: isi job jobs modify Upgrade --policy=Low To change the policy for future Upgrade jobs to Low, run the following command: isi job types modify Upgrade --policy=Low Note: This job is a key part of this particular sort of upgrade. While things like the priority of the job can be altered, the absolute requirement that this job be ran cannot be altered. If the 'job' were to somehow be canceled or fail, it resumes again. The inode changes must complete. Refer to PowerScale OneFS: Upgrade Planning and Process Guide for more information related to OneFS upgrade

Support Cases