Symptoms

Unable to failover or failback Data Mover Receiving below error when attempting failover/failback: [nasadmin@CS0 ~]$ server_standby server_2 -restore mover server_2 : Error 28: server_2 : No space left on device

Cause

Although all partitions show that they have free space, the inodes have all been used in /nbsnas: [root@CS0 ~]# df -i /nbsnas Filesystem Inodes IUsed IFree IUse% Mounted on /dev/nde1 258048 258048 0 100% /nbsnas Under the /nbsnas/tasks, there are hundreds of thousands of files. They all show up similar to the below: -rw-r--r-- 1 nasadmin nasadmin 39 Nov 24 10:31 1586886_status -rw-r--r-- 1 nasadmin nasadmin 39 Nov 24 10:31 1586898_status -rwxr-xr-x 1 root nasadmin 0 Nov 25 20:31 1587045_pipe -rw-r--r-- 1 root nasadmin 39 Nov 25 20:31 1587045_status -rw-r--r-- 1 root nasadmin 39 Nov 27 20:25 1587072_status -rwxr-xr-x 1 root nasadmin 0 Nov 27 20:25 1587075_pipe

Resolution

Find which sub folder has used most inodes in "/nbsnas." In most of the cases, it is the "/nbsnas/tasks" directory which contains thousands of stale tasks. Follow the instructions in KB article 336312 to address it. Once the tasks folder has been cleaned up, failover/failback of the Data Mover should occur without issue.

Support Cases