Symptoms

Missing base.uzip can also cause issues starting other services like webui services and httpd services, which will affect GUI access on affected node. When LWSM service has issues it will give following errors like below when we use command like following :-/usr/likewise/bin/lwsm listError: ERROR_FILE_NOT_FOUND (2) While checking JWT status: /usr/likewise/bin/lwsm status jwt LW_ERROR_STRING_CONV_FAILEDFailed to convert string format (wide/ansi) Or when we try to start service by using command below /usr/likewise/sbin/lwsmd --syslog --start-as-daemonError: ERROR_GEN_FAILURE (31) Isilon-1# isi auth ads list Unable to connect to authentication daemon: ERROR_FILE_NOT_FOUND Isilon-1# /usr/likewise/bin/lwio list zsh: no such file or directory: /usr/likewise/bin/lwio When we list /base.uzip file we will see on one of the node /base file is missing Isilon-1# isi_for_array -s 'ls -lh /base.uzip'Isilon-1: ls: /base.uzip: No such file or directoryIsilon-2: -rwxr-xr-x 1 root wheel 27M Aug 2 2018 /base.uzipIsilon-3: -rwxr-xr-x 1 root wheel 27M Aug 2 2018 /base.uzip

Cause

This could occur due to root partition gets full which can show similar output like below if we run df -h command :-Isilon-1# df -hFilesystem Size Used Avail Capacity Mounted on/dev/mirror/root0 1.9G 1.8G -126M 107% /devfs 1.0K 1.0K 0B 100% /dev/dev/mirror/var0 956M 147M 732M 17% /var/dev/mirror/var-crash 2.8G 1.7M 2.6G 0% /var/crash/dev/mirror/keystore 60M 4.0K 55M 0% /keystore/dev/md0 60M 1.2M 54M 2% /tmp/ufp/dev/md1 3.5M 422K 2.8M 13% /tmp/ipmiOneFS 130T 3.7T 119T 3% /ifs So please make sure there is no other data being written to the root partition, only /ifs should be utilized for customer data.

Resolution

Please check which node has /base.uzip file missing :Isilon-1# isi_for_array -s 'ls -lh /base.uzip'Isilon-1: ls: /base.uzip: No such file or directoryIsilon-2: -rwxr-xr-x 1 root wheel 27M Aug 2 2018 /base.uzipIsilon-3: -rwxr-xr-x 1 root wheel 27M Aug 2 2018 /base.uzipPlease copy the base.uzip file from good node which has ubase.uzip file to bad node, then reboot the node.Example: 1. Copy base.uzip from existing node to a global dir like Isilon_Support: isi_for_array -n2 'cp /base.uzip /ifs/data/Isilon_Support/' 2. copy base.uzip from backup dir to an affected node: isi_for_array -n1 'cp /ifs/data/Isilon_Support/base.uzip /' 3. Verify md5 hash value matches other nodes and check permissions: isi_for_array -s 'ls -lh /base.uzip' isi_for_array -s 'md5 /base.uzip' NOTE: if permissions do not match and need to be adjusted, use command below on affected node: chmod 755 /base.uzip verify permissions match: isi_for_array -s 'ls -lh /base.uzip'

Support Cases