Symptoms
Someone created a file "ls" in /var/lib/docker/devicemapper/metadata by mistake.SP rolling reboot with eVE errors in /EMC/C4Core/log/start_c4.log (/mnt/ssdroot/EMC/C4Core/log/start_c4.log if SP is in service mode):Mon Aug 7 21:51:41 UTC 2017 CF: iwd: fully started in 0 secondsMon Aug 7 21:54:05 UTC 2017 CF: HA: eVE: Error: job cf.eVE stoppedMon Aug 7 21:54:05 UTC 2017 CF: HA: eVE: stoppingMon Aug 7 21:54:06 UTC 2017 CF: HA: eVE: Error: failed to stop eVEMon Aug 7 21:54:11 UTC 2017 c4_shutdown.sh: called with "-f"
Cause
According to /EMC/CEM/log/eVE/eVE_startup.log on SP, docker is not running.Mon Aug 7 21:49:05 2017 /EMC/Platform/bin/eVE.pl: Cannot proceed. Docker service is not runningMon Aug 7 21:49:05 2017 /EMC/Platform/bin/eVE.pl: Cannot proceed. Docker service is not runningMon Aug 7 21:49:06 2017 /EMC/Platform/bin/eVE.pl: Cannot proceed. Docker service is not runningAccording to /var/log/messages, docker service failed because it could not load device file "ls".2017-08-07T21:41:09+00:00 self docker[18109]: time="2017-08-07T21:41:09.996387324Z" level=error msg="[graphdriver] prior storage driver \"devicemapper\" failed: devmapper: Failed to load device files:Error looking up device ls:Unknown device ls"2017-08-07T21:41:09+00:00 self docker[18109]: time="2017-08-07T21:41:09.996422502Z" level=fatal msg="Error starting daemon: error initializing graphdriver: devmapper: Failed to load device files:Error looking up device ls:Unknown device ls"2017-08-07T21:41:10+00:00 self systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Resolution
1. Remove the problematic file in /var/lib/docker/devicemapper/metadata.rm -f /var/lib/docker/devicemapper/metadata/ls2. Clear the boot rescue counter.svc_rescue_state -c3. Reboot the SP.svc_shutdown -rNote: SP re-image can also resolve this problem.
