General
System becomes unusable with read-only file-system
Symptom
Unable to run most of the commands in the router. Few XR CLIs work with very slow response time. We are not able to switch to run/bash prompt. Getting below logs in NCS540L system:-
RP/0/RP0/CPU0:#run lvs
Fri Mar 24 15:37:05.197 IST
posix_spawnp(): Input/output error (5)
% Error spawning user process 'run_cmd': Input/output error
RP/0/RP0/CPU0:#show in
start_backing_thread:bind: Read-only file system
install interfaces inventory
ctrace abort handler: unable to open trace file /var/log/ctrace/more/xr_ds_capi_msc/ctrace_1.trc (Read-only file system)ctrace abort handler: unable to open trace file /var/log/ctrace/more/xr_ds_capi_conn/ctrace_1.trc (Read-only file system)ctrace abort handler: unable to open trace file /var/log/ctrace/more/xr_ds_capi_error/ctrace_1.trc (Read-only file system)ctrace abort handler: unable to open trace file /var/log/ctrace/more/xr_ds_capi_info/ctrace_1.trc (Read-only file system)ctrace abort handler: unable to open trace file
Conditions
System is running for around 1 year.
Workaround
System Reload helps to restore the system.
Further Problem Description
After running normally for almost 1 year the entire emmc goes to Read-Only mode. There is no traffic impact due to this issue. We can't collect any logs when system is in issue state. After reload system recovers but logs of issue state are missing in kernel logs.
