Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
- Backups will not complete even if running for greater than 96hrs, due to remote storage not being accesible. - Killing the backup process will leave a lock on the files as noted in "lsof". - This will prevent change in remote storage under configuration from going through. - If remote storage is unreachable and a backup is attempted to the remote storage, the backup will attempt to run infinitely. - "shutdown -r" will not complete due to not being able to unmount the remote storage, that being due to a file lock. root@fmc:/var/log# shutdown -r now Broadcast message from root@fmc (pts/2) (Thu Feb 16 15:23:05 2023): The system is going down for reboot NOW! shutdown: timeout opening/writing control channel /run/initctl init: timeout opening/writing control channel /run/initctl - Manual intervention is required to resolve the issue, some engineers/customers may opt for a power cycle which can result in database corruption.
Backup started when remote storage is unreachable.
Contact Cisco TAC
- From backup log: Sat Feb 11 01:04:23 2023 Starting CSM backup Sat Feb 11 01:04:23 2023 Entering: main::update_status Sat Feb 11 01:04:23 2023 Update Task: Running CSM backup Sat Feb 11 01:04:23 2023 Exiting: main::update_status Sat Feb 11 01:04:23 2023 processing cfgDBBackup Sat Feb 11 01:04:23 2023 cfgdb backup command /usr/bin/mysqldump -uroot -pTc11g\@q\}V\#r\)mFJ\@VL_XW42_lm\/mnt/remote-storage/sf-storage/e4ce72ea-f4c3-11e9-8f6a-38d70e6b3e60/tmp5849/cfg/cfgdb.dump Sat Feb 11 01:04:23 2023 Entering: main::run_db_cmd - The lock on /mnt/remote-storage/ existed for 4 days before the process was killed manually, however, the remote mount failed back on Feb 7: Feb 7 06:58:16 fmc kernel: [163242.529885] CIFS: VFS: \\firewall.file.core.windows.net Send error in SessSetup = -13 Feb 7 06:58:16 fmc kernel: [163242.529899] CIFS: VFS: cifs_mount failed w/return code = -13 - Right before the lock is made on the remote-storage we see the following in the messages: Feb 11 01:03:04 fmc kernel: [487521.692745] CIFS: VFS: \\REMOTE-STORAGE\Share SMB signature verification returned error = -13 Feb 11 01:03:05 fmc kernel: [487522.788074] CIFS: VFS: sign fail cmd 0x8 message id 0x1459b Feb 11 01:03:05 fmc kernel: [487522.788076] CIFS: VFS: \\REMOTE-STORAGE\Share SMB signature verification returned error = -13 Feb 11 01:03:07 fmc kernel: [487524.299553] CIFS: VFS: sign fail cmd 0x8 message id 0x14d4b Feb 11 01:03:07 fmc kernel: [487524.299555] CIFS: VFS: \\REMOTE-STORAGE\Share SMB signature verification returned error = -13 Feb 11 01:03:11 fmc kernel: [487528.745077] CIFS: VFS: sign fail cmd 0x8 message id 0x15f99 Feb 11 01:03:11 fmc kernel: [487528.745079] CIFS: VFS: \\REMOTE-STORAGE\Share SMB signature verification returned error = -13 Feb 11 01:03:25 fmc kernel: [487541.917605] CIFS: VFS: sign fail cmd 0x8 message id 0x18455 Feb 11 01:03:25 fmc kernel: [487541.917608] CIFS: VFS: \\REMOTE-STORAGE\Share SMB signature verification returned error = -13 Feb 11 01:04:23 fmc kernel: [487600.134895] CIFS: VFS: sign fail cmd 0x8 message id 0x203f6 Feb 11 01:04:23 fmc kernel: [487600.134897] CIFS: VFS: \\REMOTE-STORAGE\Share SMB signature verification returned error = -13 Feb 11 01:07:35 fmc kernel: [487792.383776] CIFS: VFS: \\REMOTE-STORAGE has not responded in 180 seconds. Reconnecting... Feb 11 01:10:43 fmc kernel: [487980.282183] CIFS: VFS: \\REMOTE-STORAGE has not responded in 180 seconds. Reconnecting... Feb 11 01:13:49 fmc kernel: [488166.645193] CIFS: VFS: \\REMOTE-STORAGE has not responded in 180 seconds. Reconnecting... Feb 11 01:16:56 fmc kernel: [488353.008004] CIFS: VFS: \\REMOTE-STORAGE has not responded in 180 seconds. Reconnecting... Feb 11 01:20:04 fmc kernel: [488540.907668] CIFS: VFS: \\REMOTE-STORAGE has not responded in 180 seconds. Reconnecting... - The system is aware it can't reach the remote storage. However, a lock is made on the remote storage anyway.