Loading...
Loading...
- Backups will not complete even if running for greater than 96hrs, due to remote storage not being accesible. - Killing the backup process will leave a lock on the files as noted in "lsof". - This will prevent change in remote storage under configuration from going through. - If remote storage is unreachable and a backup is attempted to the remote storage, the backup will attempt to run infinitely. - "shutdown -r" will not complete due to not being able to unmount the remote storage, that being due to a file lock. root@fmc:/var/log# shutdown -r now Broadcast message from root@fmc (pts/2) (Thu Feb 16 15:23:05 2023): The system is going down for reboot NOW! shutdown: timeout opening/writing control channel /run/initctl init: timeout opening/writing control channel /run/initctl - Manual intervention is required to resolve the issue, some engineers/customers may opt for a power cycle which can result in database corruption.
Backup started when remote storage is unreachable.
Contact Cisco TAC
- From backup log: Sat Feb 11 01:04:23 2023 Starting CSM backup Sat Feb 11 01:04:23 2023 Entering: main::update_status Sat Feb 11 01:04:23 2023 Update Task: Running CSM backup Sat Feb 11 01:04:23 2023 Exiting: main::update_status Sat Feb 11 01:04:23 2023 processing cfgDBBackup Sat Feb 11 01:04:23 2023 cfgdb backup command /usr/bin/mysqldump -uroot -pTc11g\@q\}V\#r\)mFJ\@VL_XW42_lm\/mnt/remote-storage/sf-storage/e4ce72ea-f4c3-11e9-8f6a-38d70e6b3e60/tmp5849/cfg/cfgdb.dump Sat Feb 11 01:04:23 2023 Entering: main::run_db_cmd - The lock on /mnt/remote-storage/ existed for 4 days before the process was killed manually, however, the remote mount failed back on Feb 7: Feb 7 06:58:16 fmc kernel: [163242.529885] CIFS: VFS: \\firewall.file.core.windows.net Send error in SessSetup = -13 Feb 7 06:58:16 fmc kernel: [163242.529899] CIFS: VFS: cifs_mount failed w/return code = -13 - Right before the lock is made on the remote-storage we see the following in the messages: Feb 11 01:03:04 fmc kernel: [487521.692745] CIFS: VFS: \\REMOTE-STORAGE\Share SMB signature verification returned error = -13 Feb 11 01:03:05 fmc kernel: [487522.788074] CIFS: VFS: sign fail cmd 0x8 message id 0x1459b Feb 11 01:03:05 fmc kernel: [487522.788076] CIFS: VFS: \\REMOTE-STORAGE\Share SMB signature verification returned error = -13 Feb 11 01:03:07 fmc kernel: [487524.299553] CIFS: VFS: sign fail cmd 0x8 message id 0x14d4b Feb 11 01:03:07 fmc kernel: [487524.299555] CIFS: VFS: \\REMOTE-STORAGE\Share SMB signature verification returned error = -13 Feb 11 01:03:11 fmc kernel: [487528.745077] CIFS: VFS: sign fail cmd 0x8 message id 0x15f99 Feb 11 01:03:11 fmc kernel: [487528.745079] CIFS: VFS: \\REMOTE-STORAGE\Share SMB signature verification returned error = -13 Feb 11 01:03:25 fmc kernel: [487541.917605] CIFS: VFS: sign fail cmd 0x8 message id 0x18455 Feb 11 01:03:25 fmc kernel: [487541.917608] CIFS: VFS: \\REMOTE-STORAGE\Share SMB signature verification returned error = -13 Feb 11 01:04:23 fmc kernel: [487600.134895] CIFS: VFS: sign fail cmd 0x8 message id 0x203f6 Feb 11 01:04:23 fmc kernel: [487600.134897] CIFS: VFS: \\REMOTE-STORAGE\Share SMB signature verification returned error = -13 Feb 11 01:07:35 fmc kernel: [487792.383776] CIFS: VFS: \\REMOTE-STORAGE has not responded in 180 seconds. Reconnecting... Feb 11 01:10:43 fmc kernel: [487980.282183] CIFS: VFS: \\REMOTE-STORAGE has not responded in 180 seconds. Reconnecting... Feb 11 01:13:49 fmc kernel: [488166.645193] CIFS: VFS: \\REMOTE-STORAGE has not responded in 180 seconds. Reconnecting... Feb 11 01:16:56 fmc kernel: [488353.008004] CIFS: VFS: \\REMOTE-STORAGE has not responded in 180 seconds. Reconnecting... Feb 11 01:20:04 fmc kernel: [488540.907668] CIFS: VFS: \\REMOTE-STORAGE has not responded in 180 seconds. Reconnecting... - The system is aware it can't reach the remote storage. However, a lock is made on the remote storage anyway.
Click on a version to see all relevant bugs
Cisco Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.