Symptom

Customer noticed that in vCenter event, unexpected vDS reconstruction events are logged like attached ScreenShot(vCenter). In VSM "show tech", I could find VMS(PID 3333) on primary VSM crashed and in "show cores", three vms cores for each instances are saved at that timing. Checked show tech, DDTS, SR database, try to check core like below. `show logging` Logging console: enabled (Severity: critical) Logging monitor: enabled (Severity: notifications) 2017 Sep 22 15:52:13.245 xx-n1k10 ethpm[3253]: %ETHPORT-5-IF_UP: Interface Vethernet159 is up in mode access 2017 Sep 26 16:24:59.619 xx-n1k10 vms[3333]: %VMS-5-LOG_FSM_UTILS_FU_GENERIC_ERROR: FSM_ASSERT 2017 Sep 26 16:25:00.016 xx-n1k10 sysmgr[1324]: %SYSMGR-2-SERVICE_CRASHED: Service "vms" (PID 3333) hasn't caught signal 11 (core will be saved). 2017 Sep 26 16:25:10.545 xx-n1k10 vms[4727]: %VMS-5-CONN_CONNECT: Connection 'xx-vcent10' connected to the vCenter Server. 2017 Sep 26 16:25:18.151 xx-n1k10 msp[3326]: %MSP-5-DOMAIN_CFG_SYNC_DONE: Domain config successfully pushed to the management server. 2017 Sep 26 16:25:20.565 xx-n1k10 vms[4727]: %VMS-5-VMS_PPM_SYNC_TIME_EXP: Attempt #0 to resync Port-Profile Manager to local vCenter Server cache due to sync timer expiration 2017 Sep 26 16:25:20.567 xx-n1k10 msp[3326]: %MSP-5-DOMAIN_CFG_SYNC_DONE: Domain config successfully pushed to the management server. 2017 Sep 26 16:25:25.575 xx-n1k10 vms[4727]: %VMS-5-VMS_PPM_SYNC_TIME_EXP: Attempt #0 to resync Port-Profile Manager to local vCenter Server cache due to sync timer expiration 2017 Sep 26 16:25:25.576 xx-n1k10 msp[3326]: %MSP-5-DOMAIN_CFG_SYNC_DONE: Domain config successfully pushed to the management server. 2017 Sep 26 16:28:10.587 xx-n1k10 vms[4727]: %VMS-1-CONN_SSL_NOAUTH: SSL AUTHENTICATION failure. [This message can be safely ignored if vCenter Certificate is not in use.] 2017 Sep 26 16:31:10.673 xx-n1k10 vms[4727]: %VMS-1-CONN_SSL_NOAUTH1: SSL AUTHENTICATION failure. [This message can be safely ignored if vCenter Certificate is not in use.] 2017 Sep 26 16:34:10.774 xx-n1k10 vms[4727]: %VMS-1-CONN_SSL_NOAUTH: SSL AUTHENTICATION failure. [This message can be safely ignored if vCenter Certificate is not in use.] 2017 Sep 26 16:37:10.873 xx-n1k10 vms[4727]: %VMS-1-CONN_SSL_NOAUTH1: SSL AUTHENTICATION failure. [This message can be safely ignored if vCenter Certificate is not in use.] 2017 Sep 26 16:40:10.964 xx-n1k10 vms[4727]: %VMS-1-CONN_SSL_NOAUTH: SSL AUTHENTICATION failure. [This message can be safely ignored if vCenter Certificate is not in use.] 2017 Sep 26 16:57:00 xx-n1k10 %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed for user ucaasope from 10.122.154.190 - sshd[15815] 2017 Sep 26 18:39:39.158 xx-n1k10 vshd[10694]: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by ucaasope on 10.122.154.184@pts/0 2017 Sep 27 12:16:01.741 xx-n1k10 vshd[9866]: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by ucaasope on 10.122.154.184@pts/3 2017 Sep 27 13:47:39.603 xx-n1k10 last message repeated 1 time [Focus point] - The customer didn't noticed network connection issue and so on. Also HA failover didn't occur. - In "show logging" of VSM, I couldn't find network connection loss between vCenter and VSM at this timing. So I think this is not hitting following bug though our customer's version is older than fixed version. https://bst.cloudapps.cisco.com/bugsearch/bug/CSCut02412 - Tried to debug the core with UCD but error occurred. - Cannot find debug-plugin of this N1kv version. - I asked this in ucd and n1kv alias and got some reply and the decoded result is attached txt file "DecodeSession.txt" --Extracted from DecodeSession.txt-- Thread 3 (process 3333): #0 0x41523f92 in syscall () from /auto/andpkg/rep_cache/wr-x86/3.0FCS/sysroot/lib/libc.so.6 No symbol table info available. #1 0xb7840aa1 in os_syscall_select (nfds=7, readfds=0xbfadfc08, writefds=0xbfadfb08, exceptfds=0xbfadfa08, timeout=0xbfae0894) at ../routing-sw/routing/tcpudp/clt/os_syscalls.c:103 No locals. I've been checking if there are similar DDTS or SRs.

Conditions

- version 5.2(1)SV3(1.3) - VMS reboot occurred without some trigger actions, so cannot reproduce this issue.

Workaround

NOT FOUND FOR NOW

Further Problem Description