Loading...
Loading...
You may notice the MDM switching roles. The MDM service restarts when it is unable to reach its virtual IP. This is expected if the virtual IP is not reachable by all subnets, and the working subnet has a problem. Events log: 182920 2019-10-16 00:19:09.726 MDM_CLUSTER_FAILED_EXPOSE_VIRT_IP ERROR The Master MDM, MDM-N01 (ID xxxxxxxxxxxxxxxx), could not expose virtual IP addresses. 182921 2019-10-16 00:19:09.899 REMOTE_SYSLOG_MODULE_INITIALIZED INFO Initialized the remote syslog module 182922 2019-10-16 00:19:09.899 MDM_MANAGER_START INFO MDM started with the role of Manager 182923 2019-10-16 00:19:10.013 MDM_CLUSTER_CONNECTED INFO The MDM, MDM-N03 (ID xxxxxxxxxxxxxxxy), connected after 0ms 182924 2019-10-16 00:19:10.013 MDM_CLUSTER_CONNECTED INFO The MDM, MDM-N02 (ID xxxxxxxxxxxxxxxz), connected after 0ms 182925 2019-10-16 00:19:10.013 MDM_CLUSTER_CONNECTED INFO The MDM, TB-N05 (ID xxxxxxxxxxxxxxyy), connected after 0ms 182926 2019-10-16 00:19:10.013 MDM_CLUSTER_CONNECTED INFO The MDM, TB-N04 (ID xxxxxxxxxxxxxxyz), connected after 0ms 182927 2019-10-16 00:19:11.441 MDM_CLUSTER_LOST_CONNECTION WARNING The MDM, MDM-N03 (ID xxxxxxxxxxxxxxxy), has lost connection to the cluster. 182928 2019-10-16 00:19:11.727 MDM_CLUSTER_BECOMING_MASTER WARNING This MDM, MDM-N01 (ID xxxxxxxxxxxxxxxx), took control of the cluster and is now the Master MDM. 182929 2019-10-16 00:19:11.738 MDM_CLUSTER_CONNECTED INFO The MDM, MDM-N03 (ID xxxxxxxxxxxxxxxy), connected after 300ms 182930 2019-10-16 00:19:11.929 MDM_CLUSTER_NODE_DEGRADED ERROR MDM cluster node is now DEGRADED and is in offline node MDM-N03 (ID xxxxxxxxxxxxxxxy); IPs: [10.37.208.13,10.37.200.13,10.37.208.141], Port: 9011 . 182931 2019-10-16 00:19:11.930 MDM_CLUSTER_NODE_NORMAL INFO MDM cluster node MDM-N02 (ID xxxxxxxxxxxxxxxz); IPs: [10.37.208.12,10.37.200.12,10.37.208.140], Port: 9011 is now in NORMAL state. 182932 2019-10-16 00:19:11.930 MDM_BECOMING_MASTER WARNING This MDM is switching to Master mode. MDM will start running. 182933 2019-10-16 00:19:12.145 MDM_CLUSTER_NODE_NORMAL INFO MDM cluster node MDM-N03 (ID xxxxxxxxxxxxxxxy); IPs: [10.37.208.13,10.37.200.13,10.37.208.141], Port: 9011 is now in NORMAL state. 182934 2019-10-16 00:19:12.145 MDM_CLUSTER_NORMAL INFO MDM cluster is now in NORMAL mode. We see that we are unable to reach IP 10.37.200.39 16/10 00:19:09.726004 0x7f40b2417db8:virtIP_IsInterfaceUp:01046: Interface bond1.4090:mdm is down - rc: 0, flags: 0x1003, errno: 11 16/10 00:19:09.726012 0x7f40b2417db8:actor_UmtRepublishVirtualIPs:19591: Virtual interface is down (interface: bond1.4090:mdm, IP: 10.37.200.39, RC: NOT_CONN) 16/10 00:19:09.726015 0x7f40b2417db8:actor_UmtRepublishVirtualIPs:19601: Up virtual IPs (0/1) 16/10 00:19:09.726051 0x7f40b2417db8:mosEventLog_PostInternal:00608: New event added. Message: "The Master MDM, MDM-N01 (ID xxxxxxxxxxxxxxxx), could not expose virtual IP addresses.". Additional info: "" Severity: Error 16/10 00:19:09.726081 0x7f40b2417db8:virtIP_RemoveIpv4:01114: Removed (bond1.4090:mdm) 16/10 00:19:09.726083 0x7f40b2417db8:actor_DoPlannedCrash:01461: --- Actor planned crash at: actor_UmtRepublishVirtualIPs, reason: Failed to republish virtual IPs ---16/10 00:19:09.726100 0x7f40b2417db8:mosDbg_PlannedCrashModuleWithDesc:00599: ---Planned crash, reason: Failed to republish virtual IPs --- 16/10 00:19:09.836386 (nil):mosTrcLayer_Create:00235: ---------- Process started. Version private ScaleIO R3_0.200.104_Release Aug 13 2019. PID 181845 ---------- Frequent MDM restarts may affect availability and performance.
After configuring virtual IP addresses, if the Master MDM discovers that its virtual IP addresses are unreachable, it will try to perform a switch-over. Virtual IP addresses may be unreachable because the data network switch is down and the cluster is using a different network. If no MDM can obtain the virtual IP addresses, the MDM processes might shut down.
Make sure the virtual IP is reachable.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.