Symptoms
Issue Description:MDM becomes unresponsive when adding devices into a storage pool.Symptoms:The Primary MDM may switch to a different MDM node. Sometimes, the MDM process on the same node may be unresponsive repeatedly causing systemd to block it from restarting. When another MDM takes ownership, the cluster may be in a degraded state.MDM trc includes the following:13/09 20:26:43.894298 0x7fad68c7deb0:sp_AddDevice:00743: StoragePool: DevId: #TGTs: RC: TOO_MANY_DEVICES_IN_STORAGE_POOLMDM exp includes the following:13/09 20:25:43.823370 Panic in file /emc/svc_flashbld/workspace/ScaleIO-OEL-RHEL7/src/mdm/control/obj_container.c, line 1915, function objContainer_GetObjId, PID 1814.Panic Expression (((void *)0) != (pObjHeader)) && ((pObjHeader)->magic == 0x68cab8db) . /opt/emc/scaleio/mdm/bin/mdm-2.0.14000.231(mosDbg_PanicPrepare+0x11d) [0x69813d] /opt/emc/scaleio/mdm/bin/mdm-2.0.14000.231(objContainer_GetObjId+0x34) [0x537e08] /opt/emc/scaleio/mdm/bin/mdm-2.0.14000.231(tgtDev_GetIdUnlocked+0x3a) [0x50a227] /opt/emc/scaleio/mdm/bin/mdm-2.0.14000.231(tgtMgr_AddDevice+0x4e1) [0x50ec68] /opt/emc/scaleio/mdm/bin/mdm-2.0.14000.231(ctrl_AddDevice+0x2ae) [0x5eb90f] /opt/emc/scaleio/mdm/bin/mdm-2.0.14000.231() [0x5158c7] /opt/emc/scaleio/mdm/bin/mdm-2.0.14000.231(netRecvGroup_CallRecvRequest+0x14) [0x62f01e] /opt/emc/scaleio/mdm/bin/mdm-2.0.14000.231(netRecvGroup_WaitForWork+0x4be) [0x62f50a] /opt/emc/scaleio/mdm/bin/mdm-2.0.14000.231(netRecvGroup_WaitForWorkLoop+0x16) [0x62f537] /opt/emc/scaleio/mdm/bin/mdm-2.0.14000.231(mosUmt_StartFunc+0xea) [0x6b4c7f]Impact:Degraded cluster.
Cause
The system does not allow adding more than 512 devices into the same storage pool. The recommendation is not to exceed 300 in the same storage pool, due to MTBF considerations. However, an issue in MDM causes it to be unresponsive when an attempt is made to add more than 512 devices.
Resolution
Avoid adding more than 512 devices into one storage pool.Sometimes, MDM process must be enabled again in systemd. Restart the MDM process, if necessary.
