Symptoms

System memory usage on certain node exceeds 95% of the reservation. 

Cause

Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).

Resolution

Diagnostic Check the systemReserved configuration applied to the nodes: $ oc get nodes [...] $ oc debug node/[node_name] -- cat /host/etc/kubernetes/kubelet.conf [...] systemReserved: cpu: 500m memory: 1Gi ephemeral-storage: 1Gi [...] Check the resources used in the nodes (check the values used by kubelet and runtime if youwant to compare with the configured systemReserved values): $ oc get --raw /api/v1/nodes/<node>/proxy/stats/summary [...] { "node": { "nodeName": "cluster.node22", "systemContainers": [ { "cpu": { "usageCoreNanoSeconds": 929684480915, "usageNanoCores": 190998084 }, "memory": { "rssBytes": 176726016, "usageBytes": 1397895168, "workingSetBytes": 1050509312 }, "name": "kubelet" }, { "cpu": { "usageCoreNanoSeconds": 128521955903, "usageNanoCores": 5928600 }, "memory": { "rssBytes": 35958784, "usageBytes": 129671168, "workingSetBytes": 102416384 }, "name": "runtime" } [... ] ] } } [...] Mitigation Increasing the memory reservations to resolve the issue. For example, to set resource allocation for the default worker MachineConfigPool , create the following CR inside project openshift-config-operator: WARNING: This will trigger a serial restart of nodes using this MachineConfigPool: apiVersion: machineconfiguration.openshift.io/v1 kind: KubeletConfig metadata: name: set-allocatable spec: machineConfigPoolSelector: matchLabels: pools.operator.machineconfiguration.openshift.io/worker: "" kubeletConfig: systemReserved: cpu: 500m # change the CPU if needed memory: 1512Mi # change the memory if needed Support If all the above steps cannot resolve the issue, contact the Dell EMC technical support for further investigation.

Support Cases