Loading...
Loading...
admin@ecse01n01:~> doit uptime 192.168.219.1: 09:38am up 6 days 20:51, 3 users, load average: 10.14, 10.42, 9.66 192.168.219.2: 09:38am up 6 days 23:33, 0 users, load average: 24.07, 67.30, 174.18 192.168.219.3: 09:38am up 6 days 19:32, 2 users, load average: 11.78, 13.23, 39.34 192.168.219.4: 09:38am up 6 days 17:47, 0 users, load average: 8.73, 10.05, 10.82 192.168.219.5: 09:39am up 6 days 16:37, 0 users, load average: 367.39, 311.72, 230.97 192.168.219.6: 09:39am up 6 days 15:29, 0 users, load average: 10.11, 13.33, 14.00 192.168.219.7: 09:39am up 6 days 14:19, 0 users, load average: 10.16, 11.93, 12.17 192.168.219.8: 09:39am up 6 days 13:15, 0 users, load average: 13.43, 15.11, 15.71 192.168.219.9: 09:39am up 6 days 12:19, 0 users, load average: 8.44, 10.68, 11.33 192.168.219.10: 09:39am up 6 days 11:26, 1 user, load average: 8.59, 10.87, 11.10 192.168.219.11: 09:39am up 6 days 10:31, 0 users, load average: 10.00, 12.37, 12.90 192.168.219.12: 09:39am up 6 days 9:39, 0 users, load average: 12.94, 14.58, 14.58 192.168.219.13: 09:39am up 6 days 8:45, 0 users, load average: 11.00, 11.69, 11.64 192.168.219.14: 09:39am up 6 days 7:54, 0 users, load average: 9.62, 11.04, 11.42 192.168.219.15: 09:39am up 6 days 7:02, 0 users, load average: 16.09, 16.46, 15.09
Memory parameter needs adjusting on ECS nodes due to high overall memory usage or high load.
IMPORTANT! A new feature has been released in xDoctor 4-8.104 and above. This knowledge base (KB) is now automated with xDoctor, addressing most issues listed below without the need for L2 or coach involvement. If the script is unable to resolve the issue it provides a detailed summary of its findings. For more information, follow ECS: ObjectScale: How to run KB Automation Scripts (Auto Pilot) . Automated Solution: To find the master node of the rack: Command: # ssh master.rack To find the NAN IP, you can use the IP identified in the alert or from the getrackinfo command: admin@ecsnode1:~> getrackinfo Node private Node Public BMC Ip Address Id Status Mac Ip Address Mac Ip Address Private.4(NAN) Node Name =============== ====== ====== ================= =============== ================= =============== =============== ========= 192.168.219.1 1 MA 00:00:00:00:00 0.0.0.0 00:00:00:00:00 192.168.219.101 169.254.1.1 provo-red 192.168.219.2 2 SA 00:00:00:00:00 0.0.0.0 00:00:00:00:00 192.168.219.102 169.254.1.2 sandy-red 192.168.219.3 3 SA 00:00:00:00:00 0.0.0.0 00:00:00:00:00 192.168.219.103 169.254.1.3 orem-red 192.168.219.4 4 SA 00:00:00:00:00 0.0.0.0 00:00:00:00:00 192.168.219.104 169.254.1.4 ogden-red 192.168.219.5 5 SA 00:00:00:00:00 0.0.0.0 00:00:00:00:00 192.168.219.105 169.254.1.5 layton-red 192.168.219.6 6 SA 00:00:00:00:00 0.0.0.0 00:00:00:00:00 192.168.219.106 169.254.1.6 logan-red 192.168.219.7 7 SA 00:00:00:00:00 0.0.0.0 00:00:00:00:00 192.168.219.107 169.254.1.7 lehi-red 192.168.219.8 8 SA 00:00:00:00:00 0.0.0.0 00:00:00:00:00 192.168.219.108 169.254.1.8 murray-red Run the automation command from the master node with xDoctor 4-8.104.0 and above. Command: Note: --target-node, --target-vdc or --target-rack is supported for this action. # sudo xdoctor autopilot --kb 209779 --target-vdc <VDC Name> # sudo xdoctor autopilot --kb 209779 --target-rack <Rack Color> # sudo xdoctor autopilot --kb 209779 --target-node <NAN IP> Example: admin@ecsnode1:~> sudo xdoctor autopilot --kb 209779 --target-node 169.254.1.1 Checking for existing screen sessions... Starting screen session 'autopilot_kb_209779_20250630_144700'... Screen session 'autopilot_kb_209779_20250630_144700' started successfully. Attaching to screen session 'autopilot_kb_209779_20250630_144700'... Review a summary of actions taken. Example: TASK [Summary of changes] ***************************************************************************************************************************************************************************************************************************************************** ok: [169.254.1.1 -> localhost] => { "msg": [ "************************************************************************************************************************************************", "Summary of swappiness changes:", "************************************************************************************************************************************************", "FIXED: vm.swappiness changed on node(s): 169.254.1.1", "FIXED: vm.min_free_kbytes changed on node(s): 169.254.1.1", "PASS: No change needed to /etc/sysctl.d/90-ecs-os-base.conf on node(s): 169.254.1.1", "************************************************************************************************************************************************" ] } Confirm to update the MOTD with swap details and the changes applied. Command: (Optional) Note: If your current MOTD has the KB number, swapiness or vm.min_free_kbytes in it the MOTD will not be updated. # yes or no Example: The next steps are to detect keywords in the MOTD content and update the MOTD with custom content for the KB specifics. IMPORTANT! You will need customer permission to do this action for the KB. Would you like to proceed with the steps by typing 'Yes' or 'Y', or skip the MOTD update actions by typing 'No' or 'N': : y Review MOTD change summary for adding new MOTD content: Example: TASK [MOTD summary] **************************************************************************************************************************************************************************** ok: [169.254.1.7] => { "msg": [ "************************************************************************************************************************************************", "New MOTD content:", "************************************************************************************************************************************************", "************************************************************************************************************************************************", [ "", "==============================================================================================================================================", "2025-06-30 | KB 209779 has been applied to this system, adjusting vm.min_free_kbytes to 1048576 for systems with 62GB or to 3145728 for 192GB systems, and setting vm.swappiness to 1 for the cluster. Please reapply if a node has been reimaged.", "==============================================================================================================================================" ], "************************************************************************************************************************************************", "************************************************************************************************************************************************" ] } Manual steps for adjusting SWAP handling are available for support use only. If the automation does not provide the wanted resolution or encounters an error, open a service request with ECS/OBS support to investigate further. Base KB for this automation: ECS High load observed on ECS nodes/performance issues observed in 3.6.x/3.7.x/3.8.x Additional KB consolidated into this automation: ECS: High Memory Load and Potential Performance Issues on ECS 3.6, ECS 3.7, and ECS 3.8
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.