Symptoms

Isilon NDMP backup runs, but stops responding at certain progress bytes. Eventually backup fails with error message "Connection reset by peer."

Cause

Isilon code issue resulting to Isilon cluster nodes maxing out of memory.

Resolution

Troubleshooting steps: Verify the current RUP installed on Isilon. As root, run: Command: isi upgrade patches list Sample output: REDACTED_ISILON_NODE-1# isi upgrade patches list Patch Name Description Status ----------------------------------------------------------------------------- 9.1.0.9_GA-RUP_2021-07_PSP-1248 Multiple Userspace and Kernel Fixes Installed 9.X_GA-TOOLS_2021-06_PSP-1306 Support Tools Installed HealthCheck_9.1.0_2021-06 [9.1.0 UHC 2021-06] HealthCheck RUP Installed ----------------------------------------------------------------------------- Total: 3 Verify if Isilon nodes are experiencing the issue: Command: isi_for_array -QX “grep -h ‘isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition' /var/log/messages” |sort |tail -40 Sample output: REDACTED_ISILON_NODE-1# isi_for_array -QX “grep -h ‘isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition' /var/log/messages” |sort |tail -40 2021-09-07T13:59:26-07:00 REDACTED_ISILON_NODE-2(id2) isi_ndmp_d[30159]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later. 2021-09-07T16:35:15-07:00 REDACTED_ISILON_NODE-1(id1) isi_ndmp_d[32728]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later. 2021-09-07T16:39:28-07:00 REDACTED_ISILON_NODE-4(id4) isi_ndmp_d[13791]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later. 2021-09-07T16:43:42-07:00 REDACTED_ISILON_NODE-1(id1) isi_ndmp_d[32784]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later. 2021-09-07T18:18:24-07:00 REDACTED_ISILON_NODE-6(id6) isi_ndmp_d[12257]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later. 2021-09-08T08:16:08-07:00 REDACTED_ISILON_NODE-3(id3) isi_ndmp_d[64816]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later. 2021-09-08T08:22:17-07:00 REDACTED_ISILON_NODE-4(id4) isi_ndmp_d[7975]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later. 2021-09-08T08:26:24-07:00 REDACTED_ISILON_NODE-3(id3) isi_ndmp_d[66866]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later. 2021-09-08T18:04:47-07:00 REDACTED_ISILON_NODE-1(id1) isi_ndmp_d[93534]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later. 2021-09-08T18:13:09-07:00 REDACTED_ISILON_NODE-3(id3) isi_ndmp_d[84288]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later. 2021-09-08T19:09:42-07:00 REDACTED_ISILON_NODE-4(id4) isi_ndmp_d[43810]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later. 2021-09-09T14:18:57-07:00 REDACTED_ISILON_NODE-1(id1) isi_ndmp_d[59197]: ERRO:NDMP filetransfer.c:1795:isi_ar_sio_reader Failed to allocate reader buffer due to low memory condition, please try the backup operation later. Workaround:Engage Isilon support to apply the workaround.Permanent fix:Upgrade Isilon code to version 9.1.0.10 with latest RUP.

Support Cases