Symptoms

Running on low Vnodes is identified by seeing multiple vlruwk threads running: /var/log/messages 2022-03-11T17:17:01-08:00 Isilon-4(id4) /boot/kernel.amd64/kernel: [rbm_worker.c:350](pid 4738="kt: dxt1")(tid=102982) There are now 505 DXT threads, msg stats: READ_INODE: 1, callback stats: kernel:lk_lock_async_callback2+0: 502, wait stats: biord: 3, vlruwk: 507 From the vmlog, notice that VNODE is taking the highest memory: vm.uma_zalloc_pigs: UMA Zalloc Pigs: ZONE NAME SIZE LIMIT COUNT MEM USED VNODE 000560, 00000000, 00830165, 00453996K IFSINODE 000568, 00000000, 00777624, 00431338K VM OBJECT 000240, 00000000, 00813154, 00190582K lki_lin_ent 000208, 00000000, 00934988, 00189919K lkc_gen_ent 000064, 00000000, 02235094, 00139693K stat_cache 000176, 00000000, 00752869, 00129399K lki_ref_ent 000105, 00000000, 00936379, 00096015K lkc_lin_idds 000076, 00000000, 01012017, 00075110K lkc_ref_idds 000068, 00000000, 01008047, 00066940K mbuf_jumbo_p 004096, 00000000, 00016394, 00065576K mbuf_cluster 002048, 00000000, 00026630, 00053260K RADIX NODE 000144, 00000000, 00308743, 00043416K ifm_attr_cac 000048, 00000000, 00770256, 00036105K lki_mds_ent 000144, 00000000, 00255978, 00035996K UMA Slabs 000080, 00000000, 00239085, 00018678K Unshown zones account for 174014K Total: 2200045K

Cause

Nodes can run Out Of Memory (OOM) due to not reclaiming free Vnodes. In OneFS 8.2 and later, a new systcl was implemented (vfs.vnlru_reuse_freevnodes) to allow giving up free Vnodes for new Vnodes when the cache is full.By default, vnlru_reuse_freevnodes is off because Isilon prefers to defer Vnodes recycling to vnlru_proc. Turning on vnlru_reuse_freevnodes allows getvnewvnode() to recycle available free Vnodes to make room for new ones when the cache is full (but not over size). This prevents Vnode allocators from sleeping and avoids waking up vnlru_proc.

Resolution

Caution: Get Engineering Approval before disabling or enabling vfs.vnlru_reuse_freevnode. All clusters should be independently evaluated, even if they are a peer cluster or Disaster Recovery cluster. Process: Check if the value is set with the following command (the default is 0): # isi_for_array sysctl vfs.vnlru_reuse_freevnodes Set vfs.vnlru_reuse_freevnodes to 1: # isi_sysctl_cluster vfs.vnlru_reuse_freevnodes=1 Verify that the value is set by re-running the command: # isi_for_array sysctl vfs.vnlru_reuse_freevnodes To monitor after change, use the below commands to monitor usage of vnode cache and vlurwk threads: # isi_for_array -s 'sysctl vfs.numvnodes kern.maxvnodes| xargs'; sleep 300 # isi_for_array -s sysctl kern.malloc_pigs|grep vlurwk

Support Cases