Symptoms
Running on low Vnodes is identified by seeing multiple vlruwk threads running:
/var/log/messages 2022-03-11T17:17:01-08:00 Isilon-4(id4) /boot/kernel.amd64/kernel: [rbm_worker.c:350](pid 4738="kt: dxt1")(tid=102982) There are now 505 DXT threads, msg stats: READ_INODE: 1, callback stats: kernel:lk_lock_async_callback2+0: 502, wait stats: biord: 3, vlruwk: 507
From the vmlog, notice that VNODE is taking the highest memory:
vm.uma_zalloc_pigs: UMA Zalloc Pigs:
ZONE NAME SIZE LIMIT COUNT MEM USED
VNODE 000560, 00000000, 00830165, 00453996K
IFSINODE 000568, 00000000, 00777624, 00431338K
VM OBJECT 000240, 00000000, 00813154, 00190582K
lki_lin_ent 000208, 00000000, 00934988, 00189919K
lkc_gen_ent 000064, 00000000, 02235094, 00139693K
stat_cache 000176, 00000000, 00752869, 00129399K
lki_ref_ent 000105, 00000000, 00936379, 00096015K
lkc_lin_idds 000076, 00000000, 01012017, 00075110K
lkc_ref_idds 000068, 00000000, 01008047, 00066940K
mbuf_jumbo_p 004096, 00000000, 00016394, 00065576K
mbuf_cluster 002048, 00000000, 00026630, 00053260K
RADIX NODE 000144, 00000000, 00308743, 00043416K
ifm_attr_cac 000048, 00000000, 00770256, 00036105K
lki_mds_ent 000144, 00000000, 00255978, 00035996K
UMA Slabs 000080, 00000000, 00239085, 00018678K
Unshown zones account for 174014K
Total: 2200045K
Cause
Nodes can run Out Of Memory (OOM) due to not reclaiming free Vnodes. In OneFS 8.2 and later, a new systcl was implemented (vfs.vnlru_reuse_freevnodes) to allow giving up free Vnodes for new Vnodes when the cache is full.By default, vnlru_reuse_freevnodes is off because Isilon prefers to defer Vnodes recycling to vnlru_proc. Turning on vnlru_reuse_freevnodes allows getvnewvnode() to recycle available free Vnodes to make room for new ones when the cache is full (but not over size). This prevents Vnode allocators from sleeping and avoids waking up vnlru_proc.
Resolution
Caution: Get Engineering Approval before disabling or enabling vfs.vnlru_reuse_freevnode. All clusters should be independently evaluated, even if they are a peer cluster or Disaster Recovery cluster.
Process:
Check if the value is set with the following command (the default is 0):
# isi_for_array sysctl vfs.vnlru_reuse_freevnodes
Set vfs.vnlru_reuse_freevnodes to 1:
# isi_sysctl_cluster vfs.vnlru_reuse_freevnodes=1
Verify that the value is set by re-running the command:
# isi_for_array sysctl vfs.vnlru_reuse_freevnodes
To monitor after change, use the below commands to monitor usage of vnode cache and vlurwk threads:
# isi_for_array -s 'sysctl vfs.numvnodes kern.maxvnodes| xargs'; sleep 300
# isi_for_array -s sysctl kern.malloc_pigs|grep vlurwk
