Loading...
Loading...
The NFS protocol for file sharing is rather simple, and by default when an administrator mounts an exported path on a client host, the path and all the fines beneath show on the client side below the specified mount point. NFS may use the files' inode on the server as a means to tell files apart from each other, but file inodes are only guaranteed to be unique within a single filesystem. For example, a NFS server may have several different filesystems, all of which contain a set of equally numbered inodes identifying files. That's why the NFS protocol uses the FSID (the filesystem ID) to communicate with the client about files inodes being for one filesystem or another. On the common case, a NFS export is for the whole or a subset of a single filesystem: NFS clients can mount the NFS export locally and all files below the mount point will be reported on the the same FSID. However, there are situations in which the NFS server exports a path for a particular filesystem, but it has one or more additional filesystems showing because they are locally mounted on the NFS server: if that an export is mounted on an NFS client, there will be files accessible with the FSID for the main filesystem, and others will correspond to the nested filesystems, and hence with their very own FSID. NFS client support in Linux kernels 2.6.18 and later will automatically detect this on the NFS client and, when any FSID change occurs within a mounted NFS export, it will automatically mount the nested NFS server filesystem path on the client, showing as a separate mount, that the administrator has not manually done. For the DataDomain this can occur when exporting whole MTrees through NFS , as all MTrees have a (hidden) ".snapshot/" subdirectory at the top, below which all existing user-accessible (non expired) snapshots for the MTree are shown. DDOS will have different FSIDs for the MTree and for each one of the existing snapshots, so if a recent Linux NFS client mounts the exported MTree, and traverses below the ".snapshot/" subdirectory, a new mount for the ".snapshot/" directory will automatically be made on the NFS client. For example, take a DD device with the following NFS export: NFS Export: Exported_MTree Path: /data/col1/Exported_MTree Tenant-Unit: - NFSv3 Mounts: 0 Active NFSv3 clients: 2 Active NFSv4 clients: 0 Client Options ----------- ---------------------------------------------------------- 192.168.1.1 (sec=sys,rw,no_root_squash,no_all_squash,secure,version=3) 192.168.1.2 (sec=sys,rw,no_root_squash,no_all_squash,secure,version=3) ----------- ---------------------------------------------------------- Total Client Entries: 2 We can see the export is mounted remotely on two NFS clients. On one of those two NFS clients we run "df -h" to check disk space used, and we may see the following: Filesystem Size Used Avail Use% Mounted on 192.168.1.100:/data/col1/Exported_MTree 31T 19T 13T 61% /mountpoint 192.168.1.100:/data/col1/Exported_MTree/.snapshot 31T 19T 13T 61% /mountpoint/.snapshot Without the administrator or any script in the Linux NFS client to have specifically requested a mount for the remote "/data/col1/Exported_MTree/.snapshot" path.
The NFS protocol needs to tell files apart from each other, that's done through the files' inode number on the server, but that's not enough when there are nested filesystems on the exported path. When that's the case each filesystem has its own ID, and the Linux kernel 2.6.18 and later NFS support will detect a FSID change and automatically mount the different filesystems when found below an previously mounted NFS export. However, this behavior is per design, and doesn't indicate a bug of any kind on either the NFS client (Linux) or the NFS server (DataDomain).
There is currently no tuning available on DDOS for the NFS server to instruct the NFS client to not mount new nested filesystems, particularly so for the ".snapshot/" subdirectory present on all MTrees' root directory. This is a comonplace issue with the NFS protocol, which for example also affects NetApp NFS server. DDOS doesn't currently have any option similar to that in NetApp to force the Linux NFS client stop automatically mounting nested filesystems when found, but plans are to add the necessary tunable in future DDOS releases. Note the major issue with ".snapshot/" being automatically mounted is the fact for exported MTrees being part of MTree replication (or MTrees not engaged in replication but with a snapshot schedule in place), snapshots may come and go with time. If access to the files on the NFS client traverses the ephemeral snapshots when they exist and later when they have gone away (have been expired), the NFS client will report errors regarding "stale file handle" such as the ones below: df: /mountpoint/.snapshot/REPL-MTREE-AUTO-2018-05-21-02-11-33 : Stale file handle df: /mountpoint/.snapshot/REPL-MTREE-AUTO-2018-05-21-02-27-17 : Stale file handle In this case, the stale file handles occur because the NFS client had accessed the replication snapshots in the past, and have been since expired, so the file handle to the top directory for the snapshots point to a now non-existent path on the server. The only workaround would be to either totally avoid accessing the ".snapshot/" directory or below on the NFS client, or when stale file handles occur, to unmount and remount the exported path again.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.