Loading...
Loading...
When using NFSv4 with krb5 security style, when you reboot a node while performing a 'cp' or 'dd' of a large file, you receive a "permission denied" error and the copy operation starts all over. Node is part of a dynamic SmartConnect pool. Findings indicate that the issue occurs upon node IP failback and *not* node IP failover. In other words, the issue occurs right at the point that the rebooted node boots up and at the point the IP address is reassigned back to the rebooted node.Dell Technologies is unable to reproduce the issue with a manual failback once the node is booted back up.Example of error as seen on the Network File System (NFS) client: dd: error writing ./mnt/nfs2/largefile.: Bad file descriptor dd: closing output file ./mnt/nfs2/largefile.: Input/output error dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied Replication: Export:ID: 3 Zone: System Paths: /ifs/data/nfs4kerb2 Description: NFSv4 kerberos test Clients: 10.10.10.10 Root Clients: - Read Only Clients: - Read Write Clients: 10.10.10.10 All Dirs: No Block Size: 8.0k Can Set Time: Yes Case Insensitive: No Case Preserving: Yes Chown Restricted: No Commit Asynchronous: No Directory Transfer Size: 128.0k Encoding: DEFAULT Link Max: 32767 Map Lookup UID: No Map Retry: Yes Security Type: krb5p ----> see here Mount option:s1.isilon.com:/ifs/data/nfs4kerb2 on /mnt/nfs2 type nfs4 (rw,relatime,vers=4.0,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5p,clientaddr=xxx.xxx.xxx.xxx,local_lock=none,addr=xxx.xxx.xxx.xxx)script started. It writes large file#cat nfs4.ksh #!/bin/ksh while true do dd if=/dev/random of=/mnt/nfs2/largefile bs=1024 count=4000000 done$ ./nfs4.ksh dd: warning: partial read (115 bytes); suggest iflag=fullblock$ ls -l total 288 -rw------- 1 isilon isilon 67244 Nov 7 22:46 largefile -rwx------ 1 isilon isilon 94 Nov 5 21:41 nfs4.ksh $ ls -l total 480 -rw------- 1 isilon isilon 154994 Nov 7 22:46 largefile -rwx------ 1 isilon isilon 94 Nov 5 21:41 nfs4.ksh $During copying on NFS rebooted Node 8 (This was failover test)isilon01-1# isi config Welcome to the Isilon IQ configuration console. Copyright (c) 2001-2018 EMC Corporation. All Rights Reserved. Enter 'help' to see list of available commands. Enter 'help command>' to see help for a specific command. Enter 'quit' at any prompt to discard changes and exit.isilon01 >>> reboot 8!! You are about to reboot the following nodes: 8Are you sure you wish to continue? [no] >>> yes isilon01 >>>Saw pause when tried to list "ls -l "$ ls -lalso no error on the copy script window$ ./nfs4.ksh dd: warning: partial read (115 bytes); suggest iflag=fullblockls command resumed and copy also resumed as you can see below after 3 seconds. You can see file size is not starting from the start.$ ls -l total 736 -rw------- 1 isilon isilon 414942 Nov 7 22:48 largefile -rwx------ 1 isilon isilon 94 Nov 5 21:41 nfs4.ksh $ ls -l total 824 -rw------- 1 isilon isilon 500294 Nov 7 22:49 largefile -rwx------ 1 isilon isilon 94 Nov 5 21:41 nfs4.ksh $ ls -l total 832 -rw------- 1 isilon isilon 511567 Nov 7 22:49 largefile -rwx------ 1 isilon isilon 94 Nov 5 21:41 nfs4.kshat cluster end node is offline and rebootingisilon01-1# isi status Cluster Name: isilon01 Cluster Health: [ ATTN] Cluster Storage: HDD SSD Storage Size: 417.4T (432.5T Raw) 61.5T (64.6T Raw) VHS Size: 18.3T Used: 10.1G ( 1%) 9.3G ( 1%) Avail: 417.4T (> 99%) 61.5T (> 99%)Health Throughput (bps) HDD Storage SSD Storage ID |IP Address |DASR | In Out Total| Used / Size |Used / Size ---+----------------+-----+-----+-----+-----+-----------------+----------------- 1|10.10.10.235 | OK | 0|39.8k|39.8k| 2.5G/ 104T( 1%)| L3: 373G 2|10.10.10.236 | OK | 0| 213k| 213k| 2.6G/ 104T( 1%)| L3: 373G 3|10.10.10.237 | OK | 7.4M|99.5k| 7.5M| 2.5G/ 104T( 1%)| L3: 373G 4|10.10.10.238 | OK | 0|53.2k|53.2k| 2.5G/ 104T( 1%)| L3: 373G 5|10.10.10.218 | OK | 0| 0| 0|(No Storage HDDs)| 3.3G/20.5T( 1%) 6|10.10.10.212 | OK | 0| 298k| 298k|(No Storage HDDs)| 2.7G/20.5T( 1%) 7|10.10.10.211 | OK | 179k| 348k| 527k|(No Storage HDDs)| 3.3G/20.5T( 1%) 8|n/a |D--- | n/a| n/a| n/a| n/a/ n/a( n/a)| n/a/ n/a( n/a) ---+----------------+-----+-----+-----+-----+-----------------+----------------- Cluster Totals: | n/a| n/a| n/a|10.1G/ 417T( 1%)| 9.3G/61.5T( 1%)Health Fields: D = Down, A = Attention, S = Smartfailed, R = Read-OnlyCritical Events:11/07 17:48 8 Node 8 is offline.230 client Ip now moved to node 7 during node failureisilon01-1# isi_for_array -n 7 "ifconfig -a" |grep 210 isilon01-7: inet 10.10.10.229 netmask 0xffffff80 broadcast 10.10.10.255 zone 1 isilon01-7: inet 10.10.10.217 netmask 0xffffff80 broadcast 10.10.10.255 zone 1 isilon01-7: inet 10.10.10.224 netmask 0xffffff80 broadcast 10.10.10.255 zone 1 isilon01-7: inet 10.10.10.225 netmask 0xffffff80 broadcast 10.10.10.255 zone 1 isilon01-7: inet 10.10.10.211 netmask 0xffffff80 broadcast 10.10.10.255 zone 1 isilon01-7: inet 10.10.10.216 netmask 0xffffff80 broadcast 10.10.10.255 zone 1 isilon01-7: inet 10.10.10.230 netmask 0xffffff80 broadcast 10.10.10.255 zone 1Noticed when node 8 was coming online below error message was scrolling on the screen and noticed write operation aborted and resumed freshly ( see below )$ ./nfs4.ksh dd: warning: partial read (115 bytes); suggest iflag=fullblock dd: error writing ./mnt/nfs2/largefile.: Bad file descriptor dd: closing output file ./mnt/nfs2/largefile.: Input/output error dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission denied dd: failed to open ./mnt/nfs2/largefile.: Permission deniedyou can see large file copy restarted newly with small size ls -l total 288 -rw------- 1 isilon isilon 66661 Nov 7 22:54 largefile -rwx------ 1 isilon isilon 94 Nov 5 21:41 nfs4.ksh $ ls -l total 472 -rw------- 1 isilon isilon 145475 Nov 7 22:55 largefile -rwx------ 1 isilon isilon 94 Nov 5 21:41 nfs4.ksh
The PowerScale service/daemon which manages IP assignment and IP failover/failback is called FlexNet, or as seen in the process listing (ps), isi_flexnet_d. The PowerScale service/daemon which manages Authentication on Isilon is called 'lsass'. When using krb5 security with NFS, node communication with Active Directory is typically heavily involved. For example, when OneFS uses krb5(i|p) security, OneFS pull the UPN (user principal name) from the ticket and look to lsass for a token based on UPN (user@domain.com). If the domain (domain.com) is not found or not yet available (due to the lsass process loading), a customer may end up with "no such user" or "permission denied" errors. In some environments, a delay in AD communication upon node bootup may occur. In our scenario above, when the node booted up, FlexNet began to initialize before lsass finished loading the full configuration from AD. FlexNet is loading and rebalancing the IPs before lsass is *ready* to handle token requests. In other words, Download Center began to initialize before Lsass finished loading the full configuration from AD Download Center is not equipped to "wait" until everything is ready. Note the wording: "lsass is ready": Lsass is starting up fine, but there is a delay in loading the full configuration from AD.
Implement the IP rebalance delay, based on the following: FlexNet is loading and rebalancing the IPs before lsass is ready to handle token requests. OneFS cannot control the loading time of lsass, but it can induce IP rebalance delay. It all comes down to whether or not lsass has loaded the full configuration from AD when the client moves over, and in this particular scenario, the delay was necessary. To set rebalance delay: (The default setting is '0', meaning no delay) # isi network external modify --sc-rebalance-delay=10confirm new setting:# isi network external view Client TCP Ports: 2049, 445, 20, 21, 80 Default Groupnet: groupnet0 SC Rebalance Delay: 10 Source Based Routing: False Note : The maximum rebalance delay, by default, is '10' If a longer delay is needed, contact PowerScale Support as there is an internal workaround to set the rebalance delay to a larger value. Optional: Configure a separate static IP on all nodes in the NFS SmartConnect pool, even though the NFS pool is dynamic. In one scenario, you did not have a static IP on the NFS nodes. Without a static IP, the described issue is going to be more severe. Why? Remember, there is no failover/failback process with static IPs. The IP address always stays on the same node, and when the node boots up the IP is available right away. An IP must be able to talk to the AD server and load the configuration. Reference: Dell PowerScale: Network Design Considerations https://infohub.delltechnologies.com/en-us/t/dell-powerscale-network-design-considerations/ To review, PowerScale has various "moving parts" or components at play here: --FlexNet - not equipped to wait for lsass. --Static IP --Dynamic IP --Rebalance delay --Lsass - once it starts, it must load AD configuration, exponential timer makes things worse. --NFS sec style: krb5 - dependent on AD and lsass loading full configuration from AD --AD server
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.