Recent Changes - Search:

Disclaimer

edit SideBar

Install NFSv4

1.  NFSv4

is the Network File System version 4. Compared to the common NFSv3, it uses a single port (nice for firewalls), is supposed to be able to do mapping of user IDs (no need to keep UIDs in sync), can use Kerberos authentication, and could have slightly better performance.

1.1  idmap problem

NFSv4 was supposed to do user mapping based on names, but this is only implemented "one-way" in Linux, on current (Jan 2009) SL4 and SL5. What follows is a test between gridvm (SL4) and physfs (SL5), which do not share NIS, with one user with same UID on both (sergio) and one with different UIDs (benc)

[root@gridvm]# sudo -u benc touch /nfs/data/users/benc_gridvm
[root@gridvm]# sudo -u sergio touch /nfs/data/users/sergio_gridvm
[root@physfs]# sudo -u sergio touch /mnt/t/users/sergio_physfs
[root@physfs]# sudo -u benc touch /mnt/t/users/benc_physfs
[root@gridvm]# ls -la /nfs/data/users
total 8
drwxrwxr-x  2 root   users 4096 Jan 22 11:30 .
drwxrwxr-x  3 root   wheel 4096 Jan 22 11:27 ..
-rw-r--r--  1 benc   users    0 Jan 22 11:29 benc_gridvm
-rw-r--r--  1    701 users    0 Jan 22 11:28 benc_physfs
-rw-r--r--  1 sergio users    0 Jan 22 11:30 sergio_gridvm
-rw-r--r--  1 sergio users    0 Jan 22 11:28 sergio_physfs
[root@physfs]# ls -la /mnt/t/users
total 8
drwxrwxr-x 2 root   users 4096 Jan 22 11:30 .
drwxrwxr-x 3 root   wheel 4096 Jan 22 11:27 ..
-rw-r--r-- 1 benc   users    0 Jan 22 11:29 benc_gridvm
-rw-r--r-- 1 nobody users    0 Jan 22 11:28 benc_physfs
-rw-r--r-- 1 sergio users    0 Jan 22 11:30 sergio_gridvm
-rw-r--r-- 1 sergio users    0 Jan 22 11:28 sergio_physfs
[root@physfs]# ls -lan /mnt/t/users
total 8
drwxrwxr-x 2   0 100 4096 Jan 22 11:30 .
drwxrwxr-x 3   0  10 4096 Jan 22 11:27 ..
-rw-r--r-- 1 701 100    0 Jan 22 11:29 benc_gridvm
-rw-r--r-- 1  99 100    0 Jan 22 11:28 benc_physfs
-rw-r--r-- 1 599 100    0 Jan 22 11:30 sergio_gridvm
-rw-r--r-- 1 599 100    0 Jan 22 11:28 sergio_physfs

The net result is that NFSv4 does not yet provide the ID mapping, and one must be very careful that users have the same ID and same name between client and server.

2.  NFSv4 File Server on GridVM

Configuring NFSv4 following http://www.vanemery.com/Linux/NFSv4/NFSv4-no-rpcsec.html#setup

/etc/exports:

/nfs            *(ro,sync,fsid=0,no_subtree_check,anonuid=65534,anongid=65534)
/nfs/osg        152.106.18.251(rw,async,nohide,no_subtree_check,no_root_squash)
/nfs/spool_pbs  152.106.18.251(ro,async,nohide,no_subtree_check,no_root_squash)
/nfs/data       152.106.18.0/24(rw,sync,nohide,no_subtree_check,anonuid=65534,anongid=65534)
/nfs/osg        152.106.18.0/24(rw,sync,nohide,no_subtree_check,anonuid=65534,anongid=65534)
/nfs/glite      152.106.18.0/24(rw,sync,nohide,no_subtree_check,anonuid=65534,anongid=65534)
/nfs/data       10.0.0.0/24(rw,sync,nohide,no_subtree_check,anonuid=65534,anongid=65534)
/nfs/osg        10.0.0.0/24(rw,sync,nohide,no_subtree_check,anonuid=65534,anongid=65534)
/nfs/glite      10.0.0.0/24(rw,sync,nohide,no_subtree_check,anonuid=65534,anongid=65534)

Please note the /nfs *(.... line. It seems that there is no way to have multiple fsid=0 lines for NFSv4 in linux, and if you specify a subnet there, hosts outside that subnet get a Permission denied when they attempt to mount. So one has to rely on iptables to restrict access - which is a better idea anyway.

hosts.allow:

portmap : 127. : ALLOW
portmap : 10.0.0. : ALLOW
portmap : ALL : DENY

Contrary to what is written there, the nfs service must be on for anything to work!

chkconfig nfs on
chkconfig nfslock off
chkconfig rpcgssd off
chkconfig rpcsvcgssd off
chkconfig portmap on
chkconfig rpcidmapd on
service nfslock stop
service rpcgssd stop
service rpcsvcgssd stop
service portmap restart
service rpcidmapd restart
service nfs restart
rpcinfo -p
exportfs -rav

3.  Configure NFSv4 on clients

http://www.nfsv4.org/nfsv4-wg-archive-feb-03-feb-05/0543.html

Configure user ID mapping for NFSv4 when using NIS - /etc/idmapd.conf :

[General]
Verbosity = 0
Pipefs-Directory = /var/lib/nfs/rpc_pipefs
Domain = ujcluster

[Mapping]
Nobody-User = nobody
Nobody-Group = nobody

[Translation]
Method = nsswitch
shm "chkconfig portmap on;chkconfig rpcidmapd on"
shm "service portmap start; service rpcidmapd start; rpcinfo -p"
shm "mkdir -p /nfs/"
shm "service autofs start;chkconfig autofs on"

Check that NFS mounting and user ID mapping works correctly:

]# umount /nfs/home
# service rpcidmapd restart;
Shutting down RPC idmapd:                                  [  OK  ]
Starting RPC idmapd:                                       [  OK  ]
# ls -la /nfs/home
total 28
drwxr-xr-x   8 root    root   4096 Jan 20 19:45 .
drwxr-xr-x   3 root    root      0 Jan 21 10:12 ..
drwx------   7 benc    users  4096 Jan 21 09:23 clusteradm
drwx------   4 cookish users  4096 Nov 24 22:20 cookish
drwx------   4 ives    users  4096 Dec  5 15:15 ives
drwx------   4 osgedu  nobody 4096 Dec  5 15:51 osgedu
drwx------   5 nobody  nobody 4096 Nov 24 21:21 psiguest
drwx--x--x  10 sergio  users  4096 Jan 20 18:24 sergio

4.  debug

By default there is almost no debug info in logs for nfs. To turn it on,

[root@osg-ui log]# rpcdebug  -m nfs -s all
[root@gridvm log]# rpcdebug  -m nfsd -s all

To turn it on at boot time, when it's most useful, put in gridvm:/etc/sysconfig/nfs

rpcdebug  -m nfsd -s all

showmount --exports is an alternative to exportfs -v. Not sure if there is any difference between the two.

[root@gridvm log]# showmount
[root@gridvm log]# cat /proc/net/rpc/nfsd.fh/content

5.  Startup problem

I could not get NFS running properly at the boot of the server, I was always getting mount: permission denied on the clients. But when I restarted nfs, everything worked.

I had a hint from http://linux-nfs.org/pipermail/nfsv4/2004-December/000900.html , discussing the problems of the init sequence of nfsd and idmapd.

I made a script to test this in a consistent way, and I figured out that one thing that was different was the kernel modules, so in the testing the modules are unloaded, like at boot time:

#!/bin/bash
set -x
service nfs stop
service rpcidmapd stop
sleep 2
modprobe -r nfsd nfs
sleep 2
service	rpcidmapd start
service	nfs start

What seems to solve the problem is forcing a preload of the nfsd module in /etc/init.d/nfs

case "$1" in
  start)
  	# preload kernel modules
        modprobe nfsd

Now let's reboot again and see if it works... Yes!

6.  Links

Edit - History - Print - Recent Changes - Search
Page last modified on April 11, 2009, at 01:39 PM