protocol supports an EXPORTS procedure that will enumerate the
server's exports.
7.2. Browsing Exports
The NFS version 4 protocol provides a root filehandle that clients
can use to obtain filehandles for these exports via a multi-component
LOOKUP. A common user experience is to use a graphical user
interface (perhaps a file "Open" dialog window) to find a file via
progressive browsing through a directory tree. The client must be
able to move from one export to another export via single-component,
progressive LOOKUP operations.
This style of browsing is not well supported by the NFS version 2 and
3 protocols. The client expects all LOOKUP operations to remain
within a single server file system. For example, the device
attribute will not change. This prevents a client from taking name
space paths that span exports.
An automounter on the client can obtain a snapshot of the server's
name space using the EXPORTS procedure of the MOUNT protocol. If it
understands the server's pathname syntax, it can create an image of
the server's name space on the client. The parts of the name space
that are not exported by the server are filled in with a "pseudo file
system" that allows the user to browse from one mounted file system
to another. There is a drawback to this representation of the
server's name space on the client: it is static. If the server
administrator adds a new export the client will be unaware of it.
7.3. Server Pseudo File System
NFS version 4 servers avoid this name space inconsistency by
presenting all the exports within the framework of a single server
name space. An NFS version 4 client uses LOOKUP and READDIR
operations to browse seamlessly from one export to another. Portions
of the server name space that are not exported are bridged via a
"pseudo file system" that provides a view of exported directories
only. A pseudo file system has a unique fsid and behaves like a
normal, read only file system.
Based on the construction of the server's name space, it is possible
that multiple pseudo file systems may exist. For example,
/a pseudo file system
/a/b real file system
/a/b/c pseudo file system
/a/b/c/d real file system
Each of the pseudo file systems are consider separate entities and
therefore will have a unique fsid.
7.4. Multiple Roots
The DOS and Windows operating environments are sometimes described as
having "multiple roots". File systems are commonly represented as
disk letters. MacOS represents file systems as top level names. NFS
version 4 servers for these platforms can construct a pseudo file
system above these root names so that disk letters or volume names
are simply directory names in the pseudo root.
7.5. Filehandle Volatility
The nature of the server's pseudo file system is that it is a logical
representation of file system(s) available from the server.
Therefore, the pseudo file system is most likely constructed
dynamically when the server is first instantiated. It is expected
that the pseudo file system may not have an on disk counterpart from
which persistent filehandles could be constructed. Even though it is
preferable that the server provide persistent filehandles for the
pseudo file system, the NFS client should expect that pseudo file
system filehandles are volatile. This can be confirmed by checking
the associated "fh_expire_type" attribute for those filehandles in
question. If the filehandles are volatile, the NFS client must be
prepared to recover a filehandle value (e.g. with a multi-component
LOOKUP) when receiving an error of NFS4ERR_FHEXPIRED.
7.6. Exported Root
If the server's root file system is exported, one might conclude that
a pseudo-file system is not needed. This would be wrong. Assume the
following file systems on a server:
/ disk1 (exported)
/a disk2 (not exported)
/a/b disk3 (exported)
Because disk2 is not exported, disk3 cannot be reached with simple
LOOKUPs. The server must bridge the gap with a pseudo-file system.
7.7. Mount Point Crossing
The server file system environment may be constructed in such a way
that one file system contains a directory which is 'covered' or
mounted upon by a second file system. For example:
/a/b (file system 1)
/a/b/c/d (file system 2)
The pseudo file system for this server may be constructed to look
like:
/ (place holder/not exported)
/a/b (file system 1)
/a/b/c/d (file system 2)
It is the server's responsibility to present the pseudo file system
that is complete to the client. If the client sends a lookup request
for the path "/a/b/c/d", the server's response is the filehandle of
the file system "/a/b/c/d". In previous versions of the NFS
protocol, the server would respond with the directory "/a/b/c/d"
within the file system "/a/b".
The NFS client will be able to determine if it crosses a server mount
point by a change in the value of the "fsid" attribute.
7.8. Security Policy and Name Space Presentation
The application of the server's security policy needs to be carefully
considered by the implementor. One may choose to limit the
viewability of portions of the pseudo file system based on the
server's perception of the client's ability to authenticate itself
properly. However, with the support of multiple security mechanisms
and the ability to negotiate the appropriate use of these mechanisms,
the server is unable to properly determine if a client will be able
to authenticate itself. If, based on its policies, the server
chooses to limit the contents of the pseudo file system, the server
may effectively hide file systems from a client that may otherwise
have legitimate access.
8. File Locking and Share Reservations
Integrating locking into the NFS protocol necessarily causes it to be
state-full. With the inclusion of "share" file locks the protocol
becomes substantially more dependent on state than the traditional
combination of NFS and NLM [XNFS]. There are three components to
making this state manageable:
o Clear division between client and server
o Ability to reliably detect inconsistency in state between client
and server
o Simple and robust recovery mechanisms
In this model, the server owns the state information. The client
communicates its view of this state to the server as needed. The
client is also able to detect inconsistent state before modifying a
file.
To support Win32 "share" locks it is necessary to atomically OPEN or
CREATE files. Having a separate share/unshare operation would not
allow correct implementation of the Win32 OpenFile API. In order to
correctly implement share semantics, the previous NFS protocol
mechanisms used when a file is opened or created (LOOKUP, CREATE,
ACCESS) need to be replaced. The NFS version 4 protocol has an OPEN
operation that subsumes the functionality of LOOKUP, CREATE, and
ACCESS. However, because many operations require a filehandle, the
traditional LOOKUP is preserved to map a file name to filehandle
without establishing state on the server. The policy of granting
access or modifying files is managed by the server based on the
client's state. These mechanisms can implement policy ranging from
advisory only locking to full mandatory locking.
8.1. Locking
It is assumed that manipulating a lock is rare when compared to READ
and WRITE operations. It is also assumed that crashes and network
partitions are relatively rare. Therefore it is important that the
READ and WRITE operations have a lightweight mechanism to indicate if
they possess a held lock. A lock request contains the heavyweight
information required to establish a lock and uniquely define the lock
owner.
The following sections describe the transition from the heavy weight
information to the eventual stateid used for most client and server
locking and lease interactions.
8.1.1. Client ID
For each LOCK request, the client must identify itself to the server.
This is done in such a way as to allow for correct lock
identification and crash recovery. Client identification is
accomplished with two values.
o A verifier that is used to detect client reboots.
o A variable length opaque array to uniquely define a client.
For an operating system this may be a fully qualified host name
or IP address. For a user level NFS client it may additionally
contain a process id or other unique sequence.
The data structure for the Client ID would then appear as:
struct nfs_client_id {
opaque verifier[4];
opaque id<>;
}
It is possible through the mis-configuration of a client or the
existence of a rogue client that two clients end up using the same
nfs_client_id. This situation is avoided by "negotiating" the
nfs_client_id between client and server with the use of the
SETCLIENTID and SETCLIENTID_CONFIRM operations. The following
describes the two scenarios of negotiation.
1 Client has never connected to the server
In this case the client generates an nfs_client_id and unless
another client has the same nfs_client_id.id field, the server
accepts the request. The server also records the principal (or
principal to uid mapping) from the credential in the RPC request
that contains the nfs_client_id negotiation request (SETCLIENTID
operation).
Two clients might still use the same nfs_client_id.id due to
perhaps configuration error. For example, a High Availability
configuration where the nfs_client_id.id is derived from the
ethernet controller address and both systems have the same
address. In this case, the result is a switched union that
returns, in addition to NFS4ERR_CLID_INUSE, the network address
(the rpcbind netid and universal address) of the client that is
using the id.
2 Client is re-connecting to the server after a client reboot
In this case, the client still generates an nfs_client_id but the
nfs_client_id.id field will be the same as the nfs_client_id.id
generated prior to reboot. If the server finds that the
principal/uid is equal to the previously "registered"
nfs_client_id.id, then locks associated with the old nfs_client_id
are immediately released. If the principal/uid is not equal, then
this is a rogue client and the request is returned in error. For
more discussion of crash recovery semantics, see the section on
"Crash Recovery".
It is possible for a retransmission of request to be received by
the server after the server has acted upon and responded to the
original client request. Therefore to mitigate effects of the
retransmission of the SETCLIENTID operation, the client and server
use a confirmation step. The server returns a confirmation
verifier that the client then sends to the server in the
SETCLIENTID_CONFIRM operation. Once the server receives the
confirmation from the client, the locking state for the client is
released.
In both cases, upon success, NFS4_OK is returned. To help reduce the
amount of data transferred on OPEN and LOCK, the server will also
return a unique 64-bit clientid value that is a shorthand reference
to the nfs_client_id values presented by the client. From this point
forward, the client will use the clientid to refer to itself.
The clientid assigned by the server should be chosen so that it will
not conflict with a clientid previously assigned by the server. This
applies across server restarts or reboots. When a clientid is
presented to a server and that clientid is not recognized, as would
happen after a server reboot, the server will reject the request with
the error NFS4ERR_STALE_CLIENTID. When this happens, the client must
obtain a new clientid by use of the SETCLIENTID operation and then
proceed to any other necessary recovery for the server reboot case
(See the section "Server Failure and Recovery").
The client must also employ the SETCLIENTID operation when it
receives a NFS4ERR_STALE_STATEID error using a stateid derived from
its current clientid, since this also indicates a server reboot which
has invalidated the existing clientid (see the next section
"nfs_lockowner and stateid Definition" for details).
8.1.2. Server Release of Clientid
If the server determines that the client holds no associated state
for its clientid, the server may choose to release the clientid. The
server may make this choice for an inactive client so that resources
are not consumed by those intermittently active clients. If the
client contacts the server after this release, the server must ensure
the client receives the appropriate error so that it will use the
SETCLIENTID/SETCLIENTID_CONFIRM sequence to establish a new identity.
It should be clear that the server must be very hesitant to release a
clientid since the resulting work on the client to recover from such
an event will be the same burden as if the server had failed and
restarted. Typically a server would not release a clientid unless
there had been no activity from that client for many minutes.
8.1.3. nfs_lockowner and stateid Definition
When requesting a lock, the client must present to the server the
clientid and an identifier for the owner of the requested lock.
These two fields are referred to as the nfs_lockowner and the
definition of those fields are:
o A clientid returned by the server as part of the client's use of
the SETCLIENTID operation.
o A variable length opaque array used to uniquely define the owner
of a lock managed by the client.
This may be a thread id, process id, or other unique value.
When the server grants the lock, it responds with a unique 64-bit
stateid. The stateid is used as a shorthand reference to the
nfs_lockowner, since the server will be maintaining the
correspondence between them.
The server is free to form the stateid in any manner that it chooses
as long as it is able to recognize invalid and out-of-date stateids.
This requirement includes those stateids generated by earlier
instances of the server. From this, the client can be properly
notified of a server restart. This notification will occur when the
client presents a stateid to the server from a previous
instantiation.
The server must be able to distinguish the following situations and
return the error as specified:
o The stateid was generated by an earlier server instance (i.e.
before a server reboot). The error NFS4ERR_STALE_STATEID should
be returned.
o The stateid was generated by the current server instance but the
stateid no longer designates the current locking state for the
lockowner-file pair in question (i.e. one or more locking
operations has occurred). The error NFS4ERR_OLD_STATEID should be
returned.
This error condition will only occur when the client issues a
locking request which changes a stateid while an I/O request that
uses that stateid is outstanding.
o The stateid was generated by the current server instance but the
stateid does not designate a locking state for any active
lockowner-file pair. The error NFS4ERR_BAD_STATEID should be
returned.
This error condition will occur when there has been a logic error
on the part of the client or server. This should not happen.
One mechanism that may be used to satisfy these requirements is for
the server to divide stateids into three fields:
o A server verifier which uniquely designates a particular server
instantiation.
o An index into a table of locking-state structures.
o A sequence value which is incremented for each stateid that is
associated with the same index into the locking-state table.
By matching the incoming stateid and its field values with the state
held at the server, the server is able to easily determine if a
stateid is valid for its current instantiation and state. If the
stateid is not valid, the appropriate error can be supplied to the
client.
8.1.4. Use of the stateid
All READ and WRITE operations contain a stateid. If the
nfs_lockowner performs a READ or WRITE on a range of bytes within a
locked range, the stateid (previously returned by the server) must be
used to indicate that the appropriate lock (record or share) is held.
If no state is established by the client, either record lock or share
lock, a stateid of all bits 0 is used. If no conflicting locks are
held on the file, the server may service the READ or WRITE operation.
If a conflict with an explicit lock occurs, an error is returned for
the operation (NFS4ERR_LOCKED). This allows "mandatory locking" to be
implemented.
A stateid of all bits 1 (one) allows READ operations to bypass record
locking checks at the server. However, WRITE operations with stateid
with bits all 1 (one) do not bypass record locking checks. File
locking checks are handled by the OPEN operation (see the section
"OPEN/CLOSE Operations").
An explicit lock may not be granted while a READ or WRITE operation
with conflicting implicit locking is being performed.
8.1.5. Sequencing of Lock Requests
Locking is different than most NFS operations as it requires "at-
most-one" semantics that are not provided by ONCRPC. ONCRPC over a
reliable transport is not sufficient because a sequence of locking
requests may span multiple TCP connections. In the face of
retransmission or reordering, lock or unlock requests must have a
well defined and consistent behavior. To accomplish this, each lock
request contains a sequence number that is a consecutively increasing
integer. Different nfs_lockowners have different sequences. The
server maintains the last sequence number (L) received and the
response that was returned.
Note that for requests that contain a sequence number, for each
nfs_lockowner, there should be no more than one outstanding request.
If a request with a previous sequence number (r < L) is received, it
is rejected with the return of error NFS4ERR_BAD_SEQID. Given a
properly-functioning client, the response to (r) must have been
received before the last request (L) was sent. If a duplicate of
last request (r == L) is received, the stored response is returned.
If a request beyond the next sequence (r == L + 2) is received, it is
rejected with the return of error NFS4ERR_BAD_SEQID. Sequence
history is reinitialized whenever the client verifier changes.
Since the sequence number is represented with an unsigned 32-bit
integer, the arithmetic involved with the sequence number is mod
2^32.
It is critical the server maintain the last response sent to the
client to provide a more reliable cache of duplicate non-idempotent
requests than that of the traditional cache described in [Juszczak].
The traditional duplicate request cache uses a least recently used
algorithm for removing unneeded requests. However, the last lock
request and response on a given nfs_lockowner must be cached as long
as the lock state exists on the server.
8.1.6. Recovery from Replayed Requests
As described above, the sequence number is per nfs_lockowner. As
long as the server maintains the last sequence number received and
follows the methods described above, there are no risks of a
Byzantine router re-sending old requests. The server need only
maintain the nfs_lockowner, sequence number state as long as there
are open files or closed files with locks outstanding.
LOCK, LOCKU, OPEN, OPEN_DOWNGRADE, and CLOSE each contain a sequence
number and therefore the risk of the replay of these operations
resulting in undesired effects is non-existent while the server
maintains the nfs_lockowner state.
8.1.7. Releasing nfs_lockowner State
When a particular nfs_lockowner no longer holds open or file locking
state at the server, the server may choose to release the sequence
number state associated with the nfs_lockowner. The server may make
this choice based on lease expiration, for the reclamation of server
memory, or other implementation specific details. In any event, the
server is able to do this safely only when the nfs_lockowner no
longer is being utilized by the client. The server may choose to
hold the nfs_lockowner state in the event that retransmitted requests
are received. However, the period to hold this state is
implementation specific.
In the case that a LOCK, LOCKU, OPEN_DOWNGRADE, or CLOSE is
retransmitted after the server has previously released the
nfs_lockowner state, the server will find that the nfs_lockowner has
no files open and an error will be returned to the client. If the
nfs_lockowner does have a file open, the stateid will not match and
again an error is returned to the client.
In the case that an OPEN is retransmitted and the nfs_lockowner is
being used for the first time or the nfs_lockowner state has been
previously released by the server, the use of the OPEN_CONFIRM
operation will prevent incorrect behavior. When the server observes
the use of the nfs_lockowner for the first time, it will direct the
client to perform the OPEN_CONFIRM for the corresponding OPEN. This
sequence establishes the use of an nfs_lockowner and associated
sequence number. See the section "OPEN_CONFIRM - Confirm Open" for
further details.
8.2. Lock Ranges
The protocol allows a lock owner to request a lock with one byte
range and then either upgrade or unlock a sub-range of the initial
lock. It is expected that this will be an uncommon type of request.
In any case, servers or server file systems may not be able to
support sub-range lock semantics. In the event that a server
receives a locking request that represents a sub-range of current
locking state for the lock owner, the server is allowed to return the
error NFS4ERR_LOCK_RANGE to signify that it does not support sub-
range lock operations. Therefore, the client should be prepared to
receive this error and, if appropriate, report the error to the
requesting application.
The client is discouraged from combining multiple independent locking
ranges that happen to be adjacent into a single request since the
server may not support sub-range requests and for reasons related to
the recovery of file locking state in the event of server failure.
As discussed in the section "Server Failure and Recovery" below, the
server may employ certain optimizations during recovery that work
effectively only when the client's behavior during lock recovery is
similar to the client's locking behavior prior to server failure.
8.3. Blocking Locks
Some clients require the support of blocking locks. The NFS version
4 protocol must not rely on a callback mechanism and therefore is
unable to notify a client when a previously denied lock has been
granted. Clients have no choice but to continually poll for the
lock. This presents a fairness problem. Two new lock types are
added, READW and WRITEW, and are used to indicate to the server that
the client is requesting a blocking lock. The server should maintain
an ordered list of pending blocking locks. When the conflicting lock
is released, the server may wait the lease period for the first
waiting client to re-request the lock. After the lease period
expires the next waiting client request is allowed the lock. Clients
are required to poll at an interval sufficiently small that it is
likely to acquire the lock in a timely manner. The server is not
required to maintain a list of pending blocked locks as it is used to
increase fairness and not correct operation. Because of the
unordered nature of crash recovery, storing of lock state to stable
storage would be required to guarantee ordered granting of blocking
locks.
Servers may also note the lock types and delay returning denial of
the request to allow extra time for a conflicting lock to be
released, allowing a successful return. In this way, clients can
avoid the burden of needlessly frequent polling for blocking locks.
The server should take care in the length of delay in the event the
client retransmits the request.
8.4. Lease Renewal
The purpose of a lease is to allow a server to remove stale locks
that are held by a client that has crashed or is otherwise
unreachable. It is not a mechanism for cache consistency and lease
renewals may not be denied if the lease interval has not expired.
The following events cause implicit renewal of all of the leases for
a given client (i.e. all those sharing a given clientid). Each of
these is a positive indication that the client is still active and
that the associated state held at the server, for the client, is
still valid.
o An OPEN with a valid clientid.
o Any operation made with a valid stateid (CLOSE, DELEGRETURN, LOCK,
LOCKU, OPEN, OPEN_CONFIRM, READ, RENEW, SETATTR, WRITE). This
does not include the special stateids of all bits 0 or all bits 1.
Note that if the client had restarted or rebooted, the client
would not be making these requests without issuing the
SETCLIENTID operation. The use of the SETCLIENTID operation
(possibly with the addition of the optional SETCLIENTID_CONFIRM
operation) notifies the server to drop the locking state
associated with the client.
If the server has rebooted, the stateids (NFS4ERR_STALE_STATEID
error) or the clientid (NFS4ERR_STALE_CLIENTID error) will not
be valid hence preventing spurious renewals.
This approach allows for low overhead lease renewal which scales
well. In the typical case no extra RPC calls are required for lease
renewal and in the worst case one RPC is required every lease period
(i.e. a RENEW operation). The number of locks held by the client is
not a factor since all state for the client is involved with the
lease renewal action.
Since all operations that create a new lease also renew existing
leases, the server must maintain a common lease expiration time for
all valid leases for a given client. This lease time can then be
easily updated upon implicit lease renewal actions.
8.5. Crash Recovery
The important requirement in crash recovery is that both the client
and the server know when the other has failed. Additionally, it is
required that a client sees a consistent view of data across server
restarts or reboots. All READ and WRITE operations that may have
been queued within the client or network buffers must wait until the
client has successfully recovered the locks protecting the READ and
WRITE operations.
8.5.1. Client Failure and Recovery
In the event that a client fails, the server may recover the client's
locks when the associated leases have expired. Conflicting locks
from another client may only be granted after this lease expiration.
If the client is able to restart or reinitialize within the lease
period the client may be forced to wait the remainder of the lease
period before obtaining new locks.
To minimize client delay upon restart, lock requests are associated
with an instance of the client by a client supplied verifier. This
verifier is part of the initial SETCLIENTID call made by the client.
The server returns a clientid as a result of the SETCLIENTID
operation. The client then confirms the use of the verifier with
SETCLIENTID_CONFIRM. The clientid in combination with an opaque
owner field is then used by the client to identify the lock owner for
OPEN. This chain of associations is then used to identify all locks
for a particular client.
Since the verifier will be changed by the client upon each
initialization, the server can compare a new verifier to the verifier
associated with currently held locks and determine that they do not
match. This signifies the client's new instantiation and subsequent
loss of locking state. As a result, the server is free to release
all locks held which are associated with the old clientid which was
derived from the old verifier.
For secure environments, a change in the verifier must only cause the
release of locks associated with the authenticated requester. This
is required to prevent a rogue entity from freeing otherwise valid
locks.
Note that the verifier must have the same uniqueness properties of
the verifier for the COMMIT operation.
8.5.2. Server Failure and Recovery
If the server loses locking state (usually as a result of a restart
or reboot), it must allow clients time to discover this fact and re-
establish the lost locking state. The client must be able to re-
establish the locking state without having the server deny valid
requests because the server has granted conflicting access to another
client. Likewise, if there is the possibility that clients have not
yet re-established their locking state for a file, the server must
disallow READ and WRITE operations for that file. The duration of
this recovery period is equal to the duration of the lease period.
A client can determine that server failure (and thus loss of locking
state) has occurred, when it receives one of two errors. The
NFS4ERR_STALE_STATEID error indicates a stateid invalidated by a
reboot or restart. The NFS4ERR_STALE_CLIENTID error indicates a
clientid invalidated by reboot or restart. When either of these are
received, the client must establish a new clientid (See the section
"Client ID") and re-establish the locking state as discussed below.
The period of special handling of locking and READs and WRITEs, equal
in duration to the lease period, is referred to as the "grace
period". During the grace period, clients recover locks and the
associated state by reclaim-type locking requests (i.e. LOCK requests
with reclaim set to true and OPEN operations with a claim type of
CLAIM_PREVIOUS). During the grace period, the server must reject
READ and WRITE operations and non-reclaim locking requests (i.e.
other LOCK and OPEN operations) with an error of NFS4ERR_GRACE.
If the server can reliably determine that granting a non-reclaim
request will not conflict with reclamation of locks by other clients,
the NFS4ERR_GRACE error does not have to be returned and the non-
reclaim client request can be serviced. For the server to be able to
service READ and WRITE operations during the grace period, it must
again be able to guarantee that no possible conflict could arise
between an impending reclaim locking request and the READ or WRITE
operation. If the server is unable to offer that guarantee, the
NFS4ERR_GRACE error must be returned to the client.
For a server to provide simple, valid handling during the grace
period, the easiest method is to simply reject all non-reclaim
locking requests and READ and WRITE operations by returning the
NFS4ERR_GRACE error. However, a server may keep information about
granted locks in stable storage. With this information, the server
could determine if a regular lock or READ or WRITE operation can be
safely processed.
For example, if a count of locks on a given file is available in
stable storage, the server can track reclaimed locks for the file and
when all reclaims have been processed, non-reclaim locking requests
may be processed. This way the server can ensure that non-reclaim
locking requests will not conflict with potential reclaim requests.
With respect to I/O requests, if the server is able to determine that
there are no outstanding reclaim requests for a file by information
from stable storage or another similar mechanism, the processing of
I/O requests could proceed normally for the file.
To reiterate, for a server that allows non-reclaim lock and I/O
requests to be processed during the grace period, it MUST determine
that no lock subsequently reclaimed will be rejected and that no lock
subsequently reclaimed would have prevented any I/O operation
processed during the grace period.
Clients should be prepared for the return of NFS4ERR_GRACE errors for
non-reclaim lock and I/O requests. In this case the client should
employ a retry mechanism for the request. A delay (on the order of
several seconds) between retries should be used to avoid overwhelming
the server. Further discussion of the general is included in
[Floyd]. The client must account for the server that is able to
perform I/O and non-reclaim locking requests within the grace period
as well as those that can not do so.
A reclaim-type locking request outside the server's grace period can
only succeed if the server can guarantee that no conflicting lock or
I/O request has been granted since reboot or restart.
8.5.3. Network Partitions and Recovery
If the duration of a network partition is greater than the lease
period provided by the server, the server will have not received a
lease renewal from the client. If this occurs, the server may free
all locks held for the client. As a result, all stateids held by the
client will become invalid or stale. Once the client is able to
reach the server after such a network partition, all I/O submitted by
the client with the now invalid stateids will fail with the server
returning the error NFS4ERR_EXPIRED. Once this error is received,
the client will suitably notify the application that held the lock.
As a courtesy to the client or as an optimization, the server may
continue to hold locks on behalf of a client for which recent
communication has extended beyond the lease period. If the server
receives a lock or I/O request that conflicts with one of these
courtesy locks, the server must free the courtesy lock and grant the
new request.
If the server continues to hold locks beyond the expiration of a
client's lease, the server MUST employ a method of recording this
fact in its stable storage. Conflicting locks requests from another
client may be serviced after the lease expiration. There are various
scenarios involving server failure after such an event that require
the storage of these lease expirations or network partitions. One
scenario is as follows:
A client holds a lock at the server and encounters a network
partition and is unable to renew the associated lease. A
second client obtains a conflicting lock and then frees the
lock. After the unlock request by the second client, the
server reboots or reinitializes. Once the server recovers, the
network partition heals and the original client attempts to
reclaim the original lock.
In this scenario and without any state information, the server will
allow the reclaim and the client will be in an inconsistent state
because the server or the client has no knowledge of the conflicting
lock.
The server may choose to store this lease expiration or network
partitioning state in a way that will only identify the client as a
whole. Note that this may potentially lead to lock reclaims being
denied unnecessarily because of a mix of conflicting and non-
conflicting locks. The server may also choose to store information
about each lock that has an expired lease with an associated
conflicting lock. The choice of the amount and type of state
information that is stored is left to the implementor. In any case,
the server must have enough state information to enable correct
recovery from multiple partitions and multiple server failures.
8.6. Recovery from a Lock Request Timeout or Abort
In the event a lock request times out, a client may decide to not
retry the request. The client may also abort the request when the
process for which it was issued is terminated (e.g. in UNIX due to a
signal. It is possible though that the server received the request
and acted upon it. This would change the state on the server without
the client being aware of the change. It is paramount that the
client re-synchronize state with server before it attempts any other
operation that takes a seqid and/or a stateid with the same
nfs_lockowner. This is straightforward to do without a special re-
synchronize operation.
Since the server maintains the last lock request and response
received on the nfs_lockowner, for each nfs_lockowner, the client
should cache the last lock request it sent such that the lock request
did not receive a response. From this, the next time the client does
a lock operation for the nfs_lockowner, it can send the cached
request, if there is one, and if the request was one that established
state (e.g. a LOCK or OPEN operation) the client can follow up with a
request to remove the state (e.g. a LOCKU or CLOSE operation). With
this approach, the sequencing and stateid information on the client
and server for the given nfs_lockowner will re-synchronize and in
turn the lock state will re-synchronize.
8.7. Server Revocation of Locks
At any point, the server can revoke locks held by a client and the
client must be prepared for this event. When the client detects that
its locks have been or may have been revoked, the client is
responsible for validating the state information between itself and
the server. Validating locking state for the client means that it
must verify or reclaim state for each lock currently held.
The first instance of lock revocation is upon server reboot or re-
initialization. In this instance the client will receive an error
(NFS4ERR_STALE_STATEID or NFS4ERR_STALE_CLIENTID) and the client will
proceed with normal crash recovery as described in the previous
section.
The second lock revocation event is the inability to renew the lease
period. While this is considered a rare or unusual event, the client
must be prepared to recover. Both the server and client will be able
to detect the failure to renew the lease and are capable of
recovering without data corruption. For the server, it tracks the
last renewal event serviced for the client and knows when the lease
will expire. Similarly, the client must track operations which will
renew the lease period. Using the time that each such request was
sent and the time that the corresponding reply was received, the
client should bound the time that the corresponding renewal could
have occurred on the server and thus determine if it is possible that
a lease period expiration could have occurred.
The third lock revocation event can occur as a result of
administrative intervention within the lease period. While this is
considered a rare event, it is possible that the server's
administrator has decided to release or revoke a particular lock held
by the client. As a result of revocation, the client will receive an
error of NFS4ERR_EXPIRED and the error is received within the lease
period for the lock. In this instance the client may assume that
only the nfs_lockowner's locks have been lost. The client notifies
the lock holder appropriately. The client may not assume the lease
period has been renewed as a result of failed operation.
When the client determines the lease period may have expired, the
client must mark all locks held for the associated lease as
"unvalidated". This means the client has been unable to re-establish
or confirm the appropriate lock state with the server. As described
in the previous section on crash recovery, there are scenarios in
which the server may grant conflicting locks after the lease period
has expired for a client. When it is possible that the lease period
has expired, the client must validate each lock currently held to
ensure that a conflicting lock has not been granted. The client may
accomplish this task by issuing an I/O request, either a pending I/O
or a zero-length read, specifying the stateid associated with the
lock in question. If the response to the request is success, the
client has validated all of the locks governed by that stateid and
re-established the appropriate state between itself and the server.
If the I/O request is not successful, then one or more of the locks
associated with the stateid was revoked by the server and the client
must notify the owner.
8.8. Share Reservations
A share reservation is a mechanism to control access to a file. It
is a separate and independent mechanism from record locking. When a
client opens a file, it issues an OPEN operation to the server
specifying the type of access required (READ, WRITE, or BOTH) and the
type of access to deny others (deny NONE, READ, WRITE, or BOTH). If
the OPEN fails the client will fail the application's open request.
Pseudo-code definition of the semantics:
if ((request.access & file_state.deny)) ||
(request.deny & file_state.access))
return (NFS4ERR_DENIED)
The constants used for the OPEN and OPEN_DOWNGRADE operations for the
access and deny fields are as follows:
const OPEN4_SHARE_ACCESS_READ = 0x00000001;
const OPEN4_SHARE_ACCESS_WRITE = 0x00000002;
const OPEN4_SHARE_ACCESS_BOTH = 0x00000003;
const OPEN4_SHARE_DENY_NONE = 0x00000000;
const OPEN4_SHARE_DENY_READ = 0x00000001;
const OPEN4_SHARE_DENY_WRITE = 0x00000002;
const OPEN4_SHARE_DENY_BOTH = 0x00000003;
8.9. OPEN/CLOSE Operations
To provide correct share semantics, a client MUST use the OPEN
operation to obtain the initial filehandle and indicate the desired
access and what if any access to deny. Even if the client intends to
use a stateid of all 0's or all 1's, it must still obtain the
filehandle for the regular file with the OPEN operation so the
appropriate share semantics can be applied. For clients that do not
have a deny mode built into their open programming interfaces, deny
equal to NONE should be used.
The OPEN operation with the CREATE flag, also subsumes the CREATE
operation for regular files as used in previous versions of the NFS
protocol. This allows a create with a share to be done atomically.
The CLOSE operation removes all share locks held by the nfs_lockowner
on that file. If record locks are held, the client SHOULD release
all locks before issuing a CLOSE. The server MAY free all
outstanding locks on CLOSE but some servers may not support the CLOSE
of a file that still has record locks held. The server MUST return
failure if any locks would exist after the CLOSE.
The LOOKUP operation will return a filehandle without establishing
any lock state on the server. Without a valid stateid, the server
will assume the client has the least access. For example, a file
opened with deny READ/WRITE cannot be accessed using a filehandle
obtained through LOOKUP because it would not have a valid stateid
(i.e. using a stateid of all bits 0 or all bits 1).
8.10. Open Upgrade and Downgrade
When an OPEN is done for a file and the lockowner for which the open
is being done already has the file open, the result is to upgrade the
open file status maintained on the server to include the access and
deny bits specified by the new OPEN as well as those for the existing
OPEN. The result is that there is one open file, as far as the
protocol is concerned, and it includes the union of the access and
deny bits for all of the OPEN requests completed. Only a single
CLOSE will be done to reset the effects of both OPEN's. Note that
the client, when issuing the OPEN, may not know that the same file is
in fact being opened. The above only applies if both OPEN's result
in the OPEN'ed object being designated by the same filehandle.
When the server chooses to export multiple filehandles corresponding
to the same file object and returns different filehandles on two
different OPEN's of the same file object, the server MUST NOT "OR"
together the access and deny bits and coalesce the two open files.
Instead the server must maintain separate OPEN's with separate
stateid's and will require separate CLOSE's to free them.
When multiple open files on the client are merged into a single open
file object on the server, the close of one of the open files (on the
client) may necessitate change of the access and deny status of the
open file on the server. This is because the union of the access and
deny bits for the remaining open's may be smaller (i.e. a proper
subset) than previously. The OPEN_DOWNGRADE operation is used to
make the necessary change and the client should use it to update the
server so that share reservation requests by other clients are
handled properly.
8.11. Short and Long Leases
When determining the time period for the server lease, the usual
lease tradeoffs apply. Short leases are good for fast server
recovery at a cost of increased RENEW or READ (with zero length)
requests. Longer leases are certainly kinder and gentler to large
internet servers trying to handle very large numbers of clients. The
number of RENEW requests drop in proportion to the lease time. The
disadvantages of long leases are slower recovery after server failure
(server must wait for leases to expire and grace period before
granting new lock requests) and increased file contention (if client
fails to transmit an unlock request then server must wait for lease
expiration before granting new locks).
Long leases are usable if the server is able to store lease state in
non-volatile memory. Upon recovery, the server can reconstruct the
lease state from its non-volatile memory and continue operation with
its clients and therefore long leases are not an issue.
8.12. Clocks and Calculating Lease Expiration
To avoid the need for synchronized clocks, lease times are granted by
the server as a time delta. However, there is a requirement that the
client and server clocks do not drift excessively over the duration
of the lock. There is also the issue of propagation delay across the
network which could easily be several hundred milliseconds as well as
the possibility that requests will be lost and need to be
retransmitted.
To take propagation delay into account, the client should subtract it
from lease times (e.g. if the client estimates the one-way
propagation delay as 200 msec, then it can assume that the lease is
already 200 msec old when it gets it). In addition, it will take
another 200 msec to get a response back to the server. So the client
must send a lock renewal or write data back to the server 400 msec
before the lease would expire.
8.13. Migration, Replication and State
When responsibility for handling a given file system is transferred
to a new server (migration) or the client chooses to use an alternate
server (e.g. in response to server unresponsiveness) in the context
of file system replication, the appropriate handling of state shared
between the client and server (i.e. locks, leases, stateid's, and
clientid's) is as described below. The handling differs between
migration and replication. For related discussion of file server
state and recover of such see the sections under "File Locking and
Share Reservations"
8.13.1. Migration and State
In the case of migration, the servers involved in the migration of a
file system SHOULD transfer all server state from the original to the
new server. This must be done in a way that is transparent to the
client. This state transfer will ease the client's transition when a
file system migration occurs. If the servers are successful in
transferring all state, the client will continue to use stateid's
assigned by the original server. Therefore the new server must
recognize these stateid's as valid. This holds true for the clientid
as well. Since responsibility for an entire file system is
transferred with a migration event, there is no possibility that
conflicts will arise on the new server as a result of the transfer of
locks.
As part of the transfer of information between servers, leases would
be transferred as well. The leases being transferred to the new
server will typically have a different expiration time from those for
the same client, previously on the new server. To maintain the
property that all leases on a given server for a given client expire
at the same time, the server should advance the expiration time to
the later of the leases being transferred or the leases already
present. This allows the client to maintain lease renewal of both
classes without special effort.
The servers may choose not to transfer the state information upon
migration. However, this choice is discouraged. In this case, when
the client presents state information from the original server, the
client must be prepared to receive either NFS4ERR_STALE_CLIENTID or
NFS4ERR_STALE_STATEID from the new server. The client should then
recover its state information as it normally would in response to a
server failure. The new server must take care to allow for the