recovery of state information as it would in the event of server
restart.
8.13.2. Replication and State
Since client switch-over in the case of replication is not under
server control, the handling of state is different. In this case,
leases, stateid's and clientid's do not have validity across a
transition from one server to another. The client must re-establish
its locks on the new server. This can be compared to the re-
establishment of locks by means of reclaim-type requests after a
server reboot. The difference is that the server has no provision to
distinguish requests reclaiming locks from those obtaining new locks
or to defer the latter. Thus, a client re-establishing a lock on the
new server (by means of a LOCK or OPEN request), may have the
requests denied due to a conflicting lock. Since replication is
intended for read-only use of filesystems, such denial of locks
should not pose large difficulties in practice. When an attempt to
re-establish a lock on a new server is denied, the client should
treat the situation as if his original lock had been revoked.
8.13.3. Notification of Migrated Lease
In the case of lease renewal, the client may not be submitting
requests for a file system that has been migrated to another server.
This can occur because of the implicit lease renewal mechanism. The
client renews leases for all file systems when submitting a request
to any one file system at the server.
In order for the client to schedule renewal of leases that may have
been relocated to the new server, the client must find out about
lease relocation before those leases expire. To accomplish this, all
operations which implicitly renew leases for a client (i.e. OPEN,
CLOSE, READ, WRITE, RENEW, LOCK, LOCKT, LOCKU), will return the error
NFS4ERR_LEASE_MOVED if responsibility for any of the leases to be
renewed has been transferred to a new server. This condition will
continue until the client receives an NFS4ERR_MOVED error and the
server receives the subsequent GETATTR(fs_locations) for an access to
each file system for which a lease has been moved to a new server.
When a client receives an NFS4ERR_LEASE_MOVED error, it should
perform some operation, such as a RENEW, on each file system
associated with the server in question. When the client receives an
NFS4ERR_MOVED error, the client can follow the normal process to
obtain the new server information (through the fs_locations
attribute) and perform renewal of those leases on the new server. If
the server has not had state transferred to it transparently, it will
receive either NFS4ERR_STALE_CLIENTID or NFS4ERR_STALE_STATEID from
the new server, as described above, and can then recover state
information as it does in the event of server failure.
9. Client-Side Caching
Client-side caching of data, of file attributes, and of file names is
essential to providing good performance with the NFS protocol.
Providing distributed cache coherence is a difficult problem and
previous versions of the NFS protocol have not attempted it.
Instead, several NFS client implementation techniques have been used
to reduce the problems that a lack of coherence poses for users.
These techniques have not been clearly defined by earlier protocol
specifications and it is often unclear what is valid or invalid
client behavior.
The NFS version 4 protocol uses many techniques similar to those that
have been used in previous protocol versions. The NFS version 4
protocol does not provide distributed cache coherence. However, it
defines a more limited set of caching guarantees to allow locks and
share reservations to be used without destructive interference from
client side caching.
In addition, the NFS version 4 protocol introduces a delegation
mechanism which allows many decisions normally made by the server to
be made locally by clients. This mechanism provides efficient
support of the common cases where sharing is infrequent or where
sharing is read-only.
9.1. Performance Challenges for Client-Side Caching
Caching techniques used in previous versions of the NFS protocol have
been successful in providing good performance. However, several
scalability challenges can arise when those techniques are used with
very large numbers of clients. This is particularly true when
clients are geographically distributed which classically increases
the latency for cache revalidation requests.
The previous versions of the NFS protocol repeat their file data
cache validation requests at the time the file is opened. This
behavior can have serious performance drawbacks. A common case is
one in which a file is only accessed by a single client. Therefore,
sharing is infrequent.
In this case, repeated reference to the server to find that no
conflicts exist is expensive. A better option with regards to
performance is to allow a client that repeatedly opens a file to do
so without reference to the server. This is done until potentially
conflicting operations from another client actually occur.
A similar situation arises in connection with file locking. Sending
file lock and unlock requests to the server as well as the read and
write requests necessary to make data caching consistent with the
locking semantics (see the section "Data Caching and File Locking")
can severely limit performance. When locking is used to provide
protection against infrequent conflicts, a large penalty is incurred.
This penalty may discourage the use of file locking by applications.
The NFS version 4 protocol provides more aggressive caching
strategies with the following design goals:
o Compatibility with a large range of server semantics.
o Provide the same caching benefits as previous versions of the NFS
protocol when unable to provide the more aggressive model.
o Requirements for aggressive caching are organized so that a large
portion of the benefit can be obtained even when not all of the
requirements can be met.
The appropriate requirements for the server are discussed in later
sections in which specific forms of caching are covered. (see the
section "Open Delegation").
9.2. Delegation and Callbacks
Recallable delegation of server responsibilities for a file to a
client improves performance by avoiding repeated requests to the
server in the absence of inter-client conflict. With the use of a
"callback" RPC from server to client, a server recalls delegated
responsibilities when another client engages in sharing of a
delegated file.
A delegation is passed from the server to the client, specifying the
object of the delegation and the type of delegation. There are
different types of delegations but each type contains a stateid to be
used to represent the delegation when performing operations that
depend on the delegation. This stateid is similar to those
associated with locks and share reservations but differs in that the
stateid for a delegation is associated with a clientid and may be
used on behalf of all the nfs_lockowners for the given client. A
delegation is made to the client as a whole and not to any specific
process or thread of control within it.
Because callback RPCs may not work in all environments (due to
firewalls, for example), correct protocol operation does not depend
on them. Preliminary testing of callback functionality by means of a
CB_NULL procedure determines whether callbacks can be supported. The
CB_NULL procedure checks the continuity of the callback path. A
server makes a preliminary assessment of callback availability to a
given client and avoids delegating responsibilities until it has
determined that callbacks are supported. Because the granting of a
delegation is always conditional upon the absence of conflicting
access, clients must not assume that a delegation will be granted and
they must always be prepared for OPENs to be processed without any
delegations being granted.
Once granted, a delegation behaves in most ways like a lock. There
is an associated lease that is subject to renewal together with all
of the other leases held by that client.
Unlike locks, an operation by a second client to a delegated file
will cause the server to recall a delegation through a callback.
On recall, the client holding the delegation must flush modified
state (such as modified data) to the server and return the
delegation. The conflicting request will not receive a response
until the recall is complete. The recall is considered complete when
the client returns the delegation or the server times out on the
recall and revokes the delegation as a result of the timeout.
Following the resolution of the recall, the server has the
information necessary to grant or deny the second client's request.
At the time the client receives a delegation recall, it may have
substantial state that needs to be flushed to the server. Therefore,
the server should allow sufficient time for the delegation to be
returned since it may involve numerous RPCs to the server. If the
server is able to determine that the client is diligently flushing
state to the server as a result of the recall, the server may extend
the usual time allowed for a recall. However, the time allowed for
recall completion should not be unbounded.
An example of this is when responsibility to mediate opens on a given
file is delegated to a client (see the section "Open Delegation").
The server will not know what opens are in effect on the client.
Without this knowledge the server will be unable to determine if the
access and deny state for the file allows any particular open until
the delegation for the file has been returned.
A client failure or a network partition can result in failure to
respond to a recall callback. In this case, the server will revoke
the delegation which in turn will render useless any modified state
still on the client.
9.2.1. Delegation Recovery
There are three situations that delegation recovery must deal with:
o Client reboot or restart
o Server reboot or restart
o Network partition (full or callback-only)
In the event the client reboots or restarts, the failure to renew
leases will result in the revocation of record locks and share
reservations. Delegations, however, may be treated a bit
differently.
There will be situations in which delegations will need to be
reestablished after a client reboots or restarts. The reason for
this is the client may have file data stored locally and this data
was associated with the previously held delegations. The client will
need to reestablish the appropriate file state on the server.
To allow for this type of client recovery, the server may extend the
period for delegation recovery beyond the typical lease expiration
period. This implies that requests from other clients that conflict
with these delegations will need to wait. Because the normal recall
process may require significant time for the client to flush changed
state to the server, other clients need be prepared for delays that
occur because of a conflicting delegation. This longer interval
would increase the window for clients to reboot and consult stable
storage so that the delegations can be reclaimed. For open
delegations, such delegations are reclaimed using OPEN with a claim
type of CLAIM_DELEGATE_PREV. (see the sections on "Data Caching and
Revocation" and "Operation 18: OPEN" for discussion of open
delegation and the details of OPEN respectively).
When the server reboots or restarts, delegations are reclaimed (using
the OPEN operation with CLAIM_DELEGATE_PREV) in a similar fashion to
record locks and share reservations. However, there is a slight
semantic difference. In the normal case if the server decides that a
delegation should not be granted, it performs the requested action
(e.g. OPEN) without granting any delegation. For reclaim, the server
grants the delegation but a special designation is applied so that
the client treats the delegation as having been granted but recalled
by the server. Because of this, the client has the duty to write all
modified state to the server and then return the delegation. This
process of handling delegation reclaim reconciles three principles of
the NFS Version 4 protocol:
o Upon reclaim, a client reporting resources assigned to it by an
earlier server instance must be granted those resources.
o The server has unquestionable authority to determine whether
delegations are to be granted and, once granted, whether they are
to be continued.
o The use of callbacks is not to be depended upon until the client
has proven its ability to receive them.
When a network partition occurs, delegations are subject to freeing
by the server when the lease renewal period expires. This is similar
to the behavior for locks and share reservations. For delegations,
however, the server may extend the period in which conflicting
requests are held off. Eventually the occurrence of a conflicting
request from another client will cause revocation of the delegation.
A loss of the callback path (e.g. by later network configuration
change) will have the same effect. A recall request will fail and
revocation of the delegation will result.
A client normally finds out about revocation of a delegation when it
uses a stateid associated with a delegation and receives the error
NFS4ERR_EXPIRED. It also may find out about delegation revocation
after a client reboot when it attempts to reclaim a delegation and
receives that same error. Note that in the case of a revoked write
open delegation, there are issues because data may have been modified
by the client whose delegation is revoked and separately by other
clients. See the section "Revocation Recovery for Write Open
Delegation" for a discussion of such issues. Note also that when
delegations are revoked, information about the revoked delegation
will be written by the server to stable storage (as described in the
section "Crash Recovery"). This is done to deal with the case in
which a server reboots after revoking a delegation but before the
client holding the revoked delegation is notified about the
revocation.
9.3. Data Caching
When applications share access to a set of files, they need to be
implemented so as to take account of the possibility of conflicting
access by another application. This is true whether the applications
in question execute on different clients or reside on the same
client.
Share reservations and record locks are the facilities the NFS
version 4 protocol provides to allow applications to coordinate
access by providing mutual exclusion facilities. The NFS version 4
protocol's data caching must be implemented such that it does not
invalidate the assumptions that those using these facilities depend
upon.
9.3.1. Data Caching and OPENs
In order to avoid invalidating the sharing assumptions that
applications rely on, NFS version 4 clients should not provide cached
data to applications or modify it on behalf of an application when it
would not be valid to obtain or modify that same data via a READ or
WRITE operation.
Furthermore, in the absence of open delegation (see the section "Open
Delegation") two additional rules apply. Note that these rules are
obeyed in practice by many NFS version 2 and version 3 clients.
o First, cached data present on a client must be revalidated after
doing an OPEN. This is to ensure that the data for the OPENed
file is still correctly reflected in the client's cache. This
validation must be done at least when the client's OPEN operation
includes DENY=WRITE or BOTH thus terminating a period in which
other clients may have had the opportunity to open the file with
WRITE access. Clients may choose to do the revalidation more
often (i.e. at OPENs specifying DENY=NONE) to parallel the NFS
version 3 protocol's practice for the benefit of users assuming
this degree of cache revalidation.
o Second, modified data must be flushed to the server before closing
a file OPENed for write. This is complementary to the first rule.
If the data is not flushed at CLOSE, the revalidation done after
client OPENs as file is unable to achieve its purpose. The other
aspect to flushing the data before close is that the data must be
committed to stable storage, at the server, before the CLOSE
operation is requested by the client. In the case of a server
reboot or restart and a CLOSEd file, it may not be possible to
retransmit the data to be written to the file. Hence, this
requirement.
9.3.2. Data Caching and File Locking
For those applications that choose to use file locking instead of
share reservations to exclude inconsistent file access, there is an
analogous set of constraints that apply to client side data caching.
These rules are effective only if the file locking is used in a way
that matches in an equivalent way the actual READ and WRITE
operations executed. This is as opposed to file locking that is
based on pure convention. For example, it is possible to manipulate
a two-megabyte file by dividing the file into two one-megabyte
regions and protecting access to the two regions by file locks on
bytes zero and one. A lock for write on byte zero of the file would
represent the right to do READ and WRITE operations on the first
region. A lock for write on byte one of the file would represent the
right to do READ and WRITE operations on the second region. As long
as all applications manipulating the file obey this convention, they
will work on a local file system. However, they may not work with
the NFS version 4 protocol unless clients refrain from data caching.
The rules for data caching in the file locking environment are:
o First, when a client obtains a file lock for a particular region,
the data cache corresponding to that region (if any cache data
exists) must be revalidated. If the change attribute indicates
that the file may have been updated since the cached data was
obtained, the client must flush or invalidate the cached data for
the newly locked region. A client might choose to invalidate all
of non-modified cached data that it has for the file but the only
requirement for correct operation is to invalidate all of the data
in the newly locked region.
o Second, before releasing a write lock for a region, all modified
data for that region must be flushed to the server. The modified
data must also be written to stable storage.
Note that flushing data to the server and the invalidation of cached
data must reflect the actual byte ranges locked or unlocked.
Rounding these up or down to reflect client cache block boundaries
will cause problems if not carefully done. For example, writing a
modified block when only half of that block is within an area being
unlocked may cause invalid modification to the region outside the
unlocked area. This, in turn, may be part of a region locked by
another client. Clients can avoid this situation by synchronously
performing portions of write operations that overlap that portion
(initial or final) that is not a full block. Similarly, invalidating
a locked area which is not an integral number of full buffer blocks
would require the client to read one or two partial blocks from the
server if the revalidation procedure shows that the data which the
client possesses may not be valid.
The data that is written to the server as a pre-requisite to the
unlocking of a region must be written, at the server, to stable
storage. The client may accomplish this either with synchronous
writes or by following asynchronous writes with a COMMIT operation.
This is required because retransmission of the modified data after a
server reboot might conflict with a lock held by another client.
A client implementation may choose to accommodate applications which
use record locking in non-standard ways (e.g. using a record lock as
a global semaphore) by flushing to the server more data upon an LOCKU
than is covered by the locked range. This may include modified data
within files other than the one for which the unlocks are being done.
In such cases, the client must not interfere with applications whose
READs and WRITEs are being done only within the bounds of record
locks which the application holds. For example, an application locks
a single byte of a file and proceeds to write that single byte. A
client that chose to handle a LOCKU by flushing all modified data to
the server could validly write that single byte in response to an
unrelated unlock. However, it would not be valid to write the entire
block in which that single written byte was located since it includes
an area that is not locked and might be locked by another client.
Client implementations can avoid this problem by dividing files with
modified data into those for which all modifications are done to
areas covered by an appropriate record lock and those for which there
are modifications not covered by a record lock. Any writes done for
the former class of files must not include areas not locked and thus
not modified on the client.
9.3.3. Data Caching and Mandatory File Locking
Client side data caching needs to respect mandatory file locking when
it is in effect. The presence of mandatory file locking for a given
file is indicated in the result flags for an OPEN. When mandatory
locking is in effect for a file, the client must check for an
appropriate file lock for data being read or written. If a lock
exists for the range being read or written, the client may satisfy
the request using the client's validated cache. If an appropriate
file lock is not held for the range of the read or write, the read or
write request must not be satisfied by the client's cache and the
request must be sent to the server for processing. When a read or
write request partially overlaps a locked region, the request should
be subdivided into multiple pieces with each region (locked or not)
treated appropriately.
9.3.4. Data Caching and File Identity
When clients cache data, the file data needs to organized according
to the file system object to which the data belongs. For NFS version
3 clients, the typical practice has been to assume for the purpose of
caching that distinct filehandles represent distinct file system
objects. The client then has the choice to organize and maintain the
data cache on this basis.
In the NFS version 4 protocol, there is now the possibility to have
significant deviations from a "one filehandle per object" model
because a filehandle may be constructed on the basis of the object's
pathname. Therefore, clients need a reliable method to determine if
two filehandles designate the same file system object. If clients
were simply to assume that all distinct filehandles denote distinct
objects and proceed to do data caching on this basis, caching
inconsistencies would arise between the distinct client side objects
which mapped to the same server side object.
By providing a method to differentiate filehandles, the NFS version 4
protocol alleviates a potential functional regression in comparison
with the NFS version 3 protocol. Without this method, caching
inconsistencies within the same client could occur and this has not
been present in previous versions of the NFS protocol. Note that it
is possible to have such inconsistencies with applications executing
on multiple clients but that is not the issue being addressed here.
For the purposes of data caching, the following steps allow an NFS
version 4 client to determine whether two distinct filehandles denote
the same server side object:
o If GETATTR directed to two filehandles have different values of
the fsid attribute, then the filehandles represent distinct
objects.
o If GETATTR for any file with an fsid that matches the fsid of the
two filehandles in question returns a unique_handles attribute
with a value of TRUE, then the two objects are distinct.
o If GETATTR directed to the two filehandles does not return the
fileid attribute for one or both of the handles, then the it
cannot be determined whether the two objects are the same.
Therefore, operations which depend on that knowledge (e.g. client
side data caching) cannot be done reliably.
o If GETATTR directed to the two filehandles returns different
values for the fileid attribute, then they are distinct objects.
o Otherwise they are the same object.
9.4. Open Delegation
When a file is being OPENed, the server may delegate further handling
of opens and closes for that file to the opening client. Any such
delegation is recallable, since the circumstances that allowed for
the delegation are subject to change. In particular, the server may
receive a conflicting OPEN from another client, the server must
recall the delegation before deciding whether the OPEN from the other
client may be granted. Making a delegation is up to the server and
clients should not assume that any particular OPEN either will or
will not result in an open delegation. The following is a typical
set of conditions that servers might use in deciding whether OPEN
should be delegated:
o The client must be able to respond to the server's callback
requests. The server will use the CB_NULL procedure for a test of
callback ability.
o The client must have responded properly to previous recalls.
o There must be no current open conflicting with the requested
delegation.
o There should be no current delegation that conflicts with the
delegation being requested.
o The probability of future conflicting open requests should be low
based on the recent history of the file.
o The existence of any server-specific semantics of OPEN/CLOSE that
would make the required handling incompatible with the prescribed
handling that the delegated client would apply (see below).
There are two types of open delegations, read and write. A read open
delegation allows a client to handle, on its own, requests to open a
file for reading that do not deny read access to others. Multiple
read open delegations may be outstanding simultaneously and do not
conflict. A write open delegation allows the client to handle, on
its own, all opens. Only one write open delegation may exist for a
given file at a given time and it is inconsistent with any read open
delegations.
When a client has a read open delegation, it may not make any changes
to the contents or attributes of the file but it is assured that no
other client may do so. When a client has a write open delegation,
it may modify the file data since no other client will be accessing
the file's data. The client holding a write delegation may only
affect file attributes which are intimately connected with the file
data: object_size, time_modify, change.
When a client has an open delegation, it does not send OPENs or
CLOSEs to the server but updates the appropriate status internally.
For a read open delegation, opens that cannot be handled locally
(opens for write or that deny read access) must be sent to the
server.
When an open delegation is made, the response to the OPEN contains an
open delegation structure which specifies the following:
o the type of delegation (read or write)
o space limitation information to control flushing of data on close
(write open delegation only, see the section "Open Delegation and
Data Caching")
o an nfsace4 specifying read and write permissions
o a stateid to represent the delegation for READ and WRITE
The stateid is separate and distinct from the stateid for the OPEN
proper. The standard stateid, unlike the delegation stateid, is
associated with a particular nfs_lockowner and will continue to be
valid after the delegation is recalled and the file remains open.
When a request internal to the client is made to open a file and open
delegation is in effect, it will be accepted or rejected solely on
the basis of the following conditions. Any requirement for other
checks to be made by the delegate should result in open delegation
being denied so that the checks can be made by the server itself.
o The access and deny bits for the request and the file as described
in the section "Share Reservations".
o The read and write permissions as determined below.
The nfsace4 passed with delegation can be used to avoid frequent
ACCESS calls. The permission check should be as follows:
o If the nfsace4 indicates that the open may be done, then it should
be granted without reference to the server.
o If the nfsace4 indicates that the open may not be done, then an
ACCESS request must be sent to the server to obtain the definitive
answer.
The server may return an nfsace4 that is more restrictive than the
actual ACL of the file. This includes an nfsace4 that specifies
denial of all access. Note that some common practices such as
mapping the traditional user "root" to the user "nobody" may make it
incorrect to return the actual ACL of the file in the delegation
response.
The use of delegation together with various other forms of caching
creates the possibility that no server authentication will ever be
performed for a given user since all of the user's requests might be
satisfied locally. Where the client is depending on the server for
authentication, the client should be sure authentication occurs for
each user by use of the ACCESS operation. This should be the case
even if an ACCESS operation would not be required otherwise. As
mentioned before, the server may enforce frequent authentication by
returning an nfsace4 denying all access with every open delegation.
9.4.1. Open Delegation and Data Caching
OPEN delegation allows much of the message overhead associated with
the opening and closing files to be eliminated. An open when an open
delegation is in effect does not require that a validation message be
sent to the server. The continued endurance of the "read open
delegation" provides a guarantee that no OPEN for write and thus no
write has occurred. Similarly, when closing a file opened for write
and if write open delegation is in effect, the data written does not
have to be flushed to the server until the open delegation is
recalled. The continued endurance of the open delegation provides a
guarantee that no open and thus no read or write has been done by
another client.
For the purposes of open delegation, READs and WRITEs done without an
OPEN are treated as the functional equivalents of a corresponding
type of OPEN. This refers to the READs and WRITEs that use the
special stateids consisting of all zero bits or all one bits.
Therefore, READs or WRITEs with a special stateid done by another
client will force the server to recall a write open delegation. A
WRITE with a special stateid done by another client will force a
recall of read open delegations.
With delegations, a client is able to avoid writing data to the
server when the CLOSE of a file is serviced. The CLOSE operation is
the usual point at which the client is notified of a lack of stable
storage for the modified file data generated by the application. At
the CLOSE, file data is written to the server and through normal
accounting the server is able to determine if the available file
system space for the data has been exceeded (i.e. server returns
NFS4ERR_NOSPC or NFS4ERR_DQUOT). This accounting includes quotas.
The introduction of delegations requires that a alternative method be
in place for the same type of communication to occur between client
and server.
In the delegation response, the server provides either the limit of
the size of the file or the number of modified blocks and associated
block size. The server must ensure that the client will be able to
flush data to the server of a size equal to that provided in the
original delegation. The server must make this assurance for all
outstanding delegations. Therefore, the server must be careful in
its management of available space for new or modified data taking
into account available file system space and any applicable quotas.
The server can recall delegations as a result of managing the
available file system space. The client should abide by the server's
state space limits for delegations. If the client exceeds the stated
limits for the delegation, the server's behavior is undefined.
Based on server conditions, quotas or available file system space,
the server may grant write open delegations with very restrictive
space limitations. The limitations may be defined in a way that will
always force modified data to be flushed to the server on close.
With respect to authentication, flushing modified data to the server
after a CLOSE has occurred may be problematic. For example, the user
of the application may have logged off of the client and unexpired
authentication credentials may not be present. In this case, the
client may need to take special care to ensure that local unexpired
credentials will in fact be available. This may be accomplished by
tracking the expiration time of credentials and flushing data well in
advance of their expiration or by making private copies of
credentials to assure their availability when needed.
9.4.2. Open Delegation and File Locks
When a client holds a write open delegation, lock operations are
performed locally. This includes those required for mandatory file
locking. This can be done since the delegation implies that there
can be no conflicting locks. Similarly, all of the revalidations
that would normally be associated with obtaining locks and the
flushing of data associated with the releasing of locks need not be
done.
9.4.3. Recall of Open Delegation
The following events necessitate recall of an open delegation:
o Potentially conflicting OPEN request (or READ/WRITE done with
"special" stateid)
o SETATTR issued by another client
o REMOVE request for the file
o RENAME request for the file as either source or target of the
RENAME
Whether a RENAME of a directory in the path leading to the file
results in recall of an open delegation depends on the semantics of
the server file system. If that file system denies such RENAMEs when
a file is open, the recall must be performed to determine whether the
file in question is, in fact, open.
In addition to the situations above, the server may choose to recall
open delegations at any time if resource constraints make it
advisable to do so. Clients should always be prepared for the
possibility of recall.
The server needs to employ special handling for a GETATTR where the
target is a file that has a write open delegation in effect. In this
case, the client holding the delegation needs to be interrogated.
The server will use a CB_GETATTR callback, if the GETATTR attribute
bits include any of the attributes that a write open delegate may
modify (object_size, time_modify, change).
When a client receives a recall for an open delegation, it needs to
update state on the server before returning the delegation. These
same updates must be done whenever a client chooses to return a
delegation voluntarily. The following items of state need to be
dealt with:
o If the file associated with the delegation is no longer open and
no previous CLOSE operation has been sent to the server, a CLOSE
operation must be sent to the server.
o If a file has other open references at the client, then OPEN
operations must be sent to the server. The appropriate stateids
will be provided by the server for subsequent use by the client
since the delegation stateid will not longer be valid. These OPEN
requests are done with the claim type of CLAIM_DELEGATE_CUR. This
will allow the presentation of the delegation stateid so that the
client can establish the appropriate rights to perform the OPEN.
(see the section "Operation 18: OPEN" for details.)
o If there are granted file locks, the corresponding LOCK operations
need to be performed. This applies to the write open delegation
case only.
o For a write open delegation, if at the time of recall the file is
not open for write, all modified data for the file must be flushed
to the server. If the delegation had not existed, the client
would have done this data flush before the CLOSE operation.
o For a write open delegation when a file is still open at the time
of recall, any modified data for the file needs to be flushed to
the server.
o With the write open delegation in place, it is possible that the
file was truncated during the duration of the delegation. For
example, the truncation could have occurred as a result of an OPEN
UNCHECKED with a object_size attribute value of zero. Therefore,
if a truncation of the file has occurred and this operation has
not been propagated to the server, the truncation must occur
before any modified data is written to the server.
In the case of write open delegation, file locking imposes some
additional requirements. The flushing of any modified data in any
region for which a write lock was released while the write open
delegation was in effect is what is required to precisely maintain
the associated invariant. However, because the write open delegation
implies no other locking by other clients, a simpler implementation
is to flush all modified data for the file (as described just above)
if any write lock has been released while the write open delegation
was in effect.
9.4.4. Delegation Revocation
At the point a delegation is revoked, if there are associated opens
on the client, the applications holding these opens need to be
notified. This notification usually occurs by returning errors for
READ/WRITE operations or when a close is attempted for the open file.
If no opens exist for the file at the point the delegation is
revoked, then notification of the revocation is unnecessary.
However, if there is modified data present at the client for the
file, the user of the application should be notified. Unfortunately,
it may not be possible to notify the user since active applications
may not be present at the client. See the section "Revocation
Recovery for Write Open Delegation" for additional details.
9.5. Data Caching and Revocation
When locks and delegations are revoked, the assumptions upon which
successful caching depend are no longer guaranteed. The owner of the
locks or share reservations which have been revoked needs to be
notified. This notification includes applications with a file open
that has a corresponding delegation which has been revoked. Cached
data associated with the revocation must be removed from the client.
In the case of modified data existing in the client's cache, that
data must be removed from the client without it being written to the
server. As mentioned, the assumptions made by the client are no
longer valid at the point when a lock or delegation has been revoked.
For example, another client may have been granted a conflicting lock
after the revocation of the lock at the first client. Therefore, the
data within the lock range may have been modified by the other
client. Obviously, the first client is unable to guarantee to the
application what has occurred to the file in the case of revocation.
Notification to a lock owner will in many cases consist of simply
returning an error on the next and all subsequent READs/WRITEs to the
open file or on the close. Where the methods available to a client
make such notification impossible because errors for certain
operations may not be returned, more drastic action such as signals
or process termination may be appropriate. The justification for
this is that an invariant for which an application depends on may be
violated. Depending on how errors are typically treated for the
client operating environment, further levels of notification
including logging, console messages, and GUI pop-ups may be
appropriate.
9.5.1. Revocation Recovery for Write Open Delegation
Revocation recovery for a write open delegation poses the special
issue of modified data in the client cache while the file is not
open. In this situation, any client which does not flush modified
data to the server on each close must ensure that the user receives
appropriate notification of the failure as a result of the
revocation. Since such situations may require human action to
correct problems, notification schemes in which the appropriate user
or administrator is notified may be necessary. Logging and console
messages are typical examples.
If there is modified data on the client, it must not be flushed
normally to the server. A client may attempt to provide a copy of
the file data as modified during the delegation under a different
name in the file system name space to ease recovery. Unless the
client can determine that the file has not modified by any other
client, this technique must be limited to situations in which a
client has a complete cached copy of the file in question. Use of
such a technique may be limited to files under a certain size or may
only be used when sufficient disk space is guaranteed to be available
within the target file system and when the client has sufficient
buffering resources to keep the cached copy available until it is
properly stored to the target file system.
9.6. Attribute Caching
The attributes discussed in this section do not include named
attributes. Individual named attributes are analogous to files and
caching of the data for these needs to be handled just as data
caching is for ordinary files. Similarly, LOOKUP results from an
OPENATTR directory are to be cached on the same basis as any other
pathnames and similarly for directory contents.
Clients may cache file attributes obtained from the server and use
them to avoid subsequent GETATTR requests. Such caching is write
through in that modification to file attributes is always done by
means of requests to the server and should not be done locally and
cached. The exception to this are modifications to attributes that
are intimately connected with data caching. Therefore, extending a
file by writing data to the local data cache is reflected immediately
in the object_size as seen on the client without this change being
immediately reflected on the server. Normally such changes are not
propagated directly to the server but when the modified data is
flushed to the server, analogous attribute changes are made on the
server. When open delegation is in effect, the modified attributes
may be returned to the server in the response to a CB_RECALL call.
The result of local caching of attributes is that the attribute
caches maintained on individual clients will not be coherent. Changes
made in one order on the server may be seen in a different order on
one client and in a third order on a different client.
The typical file system application programming interfaces do not
provide means to atomically modify or interrogate attributes for
multiple files at the same time. The following rules provide an
environment where the potential incoherences mentioned above can be
reasonably managed. These rules are derived from the practice of
previous NFS protocols.
o All attributes for a given file (per-fsid attributes excepted) are
cached as a unit at the client so that no non-serializability can
arise within the context of a single file.
o An upper time boundary is maintained on how long a client cache
entry can be kept without being refreshed from the server.
o When operations are performed that change attributes at the
server, the updated attribute set is requested as part of the
containing RPC. This includes directory operations that update
attributes indirectly. This is accomplished by following the
modifying operation with a GETATTR operation and then using the
results of the GETATTR to update the client's cached attributes.
Note that if the full set of attributes to be cached is requested by
READDIR, the results can be cached by the client on the same basis as
attributes obtained via GETATTR.
A client may validate its cached version of attributes for a file by
fetching only the change attribute and assuming that if the change
attribute has the same value as it did when the attributes were
cached, then no attributes have changed. The possible exception is
the attribute time_access.
9.7. Name Caching
The results of LOOKUP and READDIR operations may be cached to avoid
the cost of subsequent LOOKUP operations. Just as in the case of
attribute caching, inconsistencies may arise among the various client
caches. To mitigate the effects of these inconsistencies and given
the context of typical file system APIs, the following rules should
be followed:
o The results of unsuccessful LOOKUPs should not be cached, unless
they are specifically reverified at the point of use.
o An upper time boundary is maintained on how long a client name
cache entry can be kept without verifying that the entry has not
been made invalid by a directory change operation performed by
another client.
When a client is not making changes to a directory for which there
exist name cache entries, the client needs to periodically fetch
attributes for that directory to ensure that it is not being
modified. After determining that no modification has occurred, the
expiration time for the associated name cache entries may be updated
to be the current time plus the name cache staleness bound.
When a client is making changes to a given directory, it needs to
determine whether there have been changes made to the directory by
other clients. It does this by using the change attribute as
reported before and after the directory operation in the associated
change_info4 value returned for the operation. The server is able to
communicate to the client whether the change_info4 data is provided
atomically with respect to the directory operation. If the change
values are provided atomically, the client is then able to compare
the pre-operation change value with the change value in the client's
name cache. If the comparison indicates that the directory was
updated by another client, the name cache associated with the
modified directory is purged from the client. If the comparison
indicates no modification, the name cache can be updated on the
client to reflect the directory operation and the associated timeout
extended. The post-operation change value needs to be saved as the
basis for future change_info4 comparisons.
As demonstrated by the scenario above, name caching requires that the
client revalidate name cache data by inspecting the change attribute
of a directory at the point when the name cache item was cached.
This requires that the server update the change attribute for
directories when the contents of the corresponding directory is
modified. For a client to use the change_info4 information
appropriately and correctly, the server must report the pre and post
operation change attribute values atomically. When the server is
unable to report the before and after values atomically with respect
to the directory operation, the server must indicate that fact in the
change_info4 return value. When the information is not atomically
reported, the client should not assume that other clients have not
changed the directory.
9.8. Directory Caching
The results of READDIR operations may be used to avoid subsequent
READDIR operations. Just as in the cases of attribute and name
caching, inconsistencies may arise among the various client caches.
To mitigate the effects of these inconsistencies, and given the
context of typical file system APIs, the following rules should be
followed:
o Cached READDIR information for a directory which is not obtained
in a single READDIR operation must always be a consistent snapshot
of directory contents. This is determined by using a GETATTR
before the first READDIR and after the last of READDIR that
contributes to the cache.
o An upper time boundary is maintained to indicate the length of
time a directory cache entry is considered valid before the client
must revalidate the cached information.