Request for Comments: 4549 Isode Ltd.
Category: Informational June 2006
Synchronization Operations for Disconnected IMAP4 Clients
Status of This Memo
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2006).
Abstract
This document attempts to address some of the issues involved in
building a disconnected IMAP4 client. In particular, it deals with
the issues of what might be called the "driver" portion of the
synchronization tool: the portion of the code responsible for issuing
the correct set of IMAP4 commands to synchronize the disconnected
client in the way that is most likely to make the human who uses the
disconnected client happy.
This note describes different strategies that can be used by
disconnected clients and shows how to use IMAP protocol in order to
minimize the time of the synchronization process.
This note also lists IMAP extensions that a server should implement
in order to provide better synchronization facilities to disconnected
clients.
Table of Contents
1. Introduction ....................................................3
1.1. Conventions Used in This Document ..........................3
2. Design Principles ...............................................3
3. Overall Picture of Synchronization ..............................4
4. Mailbox Synchronization Steps and Strategies ....................7
4.1. Checking UID Validity ......................................7
4.2. Synchronizing Local Changes with the Server ................8
4.2.1. Uploading Messages to the Mailbox ...................8
4.2.2. Optimizing "move" and "copy" Operations .............9
4.2.3. Replaying Local Flag Changes .......................14
4.2.4. Processing Mailbox Compression (EXPUNGE) Requests ..15
4.2.5. Closing a Mailbox ..................................17
4.3. Details of "Normal" Synchronization of a Single Mailbox ...18
4.3.1. Discovering New Messages and Changes to Old
Messages ...........................................18
4.3.2. Searching for "Interesting" Messages. ..............20
4.3.3. Populating Cache with "Interesting" Messages. ......21
4.3.4. User-Initiated Synchronization .....................22
4.4. Special Case: Descriptor-Only Synchronization .............22
4.5. Special Case: Fast New-Only Synchronization ...............23
4.6. Special Case: Blind FETCH .................................23
5. Implementation Considerations ..................................24
5.1. Error Recovery during Playback ............................26
5.2. Quality of Implementation Issues ..........................28
5.3. Optimizations .............................................28
6. IMAP Extensions That May Help ..................................30
6.1. CONDSTORE Extension .......................................30
7. Security Considerations ........................................33
8. References .....................................................33
8.1. Normative References ......................................33
8.2. Informative References ....................................34
9. Acknowledgements ...............................................34
1. Introduction
Several recommendations presented in this document are generally
applicable to all types of IMAP clients. However, this document
tries to concentrate on disconnected mail clients [IMAP-MODEL]. It
also suggests some IMAP extensions* that should be implemented by
IMAP servers in order to make the life of disconnected clients
easier. In particular, the [UIDPLUS] extension was specifically
designed to streamline certain disconnected operations, like
expunging, uploading, and copying messages (see Sections 4.2.1,
4.2.2.1, and 4.2.4).
Readers of this document are also strongly advised to read RFC 2683
[RFC2683].
* Note that the functionality provided by the base IMAP protocol
[IMAP4] is sufficient to perform basic synchronization.
1.1. Conventions Used in This Document
In examples, "C:" and "S:" indicate lines sent by the client and
server, respectively. Long lines in examples are broken for
editorial clarity.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [KEYWORDS].
Let’s call an IMAP command idempotent if the result of executing the
command twice sequentially is the same as the result of executing the
command just once.
2. Design Principles
All mailbox state or content information stored on the disconnected
client should be viewed strictly as a cache of the state of the
server. The "master" state remains on the server, just as it would
with an interactive IMAP4 client. The one exception to this rule is
that information about the state of the disconnected client’s cache
(the state includes flag changes while offline and during scheduled
message uploads) remains on the disconnected client: that is, the
IMAP4 server is not responsible for remembering the state of the
disconnected IMAP4 client.
We assume that a disconnected client is a client that, for whatever
reason, wants to minimize the length of time that it is "on the
phone" to the IMAP4 server. Often this will be because the client is
using a dialup connection, possibly with very low bandwidth, but
sometimes it might just be that the human is in a hurry to catch an
airplane, or some other event beyond our control. Whatever the
reason, we assume that we must make efficient use of the network
connection, both in the usual sense (not generating spurious traffic)
and in the sense that we would prefer not to have the connection
sitting idle while the client and/or the server is performing
strictly local computation or I/O. Another, perhaps simpler way of
stating this is that we assume that network connections are
"expensive".
Practical experience with disconnected mail systems has shown that
there is no single synchronization strategy that is appropriate for
all cases. Different humans have different preferences, and the same
human’s preference will vary depending both on external circumstance
(how much of a hurry the human is in today) and on the value that the
human places on the messages being transferred. The point here is
that there is no way that the synchronization program can guess
exactly what the human wants to do, so the human will have to provide
some guidance.
Taken together, the preceding two principles lead to the conclusion
that the synchronization program must make its decisions based on
some kind of guidance provided by the human, by selecting the
appropriate options in the user interface or through some sort of
configuration file. Almost certainly, it should not pause for I/O
with the human in the middle of the synchronization process. The
human will almost certainly have several different configurations for
the synchronization program, for different circumstances.
Since a disconnected client has no way of knowing what changes might
have occurred to the mailbox while it was disconnected, message
numbers are not useful to a disconnected client. All disconnected
client operations should be performed using UIDs, so that the client
can be sure that it and the server are talking about the same
messages during the synchronization process.
3. Overall Picture of Synchronization
The basic strategy for synchronization is outlined below. Note that
the real strategy may vary from one application to another or may
depend on a synchronization mode.
a) Process any "actions" that were pending on the client that were
not associated with any mailbox. (In particular sending messages
composed offline with SMTP. This is not part of IMAP
synchronization, but it is mentioned here for completeness.)
b) Fetch the current list of "interesting" mailboxes. (The
disconnected client should allow the user to skip this step
completely.)
c) "Client-to-server synchronization": for each IMAP "action" that
was pending on the client, do the following:
1) If the action implies opening a new mailbox (any operation that
operates on messages), open the mailbox. Check its UID
validity value (see Section 4.1 for more details) returned in
the UIDVALIDITY response code. If the UIDVALIDITY value
returned by the server differs, the client MUST empty the local
cache of the mailbox and remove any pending "actions" that
refer to UIDs in that mailbox (and consider them failed). Note
that this doesn’t affect actions performed on client-generated
fake UIDs (see Section 5).
2) Perform the action. If the action is to delete a mailbox
(DELETE), make sure that the mailbox is closed first (see also
Section 3.4.12 of [RFC2683]).
d) "Server-to-client synchronization": for each mailbox that requires
synchronization, do the following:
1) Check the mailbox UIDVALIDITY (see Section 4.1 for more
details) with SELECT/EXAMINE/STATUS.
If UIDVALIDITY value returned by the server differs, the client
MUST
* empty the local cache of that mailbox;
* remove any pending "actions" that refer to UIDs in that
mailbox and consider them failed; and
* skip step 2-II.
2) Fetch the current "descriptors";
I) Discover new messages.
II) Discover changes to old messages.
3) Fetch the bodies of any "interesting" messages that the client
doesn’t already have.
e) Close all open mailboxes not required for further operations (if
staying online) or disconnect all open connections (if going
offline).
Terms used:
"Actions" are queued requests that were made by the human to the
client’s Mail User Agent (MUA) software while the client was
disconnected.
We define "descriptors" as a set of IMAP4 FETCH data items.
Conceptually, a message’s descriptor is that set of information that
allows the synchronization program to decide what protocol actions
are necessary to bring the local cache to the desired state for this
message; since this decision is really up to the human, this
information probably includes at least a few header fields intended
for human consumption. Exactly what will constitute a descriptor
depends on the client implementation. At a minimum, the descriptor
contains the message’s UID and FLAGS. Other likely candidates are
the RFC822.SIZE, RFC822.HEADER, BODYSTRUCTURE, or ENVELOPE data
items.
Comments:
1) The list of actions should be ordered. For example, if the human
deletes message A1 in mailbox A, then expunges mailbox A, and then
deletes message A2 in mailbox A, the human will expect that
message A1 is gone and that message A2 is still present but is now
deleted.
By processing all the actions before proceeding with
synchronization, we avoid having to compensate for the local MUA’s
changes to the server’s state. That is, once we have processed
all the pending actions, the steps that the client must take to
synchronize itself will be the same no matter where the changes to
the server’s state originated.
2) Steps a and b can be performed in parallel. Alternatively, step a
can be performed after d.
3) On step b, the set of "interesting" mailboxes pretty much has to
be determined by the human. What mailboxes belong to this set may
vary between different IMAP4 sessions with the same server,
client, and human. An interesting mailbox can be a mailbox
returned by LSUB command (see Section 6.3.9 of [IMAP4]). The
special mailbox "INBOX" SHOULD be in the default set of mailboxes
that the client considers interesting. However, providing the
ability to ignore INBOX for a particular session or client may be
valuable for some mail filtering strategies.
4) On step d-2-II, the client also finds out about changes to the
flags of messages that the client already has in its local cache,
and about messages in the local cache that no longer exist on the
server (i.e., messages that have been expunged).
5) "Interesting" messages are those messages that the synchronization
program thinks the human wants to have cached locally, based on
the configuration and the data retrieved in step b.
6) A disconnected IMAP client is a special case of an IMAP client, so
it MUST be able to handle any "unexpected" unsolicited responses,
like EXISTS and EXPUNGE, at any time. The disconnected client MAY
ignore EXPUNGE response during "client-to-server" synchronization
phase (step c).
The rest of this discussion will focus primarily on the
synchronization issues for a single mailbox.
4. Mailbox Synchronization Steps and Strategies
4.1. Checking UID Validity
The "UID validity" of a mailbox is a number returned in an
UIDVALIDITY response code in an OK untagged response at mailbox
selection time. The UID validity value changes between sessions when
UIDs fail to persist between sessions.
Whenever the client selects a mailbox, the client must compare the
returned UID validity value with the value stored in the local cache.
If the UID validity values differ, the UIDs in the client’s cache are
no longer valid. The client MUST then empty the local cache of that
mailbox and remove any pending "actions" that refer to UIDs in that
mailbox. The client MAY also issue a warning to the human. The
client MUST NOT cancel any scheduled uploads (i.e., APPENDs) for the
mailbox.
Note that UIDVALIDITY is not only returned on a mailbox selection.
The COPYUID and APPENDUID response codes defined in the [UIDPLUS]
extension (see also 4.2.2) and the UIDVALIDITY STATUS response data
item also contain a UIDVALIDITY value for some other mailbox. The
client SHOULD behave as described in the previous paragraph (but it
should act on the other mailbox’s cache), no matter how it obtained
the UIDVALIDITY value.
4.2. Synchronizing Local Changes with the Server
4.2.1. Uploading Messages to the Mailbox
Two of the most common examples of operations resulting in message
uploads are:
1) Saving a draft message
2) Copying a message between remote mailboxes on two different IMAP
servers or a local mailbox and a remote mailbox.
Message upload is performed with the APPEND command. A message
scheduled to be uploaded has no UID associated with it, as all UIDs
are assigned by the server. The APPEND command will effectively
associate a UID with the uploaded message that can be stored in the
local cache for future reference. However, [IMAP4] doesn’t describe
a simple mechanism to discover the message UID by just performing the
APPEND command. In order to discover the UID, the client can do one
of the following:
1) Remove the uploaded message from cache. Then, use the mechanism
described in 4.3 to fetch the information about the uploaded
message as if it had been uploaded by some other client.
2) Try to fetch header information as described in 4.2.2 in order to
find a message that corresponds to the uploaded message. One
strategy for doing this is described in 4.2.2.
Case 1 describes a not particularly smart client.
C: A003 APPEND Drafts (\Seen $MDNSent) {310}
S: + Ready for literal data
C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
C: From: Fred Foobar <foobar@blt.example.COM>
C: Subject: afternoon meeting
C: To: mooch@owatagu.siam.edu
C: Message-Id: <B27397-0100000@blt.example.COM>
C: MIME-Version: 1.0
C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
C:
C: Hello Joe, do you think we can meet at 3:30 tomorrow?
C:
S: A003 OK APPEND Completed
Fortunately, there is a simpler way to discover the message UID in
the presence of the [UIDPLUS] extension:
C: A003 APPEND Drafts (\Seen $MDNSent) {310}
S: + Ready for literal data
C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
C: From: Fred Foobar <foobar@blt.example.COM>
C: Subject: afternoon meeting
C: To: mooch@owatagu.siam.edu
C: Message-Id: <B27397-0100000@blt.example.COM>
C: MIME-Version: 1.0
C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
C:
C: Hello Joe, do you think we can meet at 3:30 tomorrow?
C:
S: A003 OK [APPENDUID 1022843275 77712] APPEND completed
The UID of the appended message is the second parameter of APPENDUID
response code.
4.2.2. Optimizing "move" and "copy" Operations
Practical experience with IMAP and other mailbox access protocols
that support multiple mailboxes suggests that moving a message from
one mailbox to another is an extremely common operation.
4.2.2.1. Moving a Message between Two Mailboxes on the Same Server
In IMAP4, a "move" operation between two mailboxes on the same server
is really a combination of a COPY operation and a STORE +FLAGS
(\Deleted) operation. This makes good protocol sense for IMAP, but
it leaves a simple-minded disconnected client in the silly position
of deleting and possibly expunging its cached copy of a message, then
fetching an identical copy via the network.
However, the presence of the UIDPLUS extension in the server can
help:
C: A001 UID COPY 567,414 "Interesting Messages"
S: A001 OK [COPYUID 1022843275 414,567 5:6] Completed
This tells the client that the message with UID 414 in the current
mailbox was successfully copied to the mailbox "Interesting Messages"
and was given the UID 5, and that the message with UID 567 was given
the UID 6.
In the absence of UIDPLUS extension support in the server, the
following trick can be used. By including the Message-ID: header and
the INTERNALDATE data item as part of the descriptor, the client can
check the descriptor of a "new" message against messages that are
already in its cache and avoid fetching the extra copy. Of course,
it’s possible that the cost of checking to see if the message is
already in the local cache may exceed the cost of just fetching it,
so this technique should not be used blindly. If the MUA implements
a "move" command, it makes special provisions to use this technique
when it knows that a copy/delete sequence is the result of a "move"
command.
Note that servers are not required (although they are strongly
encouraged with "SHOULD language") to preserve INTERNALDATE when
copying messages.
Also note that since it’s theoretically possible for this algorithm
to find the wrong message (given sufficiently malignant Message-ID
headers), implementers should provide a way to disable this
optimization, both permanently and on a message-by-message basis.
Example 1: Copying a message in the absence of UIDPLUS extension.
At some point in time the client has fetched the source message and
some information was cached:
C: C021 UID FETCH <uids> (BODY.PEEK[] INTERNALDATE FLAGS)
...
S: * 27 FETCH (UID 123 INTERNALDATE "31-May-2002 05:26:59 -0600"
FLAGS (\Draft $MDNSent) BODY[] {1036}
S: ...
S: Message-Id: <20040903110856.22a127cd@chardonnay>
S: ...
S: ...message body...
S: )
...
S: C021 OK fetch completed
Later on, the client decides to copy the message:
C: C035 UID COPY 123 "Interesting Messages"
S: C035 OK Completed
As the server hasn’t provided the COPYUID response code, the client
tries the optimization described above:
C: C036 SELECT "Interesting Messages"
...
C: C037 UID SEARCH ON 31-May-2002 HEADER
"Message-Id" "20040903110856.22a127cd@chardonnay"
S: SEARCH 12368
S: C037 OK completed
Note that if the server has returned multiple UIDs in the SEARCH
response, the client MUST NOT use any of the returned UID.
4.2.2.2. Moving a Message from a Remote Mailbox to a Local
Moving a message from a remote mailbox to a local is done with FETCH
(that includes FLAGS and INTERNALDATE) followed by UID STORE <uid>
+FLAGS.SILENT (\Deleted):
C: A003 UID FETCH 123 (BODY.PEEK[] INTERNALDATE FLAGS)
S: * 27 FETCH (UID 123 INTERNALDATE "31-May-2002 05:26:59 -0600"
FLAGS (\Seen $MDNSent) BODY[]
S: ...message body...
S: )
S: A003 OK UID FETCH completed
C: A004 UID STORE <uid> +FLAGS.SILENT (\Deleted)
S: A004 STORE completed
Note that there is no reason to fetch the message during
synchronization if it’s already in the client’s cache. Also, the
client SHOULD preserve delivery date in the local cache.
4.2.2.3. Moving a Message from a Local Mailbox to a Remote
Moving a message from a local mailbox to a remote is done with
APPEND:
C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
{310}
S: + Ready for literal data
C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
C: From: Fred Foobar <foobar@blt.example.COM>
C: Subject: afternoon meeting
C: To: mooch@owatagu.siam.edu
C: Message-Id: <B27397-0100000@blt.example.COM>
C: MIME-Version: 1.0
C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
C:
C: Hello Joe, do you think we can meet at 3:30 tomorrow?
C:
S: A003 OK [APPENDUID 1022843275 77712] completed
The client SHOULD specify the delivery date from the local cache in
the APPEND.
If the [LITERAL+] extension is available, the client can save a
round-trip*:
C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
{310+}
C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
C: From: Fred Foobar <foobar@blt.example.COM>
C: Subject: afternoon meeting
C: To: mooch@owatagu.siam.edu
C: Message-Id: <B27397-0100000@blt.example.COM>
C: MIME-Version: 1.0
C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
C:
C: Hello Joe, do you think we can meet at 3:30 tomorrow?
C:
S: A003 OK [APPENDUID 1022843275 77712] completed
* Note that there is a risk that the server will reject the message
due to its size. If this happens, the client will waste bandwidth
transferring the whole message. If the client wouldn’t have used
the LITERAL+, this could have been avoided:
C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2004 05:26:59 -0600"
{16777215}
S: A003 NO Sorry, message is too big
4.2.2.4. Moving a Message between Two Mailboxes on Different Servers
Moving a message between two mailbox on two different servers is a
combination of the operations described in 4.2.2.2 followed by the
operations described in 4.2.2.3.
4.2.2.5. Uploading Multiple Messages to a Remote Mailbox with
MULTIAPPEND
When there is a need to upload multiple messages to a remote mailbox
(e.g., as per 4.2.2.3), the presence of certain IMAP extensions may
significantly improve performance. One of them is [MULTIAPPEND].
For some mail stores, opening a mailbox for appending might be
expensive. [MULTIAPPEND] tells the server to open the mailbox once
(instead of opening and closing it "n" times per "n" messages to be
uploaded) and to keep it open while a group of messages is being
uploaded to the server.
Also, if the server supports both [MULTIAPPEND] and [LITERAL+]