the mailbox called Drafts on an IMAP server. We will also use the
following conventions: <old_uid> is the UID of the intermediate
version of the draft when it was saved for the first time. This
is a fake UID generated on the client. <new_uid> is the UID of
the final version of the draft. This is another fake UID
generated on the client.
1) APPEND Drafts (\Seen $MDNSent \Drafts) {<nnn>}
...first version of the message follows...
2) APPEND Drafts (\Seen $MDNSent \Drafts) {<mmm>}
...final version of the message follows...
3) STORE <old_uid> +FLAGS (\Deleted)
Step 1 corresponds to the first attempt to save the draft message,
step 2 corresponds to the second attempt to save the draft
message, and step 3 deletes the first version of the draft message
saved in step 1.
A naive disconnected client may send the command in step 3 without
replacing the fake client generated <old_uid> with the value
returned by the server in step 1. A server will probably reject
this command, which will make the client believe that the
synchronization sequence has failed.
2) Section 5.1 discusses common implementation errors related to
error recovery during playback.
3) Don’t assume that the disconnected client is the only client used
by the user.
Situation 2: Some clients may use the \Deleted flag as an
indicator that the message should not appear in the user’s view.
Usage of the \Deleted flag for this purpose is not safe, as other
clients (e.g., online clients) might EXPUNGE the mailbox at any
time.
4) Beware of data dependencies between synchronization operations.
It might be very tempting for a client writer to perform some
optimizations on the playback log. Such optimizations might
include removing redundant operations (for example, see
optimization 2 in Section 5.3), or their reordering.
It is not always safe to reorder or remove redundant operations
during synchronization because some operations may have
dependencies (as Situation 3 demonstrates). So, if in doubt,
don’t do this.
Situation 3: The user copies a message out of a mailbox and then
deletes the mailbox.
C: A001 SELECT Old-Mail
S: ...
C: A002 UID COPY 111 ToDo
S: A002 OK [COPYUID 1022843345 111 94] Copy completed
...
C: A015 CLOSE
S: A015 OK Completed
C: A016 DELETE Old-Mail
S: A016 OK Mailbox deletion completed successfully
If the client performs DELETE (tag A016) first and COPY (tag A002)
second, then the COPY fails. Also, the message that the user so
carefully copied into another mailbox has been lost.
5.1. Error Recovery during Playback
Error recovery during synchronization is one of the trickiest parts
to get right. Below, we will discuss certain error conditions and
suggest possible choices for handling them.
1) Lost connection to the server.
The client MUST remember the current position in the playback
(replay) log and replay it starting from the interrupted operation
(the last command issued by the client, but not acknowledged by
the server) the next time it successfully connects to the same
server. If the connection was lost while executing a non-
idempotent IMAP command (see the definition in Section 1), then
when the client is reconnected, it MUST make sure that the
interrupted command was indeed not executed. If it wasn’t
executed, the client must restart playback from the interrupted
command, otherwise from the following command.
Upon reconnect, care must be taken in order to properly reapply
logical operations that are represented by multiple IMAP commands,
e.g., UID EXPUNGE emulation when UID EXPUNGE is not supported by
the server (see Section 4.2.4).
Once the client detects that the connection to the server was
lost, it MUST stop replaying its log. There are existing
disconnected clients that, to the great annoyance of users, pop up
an error dialog for each and every playback operation that fails.
2) Copying/appending messages to a mailbox that doesn’t exist. (The
server advertises this condition by sending the TRYCREATE response
code in the tagged NO response to the APPEND or COPY command.)
The user should be advised about the situation and be given one of
the following choices:
a) Try to recreate a mailbox.
b) Copy/upload messages to another mailbox.
c) Skip copy/upload.
d) Abort replay.
3) Copying messages from a mailbox that doesn’t exist, or renaming or
getting/changing ACLs [ACL] on a mailbox that doesn’t exist:
a) Skip operation.
b) Abort replay.
4) Deleting mailboxes or deleting/expunging messages that no longer
exist.
This is actually is not an error and should be ignored by the
client.
5) Performing operations on messages that no longer exist.
a) Skip operation.
b) Abort replay.
In the case of changing flags on an expunged message, the client
should silently ignore the error.
Note 1: Several synchronization operations map to multiple IMAP
commands (for example, "move" described in 4.2.2). The client must
guarantee atomicity of each such multistep operation. For example,
when performing a "move" between two mailboxes on the same server, if
the server is unable to copy messages, the client MUST NOT attempt to
set the \Deleted flag on the messages being copied, let alone expunge
them. However, the client MAY consider that move operation to have
succeeded even if the server was unable to set the \Deleted flag on
copied messages.
Note 2: Many synchronization operations have data dependencies. A
failed operation must cause all dependent operations to fail as well.
The client should check this and MUST NOT try to perform all
dependent operations blindly (unless the user corrected the original
problem). For example, a message may be scheduled to be appended to
a mailbox on the server and later on the appended message may be
copied to another mailbox. If the APPEND operation fails, the client
must not attempt to COPY the failed message later on. (See also
Section 5, Situation 3).
5.2. Quality of Implementation Issues
Below, some quality of implementation issues are listed for
disconnected clients. They will help to write a disconnected client
that works correctly, performs synchronization as quickly as possible
(and thus can make the user happier as well as save her some money),
and minimizes the server load:
1) Don’t lose information.
No matter how smart your client is in other areas, if it loses
information, users will get very upset.
2) Don’t do work unless explicitly asked. Be flexible. Ask all
questions BEFORE starting synchronization, if possible.
3) Minimize traffic.
The client MUST NOT issue a command if the client already received
the required information from the server.
The client MUST make use of UIDPLUS extension if it is supported
by the server.
See also optimization 1 in Section 5.3.
4) Minimize the number of round-trips.
Round-trips kill performance, especially on links with high
latency. Sections 4.2.2.5 and 5.2 give some advice on how to
minimize the number of round-trips.
See also optimization 1 in Section 5.3.
5.3. Optimizations
Some useful optimizations are described in this section. A
disconnected client that supports the recommendations listed below
will give the user a more pleasant experience.
1) The initial OK or PREAUTH responses may contain the CAPABILITY
response code as described in Section 7.1 of [IMAP4]. This
response code gives the same information as returned by the
CAPABILITY command*. A disconnected client that pays attention to
this response code can avoid sending CAPABILITY command and will
save a round-trip.
* Note: Some servers report in the CAPABILITY response code
extensions that are only relevant in unauthenticated state or in
all states. Such servers usually send another CAPABILITY
response code upon successful authentication using LOGIN or
AUTHENTICATE command (that negotiates no security layer; see
Section 6.2.2 of [IMAP4]). The CAPABILITY response code sent
upon successful LOGIN/AUTHENTICATE might be different from the
CAPABILITY response code in the initial OK response, as
extensions only relevant for unauthenticated state will not be
advertised, and some additional extensions available only in
authenticated and/or selected state will be.
Example 9:
S: * OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS
AUTH=DIGEST-MD5 AUTH=SRP] imap.example.com ready
C: 2 authenticate DIGEST-MD5
S: 2 OK [CAPABILITY IMAP4REV1 IDLE NAMESPACE MAILBOX-REFERRALS SCAN
SORT THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND]
User authenticated (no layer)
2) An advanced disconnected client may choose to optimize its replay
log. For example, there might be some operations that are
redundant (the list is not complete):
a) an EXPUNGE followed by another EXPUNGE or CLOSE;
b) changing flags (other than the \Deleted flag) on a message that
gets immediately expunged;
c) opening and closing the same mailbox.
When optimizing, be careful about data dependencies between commands.
For example, if the client is wishing to optimize (see case b, above)
tag1 UID STORE <uid1> +FLAGS (\Deleted)
...
tag2 UID STORE <uid1> +FLAGS (\Flagged)
...
tag3 UID COPY <uid1> "Backup"
...
tag4 UID EXPUNGE <uid1>
it can’t remove the second UID STORE command because the message is
being copied before it gets expunged.
In general, it might be a good idea to keep mailboxes open during
synchronization (see case c above), if possible. This can be more
easily achieved in conjunction with optimization 3 described below.
3) Perform some synchronization steps in parallel, if possible.
Several synchronization steps don’t depend on each other and thus
can be performed in parallel. Because the server machine is
usually more powerful than the client machine and can perform some
operations in parallel, this may speed up the total time of
synchronization.
In order to achieve such parallelization, the client will have to
open more than one connection to the same server. Client writers
should not forget about non-trivial cost associated with
establishing a TCP connection and performing an authentication.
The disconnected client MUST NOT use one connection per mailbox.
In most cases, it is sufficient to have two connections. The
disconnected client SHOULD avoid selecting the same mailbox in
more than one connection; see Section 3.1.1 of [RFC2683] for more
details.
Any mailbox synchronization MUST start with checking the
UIDVALIDITY as described in Section 4.1 of this document. The
client MAY use STATUS command to check UID Validity of a non-
selected mailbox. This is preferable to opening many connections
to the same server to perform synchronization of multiple
mailboxes simultaneously. As described in Section 5.3.10 of
[IMAP4], this SHOULD NOT be used on the selected mailbox.
6. IMAP Extensions That May Help
The following extensions can save traffic and/or the number of
round-trips:
1) The use of [UIDPLUS] is discussed in Sections 4.1, 4.2.1, 4.2.2.1
and 4.2.4.
2) The use of the MULTIAPPEND and LITERAL+ extensions for uploading
messages is discussed in Section 4.2.2.5.
3) Use the CONDSTORE extension (see Section 6.1) for quick flag
resynchronization.
6.1. CONDSTORE Extension
An advanced disconnected mail client should use the [CONDSTORE]
extension when it is supported by the server. The client must cache
the value from HIGHESTMODSEQ OK response code received on mailbox
opening and update it whenever the server sends MODSEQ FETCH data
items.
If the client receives NOMODSEQ OK untagged response instead of
HIGHESTMODSEQ, it MUST remove the last known HIGHESTMODSEQ value from
its cache and follow the more general instructions in Section 3.
When the client opens the mailbox for synchronization, it first
compares UIDVALIDITY as described in step d-1 in Section 3. If the
cached UIDVALIDITY value matches the one returned by the server, the
client MUST compare the cached value of HIGHESTMODSEQ with the one
returned by the server. If the cached HIGHESTMODSEQ value also
matches the one returned by the server, then the client MUST NOT
fetch flags for cached messages, as they hasn’t changed. If the
value on the server is higher than the cached one, the client MAY use
"SEARCH MODSEQ <cached-value>" to find all messages with flags
changed since the last time the client was online and had the mailbox
opened. Alternatively, the client MAY use "FETCH 1:* (FLAGS)
(CHANGEDSINCE <cached-value>)". The latter operation combines
searching for changed messages and fetching new information.
In all cases, the client still needs to fetch information about new
messages (if requested by the user) as well as discover which
messages have been expunged.
Step d ("Server-to-client synchronization") in Section 4 in the
presence of the CONDSTORE extension is amended as follows:
d) "Server-to-client synchronization" - For each mailbox that
requires synchronization, do the following:
1a) Check the mailbox UIDVALIDITY (see section 4.1 for more
details) with SELECT/EXAMINE/STATUS.
If the UIDVALIDITY value returned by the server differs, the
client MUST
* empty the local cache of that mailbox;
* "forget" the cached HIGHESTMODSEQ value for the mailbox;
* remove any pending "actions" that refer to UIDs in that
mailbox (note that this doesn’t affect actions performed on
client-generated fake UIDs; see Section 5); and
* skip steps 1b and 2-II;
1b) Check the mailbox HIGHESTMODSEQ. If the cached value is the
same as the one returned by the server, skip fetching message
flags on step 2-II, i.e., the client only has to find out
which messages got expunged.
2) Fetch the current "descriptors".
I) Discover new messages.
II) Discover changes to old messages and flags for new messages
using
"FETCH 1:* (FLAGS) (CHANGEDSINCE <cached-value>)" or
"SEARCH MODSEQ <cached-value>".
Discover expunged messages; for example, using
"UID SEARCH 1:<lastseenuid>". (All messages not returned
in this command are expunged.)
3) Fetch the bodies of any "interesting" messages that the client
doesn’t already have.
Example 10:
The UIDVALIDITY value is the same, but the HIGHESTMODSEQ value
has changed on the server while the client was offline.
C: A142 SELECT INBOX
S: * 172 EXISTS
S: * 1 RECENT
S: * OK [UNSEEN 12] Message 12 is first unseen
S: * OK [UIDVALIDITY 3857529045] UIDs valid
S: * OK [UIDNEXT 201] Predicted next UID
S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
S: * OK [PERMANENTFLAGS (\Deleted \Seen \*)] Limited
S: * OK [HIGHESTMODSEQ 20010715194045007]
S: A142 OK [READ-WRITE] SELECT completed
After that, either:
C: A143 UID FETCH 1:* (FLAGS) (CHANGEDSINCE 20010715194032001)
S: * 2 FETCH (UID 6 MODSEQ (20010715205008000) FLAGS (\Deleted))
S: * 5 FETCH (UID 9 MODSEQ (20010715195517000) FLAGS ($NoJunk
$AutoJunk $MDNSent))
...
S: A143 OK FETCH completed
or:
C: A143 UID SEARCH MODSEQ 20010715194032001 UID 1:20
S: * SEARCH 6 9 11 12 18 19 20 23 (MODSEQ 20010917162500)
S: A143 OK Search complete
C: A144 UID SEARCH 1:20
S: * SEARCH 6 9 ...
S: A144 OK FETCH completed
7. Security Considerations
It is believed that this document does not raise any new security
concerns that are not already present in the base [IMAP4] protocol,
and these issues are discussed in [IMAP4]. Additional security
considerations may be found in different extensions mentioned in this
document; in particular, in [UIDPLUS], [LITERAL+], [CONDSTORE],
[MULTIAPPEND], and [UNSELECT].
Implementers are also reminded about the importance of thorough
testing.
8. References
8.1. Normative References
[KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[IMAP4] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL -
VERSION 4rev1", RFC 3501, March 2003.
[UIDPLUS] Crispin, M., "Internet Message Access Protocol (IMAP) -
UIDPLUS extension", RFC 4315, December 2005.
[LITERAL+] Myers, J., "IMAP4 non-synchronizing literals", RFC
2088, January 1997.
[CONDSTORE] Melnikov, A. and S. Hole, "IMAP Extension for
Conditional STORE Operation or Quick Flag Changes
Resynchronization", RFC 4551, June 2006.
[MULTIAPPEND] Crispin, M., "Internet Message Access Protocol (IMAP) -
MULTIAPPEND Extension", RFC 3502, March 2003.
[UNSELECT] Melnikov, A., "Internet Message Access Protocol (IMAP)
UNSELECT command", RFC 3691, February 2004.
[RFC2683] Leiba, B., "IMAP4 Implementation Recommendations", RFC
2683, September 1999.
8.2. Informative References
[ACL] Melnikov, A., "IMAP4 Access Control List (ACL)
Extension", RFC 4314, December 2005.
[IMAP-MODEL] Crispin, M., "Distributed Electronic Mail Models in
IMAP4", RFC 1733, December 1994.
9. Acknowledgements
This document is based on version 01 of the text written by Rob
Austein in November 1994.
The editor appreciates comments posted by Mark Crispin to the IMAP
mailing list and the comments/corrections/ideas received from Grant
Baillie, Cyrus Daboo, John G. Myers, Chris Newman, and Timo Sirainen.
The editor would also like to thank the developers of Netscape
Messenger and Mozilla mail clients for providing examples of
disconnected mail clients that served as a base for many
recommendations in this document.
Editor’s Address
Alexey Melnikov
Isode Limited
5 Castle Business Village
36 Station Road
Hampton, Middlesex
TW12 2BX
United Kingdom
Phone: +44 77 53759732
EMail: alexey.melnikov@isode.com
Full Copyright Statement
Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Acknowledgement
Funding for the RFC Editor function is provided by the IETF
Administrative Support Activity (IASA).