2. There is actually some uncertainty as to whether or not the "+"
characters requires escaping in MAILTO URLs (the standards are
not precisely clear). But, since any character in the address
specification may optionally be encoded, it is probably safer to
encode it.
3. The "/" character is generally reserved in URLs, and must be
encoded as %2F.
4. Neither the "$" nor the "_" character are given any special
interpretation in MAILTO URLs, so need not be encoded.
5. While the "!" character has no special interpretation, the "%"
character is used to introduce encoded sequences and hence it
must always be encoded.
4.4. Guessing domain names in web contexts
Several web browsers have adopted a practice that permits an
incomplete domain name to be used as input instead of a complete URL.
This has, for example, permitted users to type "microsoft" and have
the browser interpret the input as "http://www.microsoft.com/".
Other browser versions have gone even further, trying to build DNS
names up through a series of heuristics, testing each variation in
turn to see if it appears in the DNS, and accepting the first one
found as the intended domain name. Still, others automatically
invoke search engines if no period appears or if the reference fails.
If any of these approaches are to be used, it is often critical that
the browser recognize the complete list of TLDs. If an incomplete
list is used, complete domain names may not be recognized as such and
the system may try to turn them into completely different names. For
example, "example.aero" is a fully-qualified name, since "AERO." is a
TLD name. But, if the system doesn’t recognize "AERO" as a TLD name,
it is likely to try to look up "example.aero.com" and
"www.example.aero.com" (and then fail or find the wrong host), rather
than simply looking up the user-supplied name.
As discussed in Section 2 above, there are dangers associated with
software that attempts to "know" the list of top-level domain names
locally and take advantage of that knowledge. These name-guessing
heuristics are another example of that situation: if the lists are
up-to-date and used carefully, the systems in which they are embedded
may provide an easier, and more attractive, experience for at least
some users. But finding the wrong host, or being unable to find a
host even when its name is precisely known, constitute bad
experiences by any measure.
More generally, there have been bad experiences with attempts to
"complete" domain names by adding additional information to them.
These issues are described in some detail in RFC 1535 [RFC1535].
5. Implications of internationalization
The IETF has adopted a series of proposals ([RFC3490] - [RFC3492])
whose purpose is to permit encoding internationalized (i.e., non-
ASCII) names in the DNS. The primary standard, and the group
generically, are known as "IDNA". The actual strings stored in the
DNS are in an encoded form: the labels begin with the characters
"xn--" followed by the encoded string. Applications should be
prepared to accept and process the encoded form (those strings are
consistent with the "LDH rule" (see Section 2) so should not raise
any separate issues) and the use of local, and potentially other,
characters as appropriate to local systems and circumstances.
The IDNA specification describes the exact process to be used to
validate a name or encoded string. The process is sufficiently
complex that shortcuts or heuristics, especially for versions of
labels written directly in Unicode or other coded character sets, are
likely to fail and cause problems. In particular, the strings cannot
be validated with syntax or semantic rules of any of the usual sorts:
syntax validity is defined only in terms of the result of executing a
particular function.
In addition to the restrictions imposed by the protocols themselves,
many domains are implementing rules about just which non-ASCII names
they will permit to be registered (see, e.g., [JET], [RegRestr]).
This work is still relatively new, and the rules and conventions are
likely to be different for each domain, or at least each language or
script group. Attempting to test for those rules in a client program
to see if a user-supplied name might possibly exist in the relevant
domain would almost certainly be ill-advised.
One quick local test however, may be reasonable: as of the time of
this writing, there should be no instances of labels in the DNS that
start with two characters, followed by two hyphens, where the two
characters are not "xn" (in, of course, either upper or lower case).
Such label strings, if they appear, are probably erroneous or
obsolete, and it may be reasonable to at least warn the user about
them.
There is ongoing work in the IETF and elsewhere to define
internationalized formats for use in other protocols, including email
addresses. Those forms may or may not conform to existing rules for
ASCII-only identifiers; anyone designing evaluators or filters should
watch that work closely.
6. Summary
When an application accepts a string from the user and ultimately
passes it on to an API for a protocol, the desirability of testing or
filtering the text in any way not required by the protocol itself is
hotly debated. If it must divide the string into its components, or
otherwise interpret it, it obviously must make at least enough tests
to validate that process. With, e.g., domain names or email
addresses that can be passed on untouched, the appropriateness of
trying to figure out which ones are valid and which ones are not
requires a more complex decision, one that should include
considerations of how to make exactly the correct tests and to keep
information that changes and evolves up-to-date. A test containing
obsolete information, can be extremely frustrating for potential
correspondents or customers and may harm desired relationships.
7. Security Considerations
Since this document merely summarizes the requirements of existing
standards, it does not introduce any new security issues. However,
many of the techniques that motivate the document raise important
security concerns of their own. Rejecting valid forms of domain
names, email addresses, or URIs often denies service to the user of
those entities. Worse, guessing at the user’s intent when an
incomplete address, or other string, is given can result in
compromises to privacy or accuracy of reference if the wrong target
is found and returned. From a security standpoint, the optimum
behavior is probably to never guess, but instead, to force the user
to specify exactly what is wanted. When that position involves a
tradeoff with an acceptable user experience, good judgment should be
used and the fact that it is a tradeoff recognized.
Some characters have special or privileged meanings on some systems
(i.e., ` on Unix). Applications should be careful to escape those
locally if necessary. By the same token, they are valid, and should
not be disallowed locally, or escaped when transmitted through
Internet protocols, for such reasons if a remote site chooses to use
them.
The presence of local checking does not permit remote checking to be
bypassed. Note that this can apply to a single machine; in
particular, a local MTA should not assume that a local MUA has
properly escaped locally-significant special characters.
8. Acknowledgements
The author would like to express his appreciation for helpful
comments from Harald Alvestrand, Eric A. Hall, and the RFC Editor,
and for partial support of this work from SITA. Responsibility for
any errors remains, of course, with the author.
The first Internet-Draft on this subject was posted in February 2003.
The document was submitted to the RFC Editor on 20 June 2003,
returned for revisions on 19 August, and resubmitted on 5 September
2003.
9. References
9.1. Normative References
[RFC1035] Mockapetris, P., "Domain names - implementation and
specification", STD 13, RFC 1035, November 1987.
[RFC1123] Braden, R., Ed., "Requirements for Internet Hosts -
Application and Support", STD 3, RFC 1123, October
1989.
[RFC1535] Gavron, E., "A Security Problem and Proposed
Correction With Widely Deployed DNS Software", RFC
1535, October 1993.
[RFC1738] Berners-Lee, T., Masinter, L. and M. McCahill,
"Uniform Resource Locators (URL)", RFC 1738, December
1994.
[RFC2181] Elz, R. and R. Bush, "Clarifications to the DNS
Specification", RFC 2181, July 1997.
[RFC2368] Hoffman, P., Masinter, L. and J. Zawinski, "The
mailto URL scheme", RFC 2368, July 1998.
[RFC2396] Berners-Lee, T., Fielding, R. and L. Masinter,
"Uniform Resource Identifiers (URI): Generic Syntax",
RFC 2396, August 1998.
[RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
Masinter, L., Leach, P. and T. Berners-Lee,
"Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616,
June 1999.
[RFC2821] Klensin, J., Ed., "Simple Mail Transfer Protocol",
RFC 2821, April 2001.
[RFC2822] Resnick, P., Ed., "Internet Message Format", RFC
2822, April 2001.
[RFC3490] Faltstrom, P., Hoffman, P. and A. Costello,
"Internationalizing Domain Names in Applications
(IDNA)", RFC 3490, March 2003.
[RFC3491] Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep
Profile for Internationalized Domain Names (IDN)",
RFC 3491, March 2003.
[RFC3492] Costello, A., "Punycode: A Bootstring encoding of
Unicode for Internationalized Domain Names in
Applications (IDNA)", RFC 3492, March 2003.
[ASCII] American National Standards Institute (formerly
United States of America Standards Institute), "USA
Code for Information Interchange", ANSI X3.4-1968.
ANSI X3.4-1968 has been replaced by newer versions
with slight modifications, but the 1968 version
remains definitive for the Internet.
[DomainList] Internet Assigned Numbers Authority (IANA), Untitled
alphabetical list of current top-level domains.
http://data.iana.org/TLD/tlds-alpha-by-domain.txt
ftp://data.iana.org/TLD/tlds-alpha-by-domain.txt
9.2. Informative References
[ISO.3166.1988] International Organization for Standardization,
"Codes for the representation of names of countries,
3rd edition", ISO Standard 3166, August 1988.
[JET] Konishi, K., et al., "Internationalized Domain Names
Registration and Administration Guideline for
Chinese, Japanese and Korean", Work in Progress.
[RFC1591] Postel, J., "Domain Name System Structure and
Delegation", RFC 1591, March 1994.
[RegRestr] Klensin, J., "Registration of Internationalized
Domain Names: Overview and Method", Work in Progress,
February 2004.
10. Author’s Address
John C Klensin
1770 Massachusetts Ave, #322
Cambridge, MA 02140
USA
Phone: +1 617 491 5735
EMail: john-ietf@jck.com
11. Full Copyright Statement
Copyright (C) The Internet Society (2004). This document is subject
to the rights, licenses and restrictions contained in BCP 78 and
except as set forth therein, the authors retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at ietf-
ipr@ietf.org.
Acknowledgement
Funding for the RFC Editor function is currently provided by the
Internet Society.