Request for Comments: 4690 P. Faltstrom
Category: Informational Cisco Systems
C. Karp
Swedish Museum of Natural History
IAB
September 2006
Review and Recommendations for Internationalized Domain Names (IDNs)
Status of This Memo
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2006).
Abstract
This note describes issues raised by the deployment and use of
Internationalized Domain Names. It describes problems both at the
time of registration and for use of those names in the DNS. It
recommends that IETF should update the RFCs relating to IDNs and a
framework to be followed in doing so, as well as summarizing and
identifying some work that is required outside the IETF. In
particular, it proposes that some changes be investigated for the
Internationalizing Domain Names in Applications (IDNA) standard and
its supporting tables, based on experience gained since those
standards were completed.
Table of Contents
1. Introduction ....................................................3
1.1. The Role of IDNs and This Document .........................3
1.2. Status of This Document and Its Recommendations ............4
1.3. The IDNA Standard ..........................................4
1.4. Unicode Documents ..........................................5
1.5. Definitions ................................................5
1.5.1. Language ............................................6
1.5.2. Script ..............................................6
1.5.3. Multilingual ........................................6
1.5.4. Localization ........................................7
1.5.5. Internationalization ................................7
1.6. Statements and Guidelines ..................................7
1.6.1. IESG Statement ......................................8
1.6.2. ICANN Statements ....................................8
2. General Problems and Issues ....................................11
2.1. User Conceptions, Local Character Sets, and Input issues ..11
2.2. Examples of Issues ........................................13
2.2.1. Language-Specific Character Matching ...............13
2.2.2. Multiple Scripts ...................................13
2.2.3. Normalization and Character Mappings ...............14
2.2.4. URLs in Printed Form ...............................16
2.2.5. Bidirectional Text .................................17
2.2.6. Confusable Character Issues ........................17
2.2.7. The IESG Statement and IDNA issues .................19
3. Migrating to New Versions of Unicode ...........................20
3.1. Versions of Unicode .......................................20
3.2. Version Changes and Normalization Issues ..................21
3.2.1. Unnormalized Combining Sequences ...................21
3.2.2. Combining Characters and Character Components ......22
3.2.3. When does normalization occur? .....................23
4. Framework for Next Steps in IDN Development ....................24
4.1. Issues within the Scope of the IETF .......................24
4.1.1. Review of IDNA .....................................24
4.1.2. Non-DNS and Above-DNS Internationalization
Approaches .........................................25
4.1.3. Security Issues, Certificates, etc. ................25
4.1.4. Protocol Changes and Policy Implications ...........27
4.1.5. Non-US-ASCII in Local Part of Email Addresses ......27
4.1.6. Use of the Unicode Character Set in the IETF .......27
4.2. Issues That Fall within the Purview of ICANN ..............28
4.2.1. Dispute Resolution .................................28
4.2.2. Policy at Registries ...............................28
4.2.3. IDNs at the Top Level of the DNS ...................29
5. Specific Recommendations for Next Steps ........................29
5.1. Reduction of Permitted Character List .....................29
5.1.1. Elimination of All Non-Language Characters .........30
5.1.2. Elimination of Word-Separation Punctuation .........30
5.2. Updating to New Versions of Unicode .......................30
5.3. Role and Uses of the DNS ..................................31
5.4. Databases of Registered Names .............................31
6. Security Considerations ........................................31
7. Acknowledgements ...............................................32
8. References .....................................................32
8.1. Normative References ......................................32
8.2. Informative References ....................................33
1. Introduction
1.1. The Role of IDNs and This Document
While IDNs have been advocated as the solution to a wide range of
problems, this document is written from the perspective that they are
no more and no less than DNS names, reflecting the same requirements
for use, stability, and accuracy as traditional "hostnames", but
using a much larger collection of permitted characters. In
particular, while IDNs represent a step toward an Internet that is
equally accessible from all languages and scripts, they, at best,
address only a small part of that very broad objective. There has
been controversy since IDNs were first suggested about how important
they will actually turn out to be; that controversy will probably
continue. Accessibility from all languages is an important
objective, hence it is important that our standards and definitions
for IDNs be smoothly adaptable to additional scripts as they are
added to the Unicode character set.
The utility of IDNs must be evaluated in terms of their application
by users and in protocols: the ability to simply put a name into the
DNS and retrieve it is not, in and of itself, important. From this
point of view, IDNs will be useful and effective if they provide
stable and predictable references -- references that are no less
stable and predictable, and no less secure, than their ASCII
counterparts.
This combination of objectives and criteria has proven very difficult
to satisfy. Experience in developing the IDNA standard and during
the initial years of its implementation and deployment suggests that
it may be impossible to fully satisfy all of them and that
engineering compromises are needed to yield a result that is
workable, even if not completely satisfactory. Based on that
experience and issues that have been raised, it is now appropriate to
review some of the implications of IDNs, the decisions made in
defining them, and the foundation on which they rest and determine
whether changes are needed and, if so, which ones.
The design of the DNS itself imposes some additional constraints. If
the DNS is to remain globally interoperable, there are specific
characteristics that no implementation of IDNs, or the DNS more
generally, can change. For example, because the DNS is a global
hierarchal administrative namespace with only a single name at any
given node, there is one and only one owner of each domain name.
Also, when strings are looked up in the DNS, positive responses can
only reflect exact matches: if there is no exact match, then one gets
an error reply, not a list of near matches or other supplemental
information. Searches and approximate matchings are not possible.
Finally, because the DNS is a distributed system where any server
might cache responses, and later use those cached responses to
attempt to satisfy queries before a global lookup is done, every
server must use the same matching criteria.
1.2. Status of This Document and Its Recommendations
This document reviews the IDN landscape from an IETF perspective and
presents the recommendations and conclusions of the IAB, based
partially on input from an ad hoc committee charged with reviewing
IDN issues and the path forward (see Section 7). Its recommendations
are advice to the IETF, or in a few cases to other bodies, for topics
to be investigated and actions to be taken if those bodies, after
their examinations, consider those actions appropriate.
1.3. The IDNA Standard
During 2002, the IETF completed the following RFCs that, together,
define IDNs:
RFC 3454 Preparation of Internationalized Strings ("Stringprep")
[RFC3454].
Stringprep is a generic mechanism for taking a Unicode string and
converting it into a canonical format. Stringprep itself is just
a collection of rules, tables, and operations. Any protocol or
algorithm that uses it must define a "Stringprep profile", which
specifies which of those rules are applied, how, and with which
characteristics.
RFC 3490 Internationalizing Domain Names in Applications (IDNA)
[RFC3490].
IDNA is the base specification in this group. It specifies that
Nameprep is used as the Stringprep profile for domain names, and
that Punycode is the relevant encoding mechanism for use in
generating an ASCII-compatible ("ACE") form of the name. It also
applies some additional conversions and character filtering that
are not part of Nameprep.
RFC 3491 Nameprep: A Stringprep Profile for Internationalized Domain
Names (IDN) [RFC3491].
Nameprep is designed to meet the specific needs of IDNs and, in
particular, to support case-folding for scripts that support what
are traditionally known as upper- and lowercase forms of the same
letters. The result of the Nameprep algorithm is a string
containing a subset of the Unicode Character set, normalized and
case-folded so that case-insensitive comparison can be made.
RFC 3492 Punycode: A Bootstring encoding of Unicode for
Internationalized Domain Names in Applications (IDNA) [RFC3492].
Punycode is a mechanism for encoding a Unicode string in ASCII
characters. The characters used are the same the subset of
characters that are allowed in the hostname definition of DNS,
i.e., the "letter, digit, and hyphen" characters, sometimes known
as "LDH".
1.4. Unicode Documents
Unicode is used as the base, and defining, character set for IDNs.
Unicode is standardized by the Unicode Consortium, and synchronized
with ISO to create ISO/IEC 10646 [ISO10646]. At the time the RFCs
mentioned earlier were created, Unicode was at Version 3.2. For
reasons explained later, it was necessary to pick a particular,
then-current, version of Unicode when IDNA was adopted.
Consequently, the RFCs are explicitly dependent on Unicode Version
3.2 [Unicode32]. There is, at present, no established mechanism for
modifying the IDNA RFCs to use newer Unicode versions (see
Section 3.1).
Unicode is a very large and complex character set. (The term
"character set" or "charset" is used in a way that is peculiar to the
IETF and may not be the same as the usage in other bodies and
contexts.) The Unicode Standard and related documents are created
and maintained by the Unicode Technical Committee (UTC), one of the
committees of the Unicode Consortium.
The Consortium first published The Unicode Standard [Unicode10] in
1991, and continues to develop standards based on that original work.
Unicode is developed in conjunction with the International
Organization for Standardization, and it shares its character
repertoire with ISO/IEC 10646. Unicode and ISO/IEC 10646 function
equivalently as character encodings, but The Unicode Standard
contains much more information for implementers, covering -- in depth
-- topics such as bitwise encoding, collation, and rendering. The
Unicode Standard enumerates a multitude of character properties,
including those needed for supporting bidirectional text. The
Unicode Consortium and ISO standards do use slightly different
terminology.
1.5. Definitions
The following terms and their meanings are critical to understanding
the rest of this document and to discussions of IDNs more generally.
These terms are derived from [RFC3536], which contains additional
discussion of some of them.
1.5.1. Language
A language is a way that humans interact. The use of language occurs
in many forms, including speech, writing, and signing.
Some languages have a close relationship between the written and
spoken forms, while others have a looser relationship. RFC 3066
[RFC3066] discusses languages in more detail and provides identifiers
for languages for use in Internet protocols. Computer languages are
explicitly excluded from this definition. The most recent IETF work
in this area, and on script identification (see below), is documented
in [RFC4645] and [RFC4646].
1.5.2. Script
A script is a set of graphic characters used for the written form of
one or more languages. This definition is the one used in
[ISO10646].
Examples of scripts are Arabic, Cyrillic, Greek, Han (the so-called
ideographs used in writing Chinese, Japanese, and Korean), and
"Latin". Arabic, Greek, and Latin are, of course, also names of
languages.
Historically, the script that is known as "Latin" in Unicode and most
contexts associated with information technology standards is known in
the linguistic community as "Roman" or "Roman-derived". The latter
terminology distinguishes between the Latin language and the
characters used to write it, especially in Republican times, from the
much richer and more decorated script derived and adapted from those
characters. Since IDNA is defined using Unicode and that standard
used the term "LATIN" in its character names and descriptions, that
terminology will be used in this document as well except when
"Roman-derived" is needed for clarity. However, readers approaching
this document from a cultural or linguistic standpoint should be
aware that the use of, or references to, "Latin script" in this
document refers to the entire collection of Roman-derived characters,
not just the characters used to write the Latin language. Some other
issues with script identification and relationships with other
standards are discussed in [RFC4646].
1.5.3. Multilingual
The term "multilingual" has many widely-varying definitions and thus
is not recommended for use in standards. Some of the definitions
relate to the ability to handle international characters; other
definitions relate to the ability to handle multiple charsets; and
still others relate to the ability to handle multiple languages.
While this term has been deprecated for IETF-related uses and does
not otherwise appear in this document, a discussion here seemed
appropriate since the term is still widely used in some discussions
of IDNs.
1.5.4. Localization
Localization is the process of adapting an internationalized
application platform or application to a specific cultural
environment. In localization, the same semantics are preserved while
the syntax or presentation forms may be changed.
Localization is the act of tailoring an application for a different
language or script or culture. Some internationalized applications
can handle a wide variety of languages. Typical users understand
only a small number of languages, so the program must be tailored to
interact with users in just the languages they know.
Somewhat different definitions for localization and
internationalization (see below) are used by groups other than the
IETF. See [W3C-Localization] for one example.
1.5.5. Internationalization
In the IETF, the term "internationalization" is used to describe
adding or improving the handling of non-ASCII text in a protocol.
Other bodies use the term in other ways, often with subtle variation
in meaning. The term "internationalization" is often abbreviated
"i18n" (and localization as "l10n").
Many protocols that handle text only handle the characters associated
with one script (often, a subset of the characters used in writing
English text), or leave the question of what character set is used up
to local guesswork (which leads to interoperability problems).
Adding non-ASCII text to such a protocol allows the protocol to
handle more scripts, with the intention of being able to include all
of the scripts that are useful in the world. It is naive (sic) to
believe that all English words can be written in ASCII, various
mythologies notwithstanding.
1.6. Statements and Guidelines
When the IDNA RFCs were published, the IESG and ICANN made statements
that were intended to guide deployment and future work. In recent
months, ICANN has updated its statement and others have also made
contributions. It is worth noting that the quality of understanding
of internationalization issues as applied to the DNS has evolved
considerably over the last few years. Organizations that took
specific positions a year or more ago might not make exactly the same
statements today.
1.6.1. IESG Statement
The IESG made a statement on IDNA [IESG-IDN]:
IDNA, through its requirement of Nameprep [RFC3491], uses
equivalence tables that are based only on the characters
themselves; no attention is paid to the intended language (if any)
for the domain name. However, for many domain names, the intended
language of one or more parts of the domain name actually does
matter to the users.
Similarly, many names cannot be presented and used without
ambiguity unless the scripts to which their characters belong are
known. In both cases, this additional information should be of
concern to the registry.
The statement is longer than this, but these paragraphs are the
important ones. The rest of the statement consists of explanations
and examples.
1.6.2. ICANN Statements
1.6.2.1. Initial ICANN Guidelines
Soon after the IDNA standards were adopted, ICANN produced an initial
version of its "IDN Guidelines" [ICANNv1]. This document was
intended to serve two purposes. The first was to provide a basis for
releasing the Generic Top Level Domain (gTLD) registries that had
been established by ICANN from a contractual restriction on the
registration of labels containing hyphens in the third and fourth
positions. The second was to provide a general framework for the
development of registry policies for the implementation of IDNs.
One of the key components of this framework prescribed strict
compliance with RFCs 3490, 3491, and 3492. With the framework, ICANN
specified that IDNA was to be the sole mechanism to be used in the
DNS to represent IDNs.
Limitations on the characters available for inclusion in IDNs were
mandated by two mechanisms. The first was by requiring an
"inclusion-based approach (meaning that code points that are not
explicitly permitted by the registry are prohibited) for identifying
permissible
code points from among the full Unicode repertoire." The second
mechanism required the association of every IDN with a specific
language, with additional policies also being language based:
"In implementing the IDN standards, top-level domain registries will
(a) associate each registered internationalized domain name with one
language or set of languages,
(b) employ language-specific registration and administration rules
that are documented and publicly available, such as the reservation
of all domain names with equivalent character variants in the
languages associated with the registered domain name, and,
(c) where the registry finds that the registration and administration
rules for a given language would benefit from a character variants
table, allow registrations in that language only when an appropriate
table is available. ... In implementing the IDN standards, top-level
domain registries should, at least initially, limit any given domain
label (such as a second-level domain name) to the characters
associated with one language or set of languages only."
It was left to each TLD registry to define the character repertoire
it would associate with any given language. This led to significant
variation from registry to registry, with further heterogeneity in
the underlying language-based IDN policies. If the guidelines had
made provision for IDN policies also being based on script, a
substantial amount of the resulting ambiguity could have been
avoided. However, they did not, and the sequence of events leading
to the present review of IDNA was thus triggered.
1.6.2.2. ICANN Version 2 Guidelines
One of the responses of the TLD registries to what was widely
perceived as a crisis situation was to invoke the mechanism described
in the initial guidelines: "As the deployment of IDNs proceeds, ICANN
and the IDN registries will review these Guidelines at regular
intervals, and revise them as necessary based on experience."
The pivotal requirement was the modification of the guidelines to
permit script-based policies for IDNs. Further concern was expressed
about the need for realistically implementable mechanisms for the
propagation of TLD registry policies into the lower levels of their
name trees. In addition to the anticipated increase of constraint on
the protocol level, one obvious additional approach would be to
replace the guidelines by an instrument that itself had clear status
in the IETF’s normative framework. A BCP was therefore seen as the
appropriate focus for longer-term effort. The most pressing issues
would be dealt with in the interim by incremental modification to the
guidelines, but no need was seen for the detailed further development
of those guidelines once that incremental modification was complete.
The outcome of this action was a version 2.0 of the guidelines
[ICANNv2], which was endorsed by the ICANN Board on November 8, 2005
for a period of nine months. The Board stated further that it "tasks
the IDN working group to continue its important work and return to
the board with specific IDN improvement recommendations before the
ICANN Meeting in Morocco" and "supports the working group’s continued
action to reframe the guidelines completely in a manner appropriate
for further development as a Best Current Practices (BCP) document,
to ensure that the Guideline directions will be used deeper into the
DNS hierarchy and within TLD’s where ICANN has a lesser policy
relationship."
Retaining the inclusion-based approach established in version 1.0,
the crucial addition to the policy framework is that:
"All code points in a single label will be taken from the same script
as determined by the Unicode Standard Annex #24: Script Names at
http://www.unicode.org/reports/tr24. Exception to this is
permissible for languages with established orthographies and
conventions that require the commingled use of multiple scripts. In
such cases, visually confusable characters from different scripts
will not be allowed to coexist in a single set of permissible
codepoints unless a corresponding policy and character table is
clearly defined."
Additionally:
"Permissible code points will not include: (a) line symbol-drawing
characters (as those in the Unicode Box Drawing block), (b) symbols
and icons that are neither alphanumeric nor ideographic language
characters, such as typographic and pictographic dingbats, (c)
characters with well-established functions as protocol elements, (d)
punctuation marks used solely to indicate the structure of
sentences."
Attention has been called to several points that are not adequately