Request for Comments: 3987 W3C
Category: Standards Track M. Suignard
Microsoft Corporation
January 2005
Internationalized Resource Identifiers (IRIs)
Status of This Memo
This document specifies an Internet standards track protocol for the
Internet community, and requests discussion and suggestions for
improvements. Please refer to the current edition of the "Internet
Official Protocol Standards" (STD 1) for the standardization state
and status of this protocol. Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2005).
Abstract
This document defines a new protocol element, the Internationalized
Resource Identifier (IRI), as a complement to the Uniform Resource
Identifier (URI). An IRI is a sequence of characters from the
Universal Character Set (Unicode/ISO 10646). A mapping from IRIs to
URIs is defined, which means that IRIs can be used instead of URIs,
where appropriate, to identify resources.
The approach of defining a new protocol element was chosen instead of
extending or changing the definition of URIs. This was done in order
to allow a clear distinction and to avoid incompatibilities with
existing software. Guidelines are provided for the use and
deployment of IRIs in various protocols, formats, and software
components that currently deal with URIs.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Overview and Motivation . . . . . . . . . . . . . . . . 3
1.2. Applicability . . . . . . . . . . . . . . . . . . . . . 3
1.3. Definitions . . . . . . . . . . . . . . . . . . . . . . 4
1.4. Notation . . . . . . . . . . . . . . . . . . . . . . . . 5
2. IRI Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1. Summary of IRI Syntax . . . . . . . . . . . . . . . . . 6
2.2. ABNF for IRI References and IRIs . . . . . . . . . . . . 7
3. Relationship between IRIs and URIs . . . . . . . . . . . . . . 10
3.1. Mapping of IRIs to URIs . . . . . . . . . . . . . . . . 10
3.2. Converting URIs to IRIs . . . . . . . . . . . . . . . . 14
3.2.1. Examples . . . . . . . . . . . . . . . . . . . . 15
4. Bidirectional IRIs for Right-to-Left Languages. . . . . . . . 16
4.1. Logical Storage and Visual Presentation . . . . . . . . 17
4.2. Bidi IRI Structure . . . . . . . . . . . . . . . . . . . 18
4.3. Input of Bidi IRIs . . . . . . . . . . . . . . . . . . . 19
4.4. Examples . . . . . . . . . . . . . . . . . . . . . . . . 19
5. Normalization and Comparison . . . . . . . . . . . . . . . . . 21
5.1. Equivalence . . . . . . . . . . . . . . . . . . . . . . 22
5.2. Preparation for Comparison . . . . . . . . . . . . . . . 22
5.3. Comparison Ladder . . . . . . . . . . . . . . . . . . . 23
5.3.1. Simple String Comparison . . . . . . . . . . . . 23
5.3.2. Syntax-Based Normalization . . . . . . . . . . . 24
5.3.3. Scheme-Based Normalization . . . . . . . . . . . 27
5.3.4. Protocol-Based Normalization . . . . . . . . . . 28
6. Use of IRIs . . . . . . . . . . . . . . . . . . . . . . . . . 29
6.1. Limitations on UCS Characters Allowed in IRIs . . . . . 29
6.2. Software Interfaces and Protocols . . . . . . . . . . . 29
6.3. Format of URIs and IRIs in Documents and Protocols . . . 30
6.4. Use of UTF-8 for Encoding Original Characters .. . . . . 30
6.5. Relative IRI References . . . . . . . . . . . . . . . . 32
7. URI/IRI Processing Guidelines (informative) . . . . . . . . . 32
7.1. URI/IRI Software Interfaces . . . . . . . . . . . . . . 32
7.2. URI/IRI Entry . . . . . . . . . . . . . . . . . . . . . 33
7.3. URI/IRI Transfer between Applications . . . . . . . . . 33
7.4. URI/IRI Generation . . . . . . . . . . . . . . . . . . . 34
7.5. URI/IRI Selection . . . . . . . . . . . . . . . . . . . 34
7.6. Display of URIs/IRIs . . . . . . . . . . . . . . . . . . 35
7.7. Interpretation of URIs and IRIs . . . . . . . . . . . . 36
7.8. Upgrading Strategy . . . . . . . . . . . . . . . . . . . 36
8. Security Considerations . . . . . . . . . . . . . . . . . . . 37
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 39
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 40
10.1. Normative References . . . . . . . . . . . . . . . . . . 40
10.2. Informative References . . . . . . . . . . . . . . . . . 41
A. Design Alternatives . . . . . . . . . . . . . . . . . . . . . 44
A.1. New Scheme(s) . . . . . . . . . . . . . . . . . . . . . 44
A.2. Character Encodings Other Than UTF-8 . . . . . . . . . . 44
A.3. New Encoding Convention . . . . . . . . . . . . . . . . 44
A.4. Indicating Character Encodings in the URI/IRI . . . . . 45
Authors’ Addresses . . . . . . . . . . . . . . . . . . . . . . . . 45
Full Copyright Statement . . . . . . . . . . . . . . . . . . . . . 46
1. Introduction
1.1. Overview and Motivation
A Uniform Resource Identifier (URI) is defined in [RFC3986] as a
sequence of characters chosen from a limited subset of the repertoire
of US-ASCII [ASCII] characters.
The characters in URIs are frequently used for representing words of
natural languages. This usage has many advantages: Such URIs are
easier to memorize, easier to interpret, easier to transcribe, easier
to create, and easier to guess. For most languages other than
English, however, the natural script uses characters other than A -
Z. For many people, handling Latin characters is as difficult as
handling the characters of other scripts is for those who use only
the Latin alphabet. Many languages with non-Latin scripts are
transcribed with Latin letters. These transcriptions are now often
used in URIs, but they introduce additional ambiguities.
The infrastructure for the appropriate handling of characters from
local scripts is now widely deployed in local versions of operating
system and application software. Software that can handle a wide
variety of scripts and languages at the same time is increasingly
common. Also, increasing numbers of protocols and formats can carry
a wide range of characters.
This document defines a new protocol element called Internationalized
Resource Identifier (IRI) by extending the syntax of URIs to a much
wider repertoire of characters. It also defines "internationalized"
versions corresponding to other constructs from [RFC3986], such as
URI references. The syntax of IRIs is defined in section 2, and the
relationship between IRIs and URIs in section 3.
Using characters outside of A - Z in IRIs brings some difficulties.
Section 4 discusses the special case of bidirectional IRIs, section 5
various forms of equivalence between IRIs, and section 6 the use of
IRIs in different situations. Section 7 gives additional informative
guidelines, and section 8 security considerations.
1.2. Applicability
IRIs are designed to be compatible with recommendations for new URI
schemes [RFC2718]. The compatibility is provided by specifying a
well-defined and deterministic mapping from the IRI character
sequence to the functionally equivalent URI character sequence.
Practical use of IRIs (or IRI references) in place of URIs (or URI
references) depends on the following conditions being met:
a. A protocol or format element should be explicitly designated to
be able to carry IRIs. The intent is not to introduce IRIs into
contexts that are not defined to accept them. For example, XML
schema [XMLSchema] has an explicit type "anyURI" that includes
IRIs and IRI references. Therefore, IRIs and IRI references can
be in attributes and elements of type "anyURI". On the other
hand, in the HTTP protocol [RFC2616], the Request URI is defined
as a URI, which means that direct use of IRIs is not allowed in
HTTP requests.
b. The protocol or format carrying the IRIs should have a mechanism
to represent the wide range of characters used in IRIs, either
natively or by some protocol- or format-specific escaping
mechanism (for example, numeric character references in [XML1]).
c. The URI corresponding to the IRI in question has to encode
original characters into octets using UTF-8. For new URI
schemes, this is recommended in [RFC2718]. It can apply to a
whole scheme (e.g., IMAP URLs [RFC2192] and POP URLs [RFC2384],
or the URN syntax [RFC2141]). It can apply to a specific part of
a URI, such as the fragment identifier (e.g., [XPointer]). It
can apply to a specific URI or part(s) thereof. For details,
please see section 6.4.
1.3. Definitions
The following definitions are used in this document; they follow the
terms in [RFC2130], [RFC2277], and [ISO10646].
character: A member of a set of elements used for the organization,
control, or representation of data. For example, "LATIN CAPITAL
LETTER A" names a character.
octet: An ordered sequence of eight bits considered as a unit.
character repertoire: A set of characters (in the mathematical
sense).
sequence of characters: A sequence of characters (one after another).
sequence of octets: A sequence of octets (one after another).
character encoding: A method of representing a sequence of characters
as a sequence of octets (maybe with variants). Also, a method of
(unambiguously) converting a sequence of octets into a sequence of
characters.
charset: The name of a parameter or attribute used to identify a
character encoding.
UCS: Universal Character Set. The coded character set defined by
ISO/IEC 10646 [ISO10646] and the Unicode Standard [UNIV4].
IRI reference: Denotes the common usage of an Internationalized
Resource Identifier. An IRI reference may be absolute or
relative. However, the "IRI" that results from such a reference
only includes absolute IRIs; any relative IRI references are
resolved to their absolute form. Note that in [RFC2396] URIs did
not include fragment identifiers, but in [RFC3986] fragment
identifiers are part of URIs.
running text: Human text (paragraphs, sentences, phrases) with syntax
according to orthographic conventions of a natural language, as
opposed to syntax defined for ease of processing by machines
(e.g., markup, programming languages).
protocol element: Any portion of a message that affects processing of
that message by the protocol in question.
presentation element: A presentation form corresponding to a protocol
element; for example, using a wider range of characters.
create (a URI or IRI): With respect to URIs and IRIs, the term is
used for the initial creation. This may be the initial creation
of a resource with a certain identifier, or the initial exposition
of a resource under a particular identifier.
generate (a URI or IRI): With respect to URIs and IRIs, the term is
used when the IRI is generated by derivation from other
information.
1.4. Notation
RFCs and Internet Drafts currently do not allow any characters
outside the US-ASCII repertoire. Therefore, this document uses
various special notations to denote such characters in examples.
In text, characters outside US-ASCII are sometimes referenced by
using a prefix of ’U+’, followed by four to six hexadecimal digits.
To represent characters outside US-ASCII in examples, this document
uses two notations: ’XML Notation’ and ’Bidi Notation’.
XML Notation uses a leading ’&#x’, a trailing ’;’, and the
hexadecimal number of the character in the UCS in between. For
example, я stands for CYRILLIC CAPITAL LETTER YA. In this
notation, an actual ’&’ is denoted by ’&’.
Bidi Notation is used for bidirectional examples: Lowercase letters
stand for Latin letters or other letters that are written left to
right, whereas uppercase letters represent Arabic or Hebrew letters
that are written right to left.
To denote actual octets in examples (as opposed to percent-encoded
octets), the two hex digits denoting the octet are enclosed in "<"
and ">". For example, the octet often denoted as 0xc9 is denoted
here as <c9>.
In this document, the key words "MUST", "MUST NOT", "REQUIRED",
"SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY",
and "OPTIONAL" are to be interpreted as described in [RFC2119].
2. IRI Syntax
This section defines the syntax of Internationalized Resource
Identifiers (IRIs).
As with URIs, an IRI is defined as a sequence of characters, not as a
sequence of octets. This definition accommodates the fact that IRIs
may be written on paper or read over the radio as well as stored or
transmitted digitally. The same IRI may be represented as different
sequences of octets in different protocols or documents if these
protocols or documents use different character encodings (and/or
transfer encodings). Using the same character encoding as the
containing protocol or document ensures that the characters in the
IRI can be handled (e.g., searched, converted, displayed) in the same
way as the rest of the protocol or document.
2.1. Summary of IRI Syntax
IRIs are defined similarly to URIs in [RFC3986], but the class of
unreserved characters is extended by adding the characters of the UCS
(Universal Character Set, [ISO10646]) beyond U+007F, subject to the
limitations given in the syntax rules below and in section 6.1.
Otherwise, the syntax and use of components and reserved characters
is the same as that in [RFC3986]. All the operations defined in
[RFC3986], such as the resolution of relative references, can be
applied to IRIs by IRI-processing software in exactly the same way as
they are for URIs by URI-processing software.
Characters outside the US-ASCII repertoire are not reserved and
therefore MUST NOT be used for syntactical purposes, such as to
delimit components in newly defined schemes. For example, U+00A2,
CENT SIGN, is not allowed as a delimiter in IRIs, because it is in
the ’iunreserved’ category. This is similar to the fact that it is
not possible to use ’-’ as a delimiter in URIs, because it is in the
’unreserved’ category.
2.2. ABNF for IRI References and IRIs
Although it might be possible to define IRI references and IRIs
merely by their transformation to URI references and URIs, they can
also be accepted and processed directly. Therefore, an ABNF
definition for IRI references (which are the most general concept and
the start of the grammar) and IRIs is given here. The syntax of this
ABNF is described in [RFC2234]. Character numbers are taken from the
UCS, without implying any actual binary encoding. Terminals in the
ABNF are characters, not bytes.
The following grammar closely follows the URI grammar in [RFC3986],
except that the range of unreserved characters is expanded to include
UCS characters, with the restriction that private UCS characters can
occur only in query parts. The grammar is split into two parts:
Rules that differ from [RFC3986] because of the above-mentioned
expansion, and rules that are the same as those in [RFC3986]. For
rules that are different than those in [RFC3986], the names of the
non-terminals have been changed as follows. If the non-terminal
contains ’URI’, this has been changed to ’IRI’. Otherwise, an ’i’
has been prefixed.
The following rules are different from those in [RFC3986]:
IRI = scheme ":" ihier-part [ "?" iquery ]
[ "#" ifragment ]
ihier-part = "//" iauthority ipath-abempty
/ ipath-absolute
/ ipath-rootless
/ ipath-empty
IRI-reference = IRI / irelative-ref
absolute-IRI = scheme ":" ihier-part [ "?" iquery ]
irelative-ref = irelative-part [ "?" iquery ] [ "#" ifragment ]
irelative-part = "//" iauthority ipath-abempty
/ ipath-absolute
/ ipath-noscheme
/ ipath-empty
iauthority = [ iuserinfo "@" ] ihost [ ":" port ]
iuserinfo = *( iunreserved / pct-encoded / sub-delims / ":" )
ihost = IP-literal / IPv4address / ireg-name
ireg-name = *( iunreserved / pct-encoded / sub-delims )
ipath = ipath-abempty ; begins with "/" or is empty
/ ipath-absolute ; begins with "/" but not "//"
/ ipath-noscheme ; begins with a non-colon segment
/ ipath-rootless ; begins with a segment
/ ipath-empty ; zero characters
ipath-abempty = *( "/" isegment )
ipath-absolute = "/" [ isegment-nz *( "/" isegment ) ]
ipath-noscheme = isegment-nz-nc *( "/" isegment )
ipath-rootless = isegment-nz *( "/" isegment )
ipath-empty = 0<ipchar>
isegment = *ipchar
isegment-nz = 1*ipchar
isegment-nz-nc = 1*( iunreserved / pct-encoded / sub-delims
/ "@" )
; non-zero-length segment without any colon ":"
ipchar = iunreserved / pct-encoded / sub-delims / ":"
/ "@"
iquery = *( ipchar / iprivate / "/" / "?" )
ifragment = *( ipchar / "/" / "?" )
iunreserved = ALPHA / DIGIT / "-" / "." / "_" / "~" / ucschar
ucschar = %xA0-D7FF / %xF900-FDCF / %xFDF0-FFEF
/ %x10000-1FFFD / %x20000-2FFFD / %x30000-3FFFD
/ %x40000-4FFFD / %x50000-5FFFD / %x60000-6FFFD
/ %x70000-7FFFD / %x80000-8FFFD / %x90000-9FFFD
/ %xA0000-AFFFD / %xB0000-BFFFD / %xC0000-CFFFD
/ %xD0000-DFFFD / %xE1000-EFFFD
iprivate = %xE000-F8FF / %xF0000-FFFFD / %x100000-10FFFD
Some productions are ambiguous. The "first-match-wins" (a.k.a.
"greedy") algorithm applies. For details, see [RFC3986].
The following rules are the same as those in [RFC3986]:
scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )
port = *DIGIT
IP-literal = "[" ( IPv6address / IPvFuture ) "]"
IPvFuture = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" )
IPv6address = 6( h16 ":" ) ls32
/ "::" 5( h16 ":" ) ls32
/ [ h16 ] "::" 4( h16 ":" ) ls32
/ [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32
/ [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
/ [ *3( h16 ":" ) h16 ] "::" h16 ":" ls32
/ [ *4( h16 ":" ) h16 ] "::" ls32
/ [ *5( h16 ":" ) h16 ] "::" h16
/ [ *6( h16 ":" ) h16 ] "::"
h16 = 1*4HEXDIG
ls32 = ( h16 ":" h16 ) / IPv4address
IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet
dec-octet = DIGIT ; 0-9
/ %x31-39 DIGIT ; 10-99
/ "1" 2DIGIT ; 100-199
/ "2" %x30-34 DIGIT ; 200-249
/ "25" %x30-35 ; 250-255
pct-encoded = "%" HEXDIG HEXDIG
unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"
reserved = gen-delims / sub-delims
gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
sub-delims = "!" / "$" / "&" / "’" / "(" / ")"
/ "*" / "+" / "," / ";" / "="
This syntax does not support IPv6 scoped addressing zone identifiers.
3. Relationship between IRIs and URIs
IRIs are meant to replace URIs in identifying resources for
protocols, formats, and software components that use a UCS-based
character repertoire. These protocols and components may never need
to use URIs directly, especially when the resource identifier is used
simply for identification purposes. However, when the resource
identifier is used for resource retrieval, it is in many cases
necessary to determine the associated URI, because currently most