retrieval mechanisms are only defined for URIs. In this case, IRIs
can serve as presentation elements for URI protocol elements. An
example would be an address bar in a Web user agent. (Additional
rationale is given in section 3.1.)
3.1. Mapping of IRIs to URIs
This section defines how to map an IRI to a URI. Everything in this
section also applies to IRI references and URI references, as well as
to components thereof (for example, fragment identifiers).
This mapping has two purposes:
Syntaxical. Many URI schemes and components define additional
syntactical restrictions not captured in section 2.2.
Scheme-specific restrictions are applied to IRIs by converting
IRIs to URIs and checking the URIs against the scheme-specific
restrictions.
Interpretational. URIs identify resources in various ways. IRIs also
identify resources. When the IRI is used solely for
identification purposes, it is not necessary to map the IRI to a
URI (see section 5). However, when an IRI is used for resource
retrieval, the resource that the IRI locates is the same as the
one located by the URI obtained after converting the IRI according
to the procedure defined here. This means that there is no need
to define resolution separately on the IRI level.
Applications MUST map IRIs to URIs by using the following two steps.
Step 1. Generate a UCS character sequence from the original IRI
format. This step has the following three variants,
depending on the form of the input:
a. If the IRI is written on paper, read aloud, or otherwise
represented as a sequence of characters independent of
any character encoding, represent the IRI as a sequence
of characters from the UCS normalized according to
Normalization Form C (NFC, [UTR15]).
b. If the IRI is in some digital representation (e.g., an
octet stream) in some known non-Unicode character
encoding, convert the IRI to a sequence of characters
from the UCS normalized according to NFC.
c. If the IRI is in a Unicode-based character encoding (for
example, UTF-8 or UTF-16), do not normalize (see section
5.3.2.2 for details). Apply step 2 directly to the
encoded Unicode character sequence.
Step 2. For each character in ’ucschar’ or ’iprivate’, apply steps
2.1 through 2.3 below.
2.1. Convert the character to a sequence of one or more octets
using UTF-8 [RFC3629].
2.2. Convert each octet to %HH, where HH is the hexadecimal
notation of the octet value. Note that this is identical
to the percent-encoding mechanism in section 2.1 of
[RFC3986]. To reduce variability, the hexadecimal notation
SHOULD use uppercase letters.
2.3. Replace the original character with the resulting character
sequence (i.e., a sequence of %HH triplets).
The above mapping from IRIs to URIs produces URIs fully conforming to
[RFC3986]. The mapping is also an identity transformation for URIs
and is idempotent; applying the mapping a second time will not
change anything. Every URI is by definition an IRI.
Systems accepting IRIs MAY convert the ireg-name component of an IRI
as follows (before step 2 above) for schemes known to use domain
names in ireg-name, if the scheme definition does not allow
percent-encoding for ireg-name:
Replace the ireg-name part of the IRI by the part converted using the
ToASCII operation specified in section 4.1 of [RFC3490] on each
dot-separated label, and by using U+002E (FULL STOP) as a label
separator, with the flag UseSTD3ASCIIRules set to TRUE, and with the
flag AllowUnassigned set to FALSE for creating IRIs and set to TRUE
otherwise.
The ToASCII operation may fail, but this would mean that the IRI
cannot be resolved. This conversion SHOULD be used when the goal is
to maximize interoperability with legacy URI resolvers. For example,
the IRI
"http://résumé.example.org"
may be converted to
"http://xn--rsum-bpad.example.org"
instead of
"http://r%C3%A9sum%C3%A9.example.org".
An IRI with a scheme that is known to use domain names in ireg-name,
but where the scheme definition does not allow percent-encoding for
ireg-name, meets scheme-specific restrictions if either the
straightforward conversion or the conversion using the ToASCII
operation on ireg-name result in an URI that meets the scheme-
specific restrictions.
Such an IRI resolves to the URI obtained after converting the IRI and
uses the ToASCII operation on ireg-name. Implementations do not have
to do this conversion as long as they produce the same result.
Note: The difference between variants b and c in step 1 (using
normalization with NFC, versus not using any normalization)
accounts for the fact that in many non-Unicode character
encodings, some text cannot be represented directly. For example,
the word "Vietnam" is natively written "Việt Nam"
(containing a LATIN SMALL LETTER E WITH CIRCUMFLEX AND DOT BELOW)
in NFC, but a direct transcoding from the windows-1258 character
encoding leads to "Việt Nam" (containing a LATIN SMALL
LETTER E WITH CIRCUMFLEX followed by a COMBINING DOT BELOW).
Direct transcoding of other 8-bit encodings of Vietnamese may lead
to other representations.
Note: The uniform treatment of the whole IRI in step 2 is important
to make processing independent of URI scheme. See [Gettys] for an
in-depth discussion.
Note: In practice, whether the general mapping (steps 1 and 2) or the
ToASCII operation of [RFC3490] is used for ireg-name will not be
noticed if mapping from IRI to URI and resolution is tightly
integrated (e.g., carried out in the same user agent). But
conversion using [RFC3490] may be able to better deal with
backwards compatibility issues in case mapping and resolution are
separated, as in the case of using an HTTP proxy.
Note: Internationalized Domain Names may be contained in parts of an
IRI other than the ireg-name part. It is the responsibility of
scheme-specific implementations (if the Internationalized Domain
Name is part of the scheme syntax) or of server-side
implementations (if the Internationalized Domain Name is part of
’iquery’) to apply the necessary conversions at the appropriate
point. Example: Trying to validate the Web page at
http://résumé.example.org would lead to an IRI of
http://validator.w3.org/check?uri=http%3A%2F%2Frésumé.
example.org, which would convert to a URI of
http://validator.w3.org/check?uri=http%3A%2F%2Fr%C3%A9sum%C3%A9.
example.org. The server side implementation would be responsible
for making the necessary conversions to be able to retrieve the
Web page.
Systems accepting IRIs MAY also deal with the printable characters in
US-ASCII that are not allowed in URIs, namely "<", ">", ’"’, space,
"{", "}", "|", "\", "^", and "`", in step 2 above. If these
characters are found but are not converted, then the conversion
SHOULD fail. Please note that the number sign ("#"), the percent
sign ("%"), and the square bracket characters ("[", "]") are not part
of the above list and MUST NOT be converted. Protocols and formats
that have used earlier definitions of IRIs including these characters
MAY require percent-encoding of these characters as a preprocessing
step to extract the actual IRI from a given field. This
preprocessing MAY also be used by applications allowing the user to
enter an IRI.
Note: In this process (in step 2.3), characters allowed in URI
references and existing percent-encoded sequences are not encoded
further. (This mapping is similar to, but different from, the
encoding applied when arbitrary content is included in some part
of a URI.) For example, an IRI of
"http://www.example.org/red%09rosé#red" (in XML notation) is
converted to
"http://www.example.org/red%09ros%C3%A9#red", not to something
like
"http%3A%2F%2Fwww.example.org%2Fred%2509ros%C3%A9%23red".
Note: Some older software transcoding to UTF-8 may produce illegal
output for some input, in particular for characters outside the
BMP (Basic Multilingual Plane). As an example, for the IRI with
non-BMP characters (in XML Notation):
"http://example.com/𐌀𐌁𐌂";
which contains the first three letters of the Old Italic alphabet,
the correct conversion to a URI is
"http://example.com/%F0%90%8C%80%F0%90%8C%81%F0%90%8C%82"
3.2. Converting URIs to IRIs
In some situations, converting a URI into an equivalent IRI may be
desirable. This section gives a procedure for this conversion. The
conversion described in this section will always result in an IRI
that maps back to the URI used as an input for the conversion (except
for potential case differences in percent-encoding and for potential
percent-encoded unreserved characters). However, the IRI resulting
from this conversion may not be exactly the same as the original IRI
(if there ever was one).
URI-to-IRI conversion removes percent-encodings, but not all
percent-encodings can be eliminated. There are several reasons for
this:
1. Some percent-encodings are necessary to distinguish percent-
encoded and unencoded uses of reserved characters.
2. Some percent-encodings cannot be interpreted as sequences of
UTF-8 octets.
(Note: The octet patterns of UTF-8 are highly regular.
Therefore, there is a very high probability, but no guarantee,
that percent-encodings that can be interpreted as sequences of
UTF-8 octets actually originated from UTF-8. For a detailed
discussion, see [Duerst97].)
3. The conversion may result in a character that is not appropriate
in an IRI. See sections 2.2, 4.1, and 6.1 for further details.
Conversion from a URI to an IRI is done by using the following steps
(or any other algorithm that produces the same result):
1. Represent the URI as a sequence of octets in US-ASCII.
2. Convert all percent-encodings ("%" followed by two hexadecimal
digits) to the corresponding octets, except those corresponding
to "%", characters in "reserved", and characters in US-ASCII not
allowed in URIs.
3. Re-percent-encode any octet produced in step 2 that is not part
of a strictly legal UTF-8 octet sequence.
4. Re-percent-encode all octets produced in step 3 that in UTF-8
represent characters that are not appropriate according to
sections 2.2, 4.1, and 6.1.
5. Interpret the resulting octet sequence as a sequence of characters
encoded in UTF-8.
This procedure will convert as many percent-encoded characters as
possible to characters in an IRI. Because there are some choices
when step 4 is applied (see section 6.1), results may vary.
Conversions from URIs to IRIs MUST NOT use any character encoding
other than UTF-8 in steps 3 and 4, even if it might be possible to
guess from the context that another character encoding than UTF-8 was
used in the URI. For example, the URI
"http://www.example.org/r%E9sum%E9.html" might with some guessing be
interpreted to contain two e-acute characters encoded as iso-8859-1.
It must not be converted to an IRI containing these e-acute
characters. Otherwise, in the future the IRI will be mapped to
"http://www.example.org/r%C3%A9sum%C3%A9.html", which is a different
URI from "http://www.example.org/r%E9sum%E9.html".
3.2.1. Examples
This section shows various examples of converting URIs to IRIs. Each
example shows the result after each of the steps 1 through 5 is
applied. XML Notation is used for the final result. Octets are
denoted by "<" followed by two hexadecimal digits followed by ">".
The following example contains the sequence "%C3%BC", which is a
strictly legal UTF-8 sequence, and which is converted into the actual
character U+00FC, LATIN SMALL LETTER U WITH DIAERESIS (also known as
u-umlaut).
1. http://www.example.org/D%C3%BCrst
2. http://www.example.org/D<c3><bc>rst
3. http://www.example.org/D<c3><bc>rst
4. http://www.example.org/D<c3><bc>rst
5. http://www.example.org/Dürst
The following example contains the sequence "%FC", which might
represent U+00FC, LATIN SMALL LETTER U WITH DIAERESIS, in the
iso-8859-1 character encoding. (It might represent other characters
in other character encodings. For example, the octet <fc> in
iso-8859-5 represents U+045C, CYRILLIC SMALL LETTER KJE.) Because
<fc> is not part of a strictly legal UTF-8 sequence, it is
re-percent-encoded in step 3.
1. http://www.example.org/D%FCrst
2. http://www.example.org/D<fc>rst
3. http://www.example.org/D%FCrst
4. http://www.example.org/D%FCrst
5. http://www.example.org/D%FCrst
The following example contains "%e2%80%ae", which is the percent-
encoded UTF-8 character encoding of U+202E, RIGHT-TO-LEFT OVERRIDE.
Section 4.1 forbids the direct use of this character in an IRI.
Therefore, the corresponding octets are re-percent-encoded in step 4.
This example shows that the case (upper- or lowercase) of letters
used in percent-encodings may not be preserved. The example also
contains a punycode-encoded domain name label (xn--99zt52a), which is
not converted.
1. http://xn--99zt52a.example.org/%e2%80%ae
2. http://xn--99zt52a.example.org/<e2><80><ae>
3. http://xn--99zt52a.example.org/<e2><80><ae>
4. http://xn--99zt52a.example.org/%E2%80%AE
5. http://xn--99zt52a.example.org/%E2%80%AE
Implementations with scheme-specific knowledge MAY convert
punycode-encoded domain name labels to the corresponding characters
by using the ToUnicode procedure. Thus, for the example above, the
label "xn--99zt52a" may be converted to U+7D0D U+8C46 (Japanese
Natto), leading to the overall IRI of
"http://納豆.example.org/%E2%80%AE".
4. Bidirectional IRIs for Right-to-Left Languages
Some UCS characters, such as those used in the Arabic and Hebrew
scripts, have an inherent right-to-left (rtl) writing direction.
IRIs containing these characters (called bidirectional IRIs or Bidi
IRIs) require additional attention because of the non-trivial
relation between logical representation (used for digital
representation and for reading/spelling) and visual representation
(used for display/printing).
Because of the complex interaction between the logical
representation, the visual representation, and the syntax of a Bidi
IRI, a balance is needed between various requirements. The main
requirements are
1. user-predictable conversion between visual and logical
representation;
2. the ability to include a wide range of characters in various
parts of the IRI; and
3. minor or no changes or restrictions for implementations.
4.1. Logical Storage and Visual Presentation
When stored or transmitted in digital representation, bidirectional
IRIs MUST be in full logical order and MUST conform to the IRI syntax
rules (which includes the rules relevant to their scheme). This
ensures that bidirectional IRIs can be processed in the same way as
other IRIs.
Bidirectional IRIs MUST be rendered by using the Unicode
Bidirectional Algorithm [UNIV4], [UNI9]. Bidirectional IRIs MUST be
rendered in the same way as they would be if they were in a
left-to-right embedding; i.e., as if they were preceded by U+202A,
LEFT-TO-RIGHT EMBEDDING (LRE), and followed by U+202C, POP
DIRECTIONAL FORMATTING (PDF). Setting the embedding direction can
also be done in a higher-level protocol (e.g., the dir=’ltr’
attribute in HTML).
There is no requirement to use the above embedding if the display is
still the same without the embedding. For example, a bidirectional
IRI in a text with left-to-right base directionality (such as used
for English or Cyrillic) that is preceded and followed by whitespace
and strong left-to-right characters does not need an embedding.
Also, a bidirectional relative IRI reference that only contains
strong right-to-left characters and weak characters and that starts
and ends with a strong right-to-left character and appears in a text
with right-to-left base directionality (such as used for Arabic or
Hebrew) and is preceded and followed by whitespace and strong
characters does not need an embedding.
In some other cases, using U+200E, LEFT-TO-RIGHT MARK (LRM), may be
sufficient to force the correct display behavior. However, the
details of the Unicode Bidirectional algorithm are not always easy to
understand. Implementers are strongly advised to err on the side of
caution and to use embedding in all cases where they are not
completely sure that the display behavior is unaffected without the
embedding.
The Unicode Bidirectional Algorithm ([UNI9], section 4.3) permits
higher-level protocols to influence bidirectional rendering. Such
changes by higher-level protocols MUST NOT be used if they change the
rendering of IRIs.
The bidirectional formatting characters that may be used before or
after the IRI to ensure correct display are not themselves part of
the IRI. IRIs MUST NOT contain bidirectional formatting characters
(LRM, RLM, LRE, RLE, LRO, RLO, and PDF). They affect the visual
rendering of the IRI but do not appear themselves. It would
therefore not be possible to input an IRI with such characters
correctly.
4.2. Bidi IRI Structure
The Unicode Bidirectional Algorithm is designed mainly for running
text. To make sure that it does not affect the rendering of
bidirectional IRIs too much, some restrictions on bidirectional IRIs
are necessary. These restrictions are given in terms of delimiters
(structural characters, mostly punctuation such as "@", ".", ":", and
"/") and components (usually consisting mostly of letters and
digits).
The following syntax rules from section 2.2 correspond to components
for the purpose of Bidi behavior: iuserinfo, ireg-name, isegment,
isegment-nz, isegment-nz-nc, ireg-name, iquery, and ifragment.
Specifications that define the syntax of any of the above components
MAY divide them further and define smaller parts to be components
according to this document. As an example, the restrictions of
[RFC3490] on bidirectional domain names correspond to treating each
label of a domain name as a component for schemes with ireg-name as a
domain name. Even where the components are not defined formally, it
may be helpful to think about some syntax in terms of components and
to apply the relevant restrictions. For example, for the usual
name/value syntax in query parts, it is convenient to treat each name
and each value as a component. As another example, the extensions in
a resource name can be treated as separate components.
For each component, the following restrictions apply:
1. A component SHOULD NOT use both right-to-left and left-to-right
characters.
2. A component using right-to-left characters SHOULD start and end
with right-to-left characters.
The above restrictions are given as shoulds, rather than as musts.
For IRIs that are never presented visually, they are not relevant.
However, for IRIs in general, they are very important to ensure
consistent conversion between visual presentation and logical
representation, in both directions.
Note: In some components, the above restrictions may actually be
strictly enforced. For example, [RFC3490] requires that these
restrictions apply to the labels of a host name for those schemes
where ireg-name is a host name. In some other components (for
example, path components) following these restrictions may not be
too difficult. For other components, such as parts of the query
part, it may be very difficult to enforce the restrictions because
the values of query parameters may be arbitrary character
sequences.
If the above restrictions cannot be satisfied otherwise, the affected
component can always be mapped to URI notation as described in
section 3.1. Please note that the whole component has to be mapped
(see also Example 9 below).
4.3. Input of Bidi IRIs
Bidi input methods MUST generate Bidi IRIs in logical order while
rendering them according to section 4.1. During input, rendering
SHOULD be updated after every new character is input to avoid end-
user confusion.
4.4. Examples
This section gives examples of bidirectional IRIs, in Bidi Notation.
It shows legal IRIs with the relationship between logical and visual
representation and explains how certain phenomena in this
relationship may look strange to somebody not familiar with
bidirectional behavior, but familiar to users of Arabic and Hebrew.
It also shows what happens if the restrictions given in section 4.2
are not followed. The examples below can be seen at [BidiEx], in
Arabic, Hebrew, and Bidi Notation variants.
To read the bidi text in the examples, read the visual representation
from left to right until you encounter a block of rtl text. Read the
rtl block (including slashes and other special characters) from right
to left, then continue at the next unread ltr character.
Example 1: A single component with rtl characters is inverted:
Logical representation: "http://ab.CDEFGH.ij/kl/mn/op.html"
Visual representation: "http://ab.HGFEDC.ij/kl/mn/op.html"
Components can be read one by one, and each component can be read in
its natural direction.
Example 2: More than one consecutive component with rtl characters is
inverted as a whole:
Logical representation: "http://ab.CDE.FGH/ij/kl/mn/op.html"
Visual representation: "http://ab.HGF.EDC/ij/kl/mn/op.html"
A sequence of rtl components is read rtl, in the same way as a
sequence of rtl words is read rtl in a bidi text.
Example 3: All components of an IRI (except for the scheme) are rtl.
All rtl components are inverted overall:
Logical representation: "http://AB.CD.EF/GH/IJ/KL?MN=OP;QR=ST#UV"
Visual representation: "http://VU#TS=RQ;PO=NM?LK/JI/HG/FE.DC.BA"
The whole IRI (except the scheme) is read rtl. Delimiters between
rtl components stay between the respective components; delimiters
between ltr and rtl components don’t move.
Example 4: Each of several sequences of rtl components is inverted on
its own:
Logical representation: "http://AB.CD.ef/gh/IJ/KL.html"
Visual representation: "http://DC.BA.ef/gh/LK/JI.html"
Each sequence of rtl components is read rtl, in the same way as each
sequence of rtl words in an ltr text is read rtl.
Example 5: Example 2, applied to components of different kinds:
Logical representation: "http://ab.cd.EF/GH/ij/kl.html"
Visual representation: "http://ab.cd.HG/FE/ij/kl.html"
The inversion of the domain name label and the path component may be
unexpected, but it is consistent with other bidi behavior. For
reassurance that the domain component really is "ab.cd.EF", it may be
helpful to read aloud the visual representation following the bidi
algorithm. After "http://ab.cd." one reads the RTL block
"E-F-slash-G-H", which corresponds to the logical representation.
Example 6: Same as Example 5, with more rtl components:
Logical representation: "http://ab.CD.EF/GH/IJ/kl.html"
Visual representation: "http://ab.JI/HG/FE.DC/kl.html"
The inversion of the domain name labels and the path components may
be easier to identify because the delimiters also move.
Example 7: A single rtl component includes digits:
Logical representation: "http://ab.CDE123FGH.ij/kl/mn/op.html"
Visual representation: "http://ab.HGF123EDC.ij/kl/mn/op.html"
Numbers are written ltr in all cases but are treated as an additional
embedding inside a run of rtl characters. This is completely
consistent with usual bidirectional text.
Example 8 (not allowed): Numbers are at the start or end of an rtl
component:
Logical representation: "http://ab.cd.ef/GH1/2IJ/KL.html"