character subtag, that subtag and its preceding "-" MUST also be
removed. For example:
Tag to truncate: zh-Latn-CN-variant1-a-extend1-x-wadegile-private1
1. zh-Latn-CN-variant1-a-extend1-x-wadegile
2. zh-Latn-CN-variant1-a-extend1
3. zh-Latn-CN-variant1
4. zh-Latn-CN
5. zh-Latn
6. zh
Figure 8: Example of Tag Truncation
4.4. Canonicalization of Language Tags
Since a particular language tag is sometimes used by many processes,
language tags SHOULD always be created or generated in a canonical
form.
A language tag is in canonical form when:
1. The tag is well-formed according the rules in Section 2.1 and
Section 2.2.
2. Subtags of type ’Region’ that have a Preferred-Value mapping in
the IANA registry (see Section 3.1) SHOULD be replaced with their
mapped value. Note: In rare cases, the mapped value will also
have a Preferred-Value.
3. Redundant or grandfathered tags that have a Preferred-Value
mapping in the IANA registry (see Section 3.1) MUST be replaced
with their mapped value. These items either are deprecated
mappings created before the adoption of this document (such as
the mapping of "no-nyn" to "nn" or "i-klingon" to "tlh") or are
the result of later registrations or additions to this document
(for example, "zh-guoyu" might be mapped to a language-extlang
combination such as "zh-cmn" by some future update of this
document).
4. Other subtags that have a Preferred-Value mapping in the IANA
registry (see Section 3.1) MUST be replaced with their mapped
value. These items consist entirely of clerical corrections to
ISO 639-1 in which the deprecated subtags have been maintained
for compatibility purposes.
5. If more than one extension subtag sequence exists, the extension
sequences are ordered into case-insensitive ASCII order by
singleton subtag.
Example: The language tag "en-A-aaa-B-ccc-bbb-x-xyz" is in canonical
form, while "en-B-ccc-bbb-A-aaa-X-xyz" is well-formed but not in
canonical form.
Example: The language tag "en-BU" (English as used in Burma) is not
canonical because the ’BU’ subtag has a canonical mapping to ’MM’
(Myanmar), although the tag "en-BU" maintains its validity.
Canonicalization of language tags does not imply anything about the
use of upper or lowercase letters when processing or comparing
subtags (and as described in Section 2.1). All comparisons MUST be
performed in a case-insensitive manner.
When performing canonicalization of language tags, processors MAY
regularize the case of the subtags (that is, this process is
OPTIONAL), following the case used in the registry. Note that this
corresponds to the following casing rules: uppercase all non-initial
two-letter subtags; titlecase all non-initial four-letter subtags;
lowercase everything else.
Note: Case folding of ASCII letters in certain locales, unless
carefully handled, sometimes produces non-ASCII character values.
The Unicode Character Database file "SpecialCasing.txt" defines the
specific cases that are known to cause problems with this. In
particular, the letter ’i’ (U+0069) in Turkish and Azerbaijani is
uppercased to U+0130 (LATIN CAPITAL LETTER I WITH DOT ABOVE).
Implementers SHOULD specify a locale-neutral casing operation to
ensure that case folding of subtags does not produce this value,
which is illegal in language tags. For example, if one were to
uppercase the region subtag ’in’ using Turkish locale rules, the
sequence U+0130 U+004E would result instead of the expected ’IN’.
Note: if the field ’Deprecated’ appears in a registry record without
an accompanying ’Preferred-Value’ field, then that tag or subtag is
deprecated without a replacement. Validating processors SHOULD NOT
generate tags that include these values, although the values are
canonical when they appear in a language tag.
An extension MUST define any relationships that exist between the
various subtags in the extension and thus MAY define an alternate
canonicalization scheme for the extension’s subtags. Extensions MAY
define how the order of the extension’s subtags are interpreted. For
example, an extension could define that its subtags are in canonical
order when the subtags are placed into ASCII order: that is,
"en-a-aaa-bbb-ccc" instead of "en-a-ccc-bbb-aaa". Another extension
might define that the order of the subtags influences their semantic
meaning (so that "en-b-ccc-bbb-aaa" has a different value from
"en-b-aaa-bbb-ccc"). However, extension specifications SHOULD be
designed so that they are tolerant of the typical processes described
in Section 3.7.
4.5. Considerations for Private Use Subtags
Private use subtags, like all other subtags, MUST conform to the
format and content constraints in the ABNF. Private use subtags have
no meaning outside the private agreement between the parties that
intend to use or exchange language tags that employ them. The same
subtags MAY be used with a different meaning under a separate private
agreement. They SHOULD NOT be used where alternatives exist and
SHOULD NOT be used in content or protocols intended for general use.
Private use subtags are simply useless for information exchange
without prior arrangement. The value and semantic meaning of private
use tags and of the subtags used within such a language tag are not
defined by this document.
Subtags defined in the IANA registry as having a specific private use
meaning convey more information that a purely private use tag
prefixed by the singleton subtag ’x’. For applications, this
additional information MAY be useful.
For example, the region subtags ’AA’, ’ZZ’, and in the ranges
’QM’-’QZ’ and ’XA’-’XZ’ (derived from ISO 3166 private use codes) MAY
be used to form a language tag. A tag such as "zh-Hans-XQ" conveys a
great deal of public, interchangeable information about the language
material (that it is Chinese in the simplified Chinese script and is
suitable for some geographic region ’XQ’). While the precise
geographic region is not known outside of private agreement, the tag
conveys far more information than an opaque tag such as "x-someLang",
which contains no information about the language subtag or script
subtag outside of the private agreement.
However, in some cases content tagged with private use subtags MAY
interact with other systems in a different and possibly unsuitable
manner compared to tags that use opaque, privately defined subtags,
so the choice of the best approach sometimes depends on the
particular domain in question.
5. IANA Considerations
This section deals with the processes and requirements necessary for
IANA to undertake to maintain the subtag and extension registries as
defined by this document and in accordance with the requirements of
[RFC2434].
The impact on the IANA maintainers of the two registries defined by
this document will be a small increase in the frequency of new
entries or updates.
5.1. Language Subtag Registry
Upon adoption of this document, the registry will be initialized by a
companion document: [RFC4645]. The criteria and process for
selecting the initial set of records are described in that document.
The initial set of records represents no impact on IANA, since the
work to create it will be performed externally.
The new registry MUST be listed under "Language Tags" at
<http://www.iana.org/numbers.html>, replacing the existing
registrations defined by [RFC3066]. The existing set of registration
forms and RFC 3066 registrations MUST be relabeled as "Language Tags
(Obsolete)" and maintained (but not added to or modified).
Future work on the Language Subtag Registry SHALL be limited to
inserting or replacing whole records preformatted for IANA by the
Language Subtag Reviewer as described in Section 3.3 of this document
and archiving the forwarded registration form.
Each record MUST be sent to iana@iana.org with a subject line
indicating whether the enclosed record is an insertion of a new
record (indicated by the word "INSERT" in the subject line) or a
replacement of an existing record (indicated by the word "MODIFY" in
the subject line). Records MUST NOT be deleted from the registry.
IANA MUST place any inserted or modified records into the appropriate
section of the language subtag registry, grouping the records by
their ’Type’ field. Inserted records MAY be placed anywhere in the
appropriate section; there is no guarantee of the order of the
records beyond grouping them together by ’Type’. Modified records
MUST overwrite the record they replace.
Included in any request to insert or modify records MUST be a new
File-Date record. This record MUST be placed first in the registry.
In the event that the File-Date record present in the registry has a
later date than the record being inserted or modified, the existing
record MUST be preserved.
5.2. Extensions Registry
The Language Tag Extensions Registry will also be generated and sent
to IANA as described in Section 3.7. This registry can contain at
most 35 records, and thus changes to this registry are expected to be
very infrequent.
Future work by IANA on the Language Tag Extensions Registry is
limited to two cases. First, the IESG MAY request that new records
be inserted into this registry from time to time. These requests
MUST include the record to insert in the exact format described in
Section 3.7. In addition, there MAY be occasional requests from the
maintaining authority for a specific extension to update the contact
information or URLs in the record. These requests MUST include the
complete, updated record. IANA is not responsible for validating the
information provided, only that it is properly formatted. It should
reasonably be seen to come from the maintaining authority named in
the record present in the registry.
6. Security Considerations
Language tags used in content negotiation, like any other information
exchanged on the Internet, might be a source of concern because they
might be used to infer the nationality of the sender, and thus
identify potential targets for surveillance.
This is a special case of the general problem that anything sent is
visible to the receiving party and possibly to third parties as well.
It is useful to be aware that such concerns can exist in some cases.
The evaluation of the exact magnitude of the threat, and any possible
countermeasures, is left to each application protocol (see BCP 72
[RFC3552] for best current practice guidance on security threats and
defenses).
The language tag associated with a particular information item is of
no consequence whatsoever in determining whether that content might
contain possible homographs. The fact that a text is tagged as being
in one language or using a particular script subtag provides no
assurance whatsoever that it does not contain characters from scripts
other than the one(s) associated with or specified by that language
tag.
Since there is no limit to the number of variant, private use, and
extension subtags, and consequently no limit on the possible length
of a tag, implementations need to guard against buffer overflow
attacks. See Section 4.3 for details on language tag truncation,
which can occur as a consequence of defenses against buffer overflow.
Although the specification of valid subtags for an extension (see
Section 3.7) MUST be available over the Internet, implementations
SHOULD NOT mechanically depend on it being always accessible, to
prevent denial-of-service attacks.
7. Character Set Considerations
The syntax in this document requires that language tags use only the
characters A-Z, a-z, 0-9, and HYPHEN-MINUS, which are present in most
character sets, so the composition of language tags should not have
any character set issues.
Rendering of characters based on the content of a language tag is not
addressed in this memo. Historically, some languages have relied on
the use of specific character sets or other information in order to
infer how a specific character should be rendered (notably this
applies to language- and culture-specific variations of Han
ideographs as used in Japanese, Chinese, and Korean). When language
tags are applied to spans of text, rendering engines sometimes use
that information in deciding which font to use in the absence of
other information, particularly where languages with distinct writing
traditions use the same characters.
8. Changes from RFC 3066
The main goals for this revision of language tags were the following:
*Compatibility.* All RFC 3066 language tags (including those in the
IANA registry) remain valid in this specification. The changes in
this document represent additional constraints on language tags.
That is, in no case is the syntax more permissive and processors
based on the ABNF and other provisions of RFC 3066 (such as those
described in [XMLSchema]) will be able to process the tags described
by this document. In addition, this document defines language tags
in such as way as to ensure future compatibility.
*Stability.* Because of changes in the past in the underlying ISO
standards, a valid RFC 3066 language tag could become invalid or have
its meaning change. This has the potential of invalidating content
that may have an extensive shelf-life. In this specification, once a
language tag is valid, it remains valid forever.
*Validity.* The structure of language tags defined by this document
makes it possible to determine if a particular tag is well-formed
without regard for the actual content or "meaning" of the tag as a
whole. This is important because the registry grows and underlying
standards change over time. In addition, it must be possible to
determine if a tag is valid (or not) for a given point in time in
order to provide reproducible, testable results. This process must
not be error-prone; otherwise implementations might give different
results. By having an authoritative registry with specific
versioning information, the validity of language tags at any point in
time can be precisely determined (instead of interpolating values
from many separate sources).
*Utility.* It is sometimes important to be able to differentiate
between written forms of a language -- for many implementations this
is more important than distinguishing between the spoken variants of
a language. Languages are written in a wide variety of different
scripts, so this document provides for the generative use of ISO
15924 script codes. Like the generative use of ISO language and
country codes in RFC 3066, this allows combinations to be produced
without resorting to the registration process. The addition of UN
M.49 codes provides for the generation of language tags with regional
scope, which is also required by some applications.
The recast of the registry from containing whole language tags to
subtags is a key part of this. An important feature of RFC 3066 was
that it allowed generative use of subtags. This allows people to
meaningfully use generated tags, without the delays in registering
whole tags or the need to register all of the combinations that might
be useful.
The choice of placing the extended language and script subtags
between the primary language and region subtags was widely debated.
This design was chosen because the prevalent matching and content
negotiation schemes rely on the subtags being arranged in order of
increasing specificity. That is, the subtags that mark a greater
barrier to mutual intelligibility appear left-most in a tag. For
example, when selecting content written in Azerbaijani, the script
(Arabic, Cyrillic, or Latin) represents a greater barrier to
understanding than any regional variations (those associated with
Azerbaijan or Iran, for example). Individuals who prefer documents
in a particular script, but can deal with the minor regional
differences, can therefore select appropriate content. Applications
that do not deal with written content will continue to omit these
subtags.
*Extensibility.* Because of the widespread use of language tags, it
is disruptive to have periodic revisions of the core specification,
even in the face of demonstrated need. The extension mechanism
provides for a way for independent RFCs to define extensions to
language tags. These extensions have a very constrained, well-
defined structure that prevents extensions from interfering with
implementations of language tags defined in this document.
The document also anticipates features of ISO 639-3 with the addition
of the extended language subtags, as well as the possibility of other
ISO 639 parts becoming useful for the formation of language tags in
the future.
The use and definition of private use tags have also been modified,
to allow people to use private use subtags to extend or modify
defined tags and to move as much information as possible out of
private use and into the regular structure.
The goal for each of these modifications is to reduce or eliminate
the need for future revisions of this document.
The specific changes in this document to meet these goals are:
o Defines the ABNF and rules for subtags so that the category of all
subtags can be determined without reference to the registry.
o Adds the concept of well-formed vs. validating processors,
defining the rules by which an implementation can claim to be one
or the other.
o Replaces the IANA language tag registry with a language subtag
registry that provides a complete list of valid subtags in the
IANA registry. This allows for robust implementation and ease of
maintenance. The language subtag registry becomes the canonical
source for forming language tags.
o Provides a process that guarantees stability of language tags, by
handling reuse of values by ISO 639, ISO 15924, and ISO 3166 in
the event that they register a previously used value for a new
purpose.
o Allows ISO 15924 script code subtags and allows them to be used
generatively. Defines a method for indicating in the registry
when script subtags are necessary for a given language tag.
o Adds the concept of a variant subtag and allows variants to be
used generatively.
o Adds the ability to use a class of UN M.49 tags for supra-national
regions and to resolve conflicts in the assignment of ISO 3166
codes.
o Defines the private use tags in ISO 639, ISO 15924, and ISO 3166
as the mechanism for creating private use language, script, and
region subtags, respectively.
o Adds a well-defined extension mechanism.
o Defines an extended language subtag, possibly for use with certain
anticipated features of ISO 639-3.
9. References
9.1. Normative References
[ISO10646] International Organization for Standardization,
"ISO/IEC 10646:2003. Information technology --
Universal Multiple-Octet Coded Character Set (UCS)",
2003.
[ISO15924] International Organization for Standardization, "ISO
15924:2004. Information and documentation -- Codes for
the representation of names of scripts", January 2004.
[ISO3166-1] International Organization for Standardization, "ISO
3166-1:1997. Codes for the representation of names of
countries and their subdivisions -- Part 1: Country
codes", 1997.
[ISO639-1] International Organization for Standardization, "ISO
639-1:2002. Codes for the representation of names of
languages -- Part 1: Alpha-2 code", 2002.
[ISO639-2] International Organization for Standardization, "ISO
639-2:1998. Codes for the representation of names of
languages -- Part 2: Alpha-3 code, first edition",
1998.
[ISO646] International Organization for Standardization,
"ISO/IEC 646:1991, Information technology -- ISO 7-bit
coded character set for information interchange.",
1991.
[RFC2026] Bradner, S., "The Internet Standards Process --
Revision 3", BCP 9, RFC 2026, October 1996.
[RFC2028] Hovey, R. and S. Bradner, "The Organizations Involved
in the IETF Standards Process", BCP 11, RFC 2028,
October 1996.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2434] Narten, T. and H. Alvestrand, "Guidelines for Writing
an IANA Considerations Section in RFCs", BCP 26,
RFC 2434, October 1998.
[RFC2860] Carpenter, B., Baker, F., and M. Roberts, "Memorandum
of Understanding Concerning the Technical Work of the
Internet Assigned Numbers Authority", RFC 2860,
June 2000.
[RFC3339] Klyne, G., Ed. and C. Newman, "Date and Time on the
Internet: Timestamps", RFC 3339, July 2002.
[RFC4234] Crocker, D., Ed. and P. Overell, "Augmented BNF for
Syntax Specifications: ABNF", RFC 4234, October 2005.
[UN_M.49] Statistics Division, United Nations, "Standard Country
or Area Codes for Statistical Use", UN Standard
Country or Area Codes for Statistical Use, Revision 4
(United Nations publication, Sales No. 98.XVII.9,
June 1999.
9.2. Informative References
[RFC1766] Alvestrand, H., "Tags for the Identification of
Languages", RFC 1766, March 1995.
[RFC2047] Moore, K., "MIME (Multipurpose Internet Mail
Extensions) Part Three: Message Header Extensions for
Non-ASCII Text", RFC 2047, November 1996.
[RFC2231] Freed, N. and K. Moore, "MIME Parameter Value and
Encoded Word Extensions: Character Sets, Languages,
and Continuations", RFC 2231, November 1997.
[RFC2781] Hoffman, P. and F. Yergeau, "UTF-16, an encoding of
ISO 10646", RFC 2781, February 2000.
[RFC3066] Alvestrand, H., "Tags for the Identification of
Languages", BCP 47, RFC 3066, January 2001.
[RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing
RFC Text on Security Considerations", BCP 72,
RFC 3552, July 2003.
[RFC4645] Ewell, D., Ed., "Initial Language Subtag Registry",
RFC 4645, September 2006.
[RFC4647] Phillips, A., Ed. and M. Davis, Ed., "Matching of
Language Tags", BCP 47, RFC 4647, September 2006.
[Unicode] Unicode Consortium, "The Unicode Standard, Version
5.0", Boston, MA, Addison-Wesley, 2007. ISBN 0-321-
48091-0.
[XML10] Bray (et al), T., "Extensible Markup Language (XML)
1.0", 02 2004.