In the lookup scheme, the language range is progressively truncated
from the end until a matching language tag is located. Single letter
or digit subtags (including both the letter ’x’, which introduces
private-use sequences, and the subtags that introduce extensions) are
removed at the same time as their closest trailing subtag. For
example, starting with the range "zh-Hant-CN-x-private1-private2"
(Chinese, Traditional script, China, two private-use tags) the lookup
progressively searches for content as shown below:
Example of a Lookup Fallback Pattern
Range to match: zh-Hant-CN-x-private1-private2
1. zh-Hant-CN-x-private1-private2
2. zh-Hant-CN-x-private1
3. zh-Hant-CN
4. zh-Hant
5. zh
6. (default)
This fallback behavior allows some flexibility in finding a match.
Without fallback, the default content would be returned immediately
if exactly matching content is unavailable. With fallback, a result
more closely matching the user request can be provided.
Extensions and unrecognized private-use subtags might be unrelated to
a particular application of lookup. Since these subtags come at the
end of the subtag sequence, they are removed first during the
fallback process and usually pose no barrier to interoperability.
However, an implementation MAY remove these from ranges prior to
performing the lookup (provided the implementation also removes them
from the tags being compared). Such modification is internal to the
implementation and applications, protocols, or specifications SHOULD
NOT remove or modify subtags in content that they return or forward,
because this removes information that can be used elsewhere.
The special language range "*" matches any language tag. In the
lookup scheme, this range does not convey enough information by
itself to determine which language tag is most appropriate, since it
matches everything. If the language range "*" is followed by other
language ranges, it is skipped. If the language range "*" is the
only one in the language priority list or if no other language range
follows, the default value is computed and returned.
In some cases, the language priority list can contain one or more
extended language ranges (as, for example, when the same language
priority list is used as input for both lookup and filtering
operations). Wildcard values in an extended language range normally
match any value that can occur in that position in a language tag.
Since only one item can be returned for any given lookup request,
wildcards in a language range have to be processed in a consistent
manner or the same request will produce widely varying results.
Applications, protocols, or specifications that accept extended
language ranges MUST define which item is returned when more than one
item matches the extended language range.
For example, an implementation could map the extended language ranges
to basic ranges. Another possibility would be for an implementation
to return the matching tag that is first in ASCII-order. If the
language range were "*-CH" (’CH’ represents Switzerland) and the set
of tags included "de-CH" (German as used in Switzerland), "fr-CH"
(French, Switzerland), and "it-CH" (Italian, Switzerland), then the
tag "de-CH" would be returned.
3.4.1. Default Values
Each application, protocol, or specification that uses lookup MUST
define the defaulting behavior when no tag matches the language
priority list. What this action consists of strongly depends on how
lookup is being applied. Some examples of defaulting behavior
include:
o return an item with no language tag or an item of a non-linguistic
nature, such as an image or sound
o return a null string as the language tag value, in cases where the
protocol permits the empty value (see, for example, "xml:lang" in
[XML10])
o return a particular language tag designated for the operation
o return the language tag "i-default" (see [RFC2277])
o return an error condition or error message
o return a list of available languages for the user to select from
When performing lookup using a language priority list, the
progressive search MUST process each language range in the list
before seeking or calculating the default.
The default value MAY be calculated or include additional searching
or matching. Applications, protocols, or specifications can specify
different ways in which users can specify or override the defaults.
One common way to provide for a default is to allow a specific
language range to be set as the default for a specific type of
request. If this approach is chosen, this language range MUST be
treated as if it were appended to the end of the language priority
list as a whole, rather than after each item in the language priority
list. The application, protocol, or specification MUST also define
the defaulting behavior if that search fails to find a matching tag
or item.
For example, if a particular user’s language priority list is "fr-FR,
zh-Hant" (French as used in France followed by Chinese as written in
the Traditional script) and the program doing the matching had a
default language range of "ja-JP" (Japanese as used in Japan), then
the program searches as follows:
1. fr-FR
2. fr
3. zh-Hant // next language
4. zh
5. ja-JP // now searching for the default content
6. ja
7. (implementation defined default)
4. Other Considerations
When working with language ranges and matching schemes, there are
some additional points that can influence the choice of either.
4.1. Choosing Language Ranges
Users indicate their language preferences via the choice of a
language range or the list of language ranges in a language priority
list. The type of matching affects what the best choice is for a
user.
Most matching schemes make no attempt to process the semantic meaning
of the subtags. The language range is compared, in a case-
insensitive manner, to each language tag being matched, using basic
string processing. Users SHOULD select language ranges that are
well-formed, valid language tags according to [RFC4646] (substituting
wildcards as appropriate in extended language ranges).
Applications are encouraged to canonicalize language tags and ranges
by using the Preferred-Value from the IANA Language Subtag Registry
for tags or subtags that have been deprecated. If the user is
working with content that might use the older form, the user might
want to include both the new and old forms in a language priority
list. For example, the tag "art-lojban" is deprecated. The subtag
’jbo’ is supposed to be used instead, so the user might use it to
form the language range. Or the user might include both in a
language priority list: "jbo, art-lojban".
Users SHOULD avoid subtags that add no distinguishing value to a
language range. When filtering, the fewer the number of subtags that
appear in the language range, the more content the range will
probably match, while in lookup unnecessary subtags can cause
"better", more-specific content to be skipped in favor of less
specific content. For example, the range "de-Latn-DE" returns
content tagged "de" instead of content tagged "de-DE", even though
the latter is probably a better match.
Whether a subtag adds distinguishing value can depend on the context
of the request. For example, a user who reads both Simplified and
Traditional Chinese, but who prefers Simplified, might use the range
"zh" for filtering (matching all items that user can read) but
"zh-Hans" for lookup (making sure that user gets the preferred form
if it’s available, but the fallback to "zh" will still work). On the
other hand, content in this case ought to be labeled as "zh-Hans" (or
"zh-Hant" if that applies) for filtering, while for lookup, if there
is either "zh-Hans" content or "zh-Hant" content, one of them (the
one considered ’default’) also ought to be made available with the
simple "zh". Note that the user can create a language priority list
"zh-Hans, zh" that delivers the best possible results for both
schemes. If the user cannot be sure which scheme is being used (or
if more than one might be applied to a given request), the user
SHOULD specify the most specific (largest number of subtags) range
first and then supply shorter prefixes later in the list to ensure
that filtering returns a complete set of tags.
Many languages are written predominantly in a single script. This is
usually recorded in the Suppress-Script field in that language
subtag’s registry entry. For these languages, script subtags SHOULD
NOT be used to form a language range. Thus, the language range
"en-Latn" is inappropriate in most cases (because the vast majority
of English documents are written in the Latin script and thus the
’en’ language subtag has a Suppress-Script field for ’Latn’ in the
registry).
When working with tags and ranges, note that extensions and most
private-use subtags are orthogonal to language tag matching, in that
they specify additional attributes of the text not related to the
goals of most matching schemes. Users SHOULD avoid using these
subtags in language ranges, since they interfere with the selection
of available content. When used in language tags (as opposed to
ranges), these subtags normally do not interfere with filtering
(Section 3), since they appear at the end of the tag and will match
all prefixes. Lookup (Section 3.4) implementations are advised to
ignore unrecognized private-use and extension subtags when performing
language tag fallback.
4.2. Meaning of Language Tags and Ranges
Selecting language tags using language ranges requires some
understanding by users of what they are selecting. The meanings of
the various subtags in a language range are identical to their
meanings in a language tag (see Section 4.2 in [RFC4646]), with the
addition that the wildcard "*" represents any matching sequence of
values.
4.3. Considerations for Private-Use Subtags
Private agreement is necessary between the parties that intend to use
or exchange language tags that contain private-use subtags. Great
caution SHOULD be used in employing private-use subtags in content or
protocols intended for general use. Private-use subtags are simply
useless for information exchange without prior arrangement.
The value and semantic meaning of private-use tags and of the subtags
used within such a language tag are not defined. Matching private-
use tags using language ranges or extended language ranges can result
in unpredictable content being returned.
4.4. Length Considerations for Language Ranges
Language ranges are very similar to language tags in terms of content
and usage. The same types of restrictions on length that can be
applied to language tags can also be applied to language ranges. See
[RFC4646] Section 4.3 (Length Considerations).
5. Security Considerations
Language ranges used in content negotiation might be used to infer
the nationality of the sender, and thus identify potential targets
for surveillance. In addition, unique or highly unusual language
ranges or combinations of language ranges might be used to track a
specific individual’s activities.
This is a special case of the general problem that anything you send
is visible to the receiving party. It is useful to be aware that
such concerns can exist in some cases.
The evaluation of the exact magnitude of the threat, and any possible
countermeasures, is left to each application or protocol.
6. Character Set Considerations
Language tags permit only the characters A-Z, a-z, 0-9, and HYPHEN-
MINUS (%x2D). Language ranges also use the character ASTERISK
(%x2A). These characters are present in most character sets, so
presentation or exchange of language tags or ranges should not be
constrained by character set issues.
7. References
7.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2277] Alvestrand, H., "IETF Policy on Character Sets and
Languages", BCP 18, RFC 2277, January 1998.
[RFC4234] Crocker, D., Ed. and P. Overell, "Augmented BNF for
Syntax Specifications: ABNF", RFC 4234, October 2005.
[RFC4646] Phillips, A., Ed., and M. Davis, Ed., "Tags for
Identifying Languages", BCP 47, RFC 4646, September
2006.
7.2. Informative References
[RFC1766] Alvestrand, H., "Tags for the Identification of
Languages", RFC 1766, March 1995.
[RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
Masinter, L., Leach, P., and T. Berners-Lee,
"Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616,
June 1999.
[RFC2616errata] IETF, "HTTP/1.1 Specification Errata", October 2004,
<http://purl.org/NET/http-errata>.
[RFC3066] Alvestrand, H., "Tags for the Identification of
Languages", BCP 47, RFC 3066, January 2001.
[RFC3282] Alvestrand, H., "Content Language Headers", RFC 3282,
May 2002.
[XML10] Bray, T., Paoli, J., Sperberg-McQueen, C., Maler, E.,
and F. Yergeau, "Extensible Markup Language (XML) 1.0
(Third Edition)", World Wide Web Consortium
Recommendation, February 2004,
<http://www.w3.org/TR/REC-xml>.
Appendix A. Acknowledgements
Any list of contributors is bound to be incomplete; please regard the
following as only a selection from the group of people who have
contributed to make this document what it is today.
The contributors to [RFC1766] and [RFC3066], each of which was a
precursor to this document, contributed greatly to the development of
language tag matching, and, in particular, the basic language range
and the basic matching scheme. This document was originally part of
[RFC4646], but was split off before that document’s completion.
Thus, directly or indirectly, those acknowledged in [RFC4646] also
had a hand in the development of this document, and work done prior
to the split is acknowledged in that document.
The following people (in alphabetical order by family name)
contributed to this document:
Harald Alvestrand, Stephane Bortzmeyer, Jeremy Carroll, Peter
Constable, John Cowan, Mark Crispin, Martin Duerst, Frank Ellermann,
Doug Ewell, Debbie Garside, Marion Gunn, Jon Hanna, Kent Karlsson,
Erkki Kolehmainen, Jukka Korpela, Ira McDonald, M. Patton, Randy
Presuhn, Eric van der Poel, Markus Scherer, Misha Wolf, and many,
many others.
Very special thanks must go to Harald Tveit Alvestrand, who
originated RFCs 1766 and 3066, and without whom this document would
not have been possible.
Authors’ Addresses
Addison Phillips (Editor)
Yahoo! Inc.
EMail: addison@inter-locale.com
Mark Davis (Editor)
EMail: mark.davis@macchiato.com or mark.davis@google.com
Full Copyright Statement
Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Acknowledgement
Funding for the RFC Editor function is provided by the IETF
Administrative Support Activity (IASA).