spoken. <NONE>
Unlike transliterations, which are generally designed to be
round-trip convertible, transcriptions of written material are
almost never round-trip convertible to their original form.
regular expressions
Regular expressions provide a mechanism to select specific strings
from a set of character strings. Regular expressions are a
language used to search for text within strings, and possibly
modify the text found with other text. <NONE>
Pattern matching for text involves being able to represent one or
more code points in an abstract notation, such as searching for
all capital Latin letters or all punctuation. The most common
mechanism in IETF protocols for naming such patterns is the use of
regular expressions. There is no single regular expression
language, but there are numerous very similar dialects.
The Unicode Consortium has a good discussion about how to adapt
regular expression engines to use Unicode. [UTR18]
private use
ISO/IEC 10646 code points from U+E000 to U+F8FF, U+F0000 to
U+FFFFD, and U+100000 to U+10FFFD are available for private use.
This refers to code points of the standard whose interpretation is
not specified by the standard and whose use may be determined by
private agreement among cooperating users. <UNICODE>
The use of these "private use" characters is defined by the
parties who transmit and receive them, and is thus not appropriate
for standardization. (The IETF has a long history of private use
names for things such as "x-" names in MIME types, charsets, and
languages. The experience with these has been quite negative,
with many implementors assuming that private use names are in fact
public and long-lived.)
8. Security Considerations
Security is not discussed in this document.
9. References
9.1 Normative References
[ISOIEC10646] ISO/IEC 10646-1:2000. International Standard --
Information technology -- Universal Multiple-Octet
Coded Character Set (UCS) -- Part 1: Architecture and
Basic Multilingual Plane, 2000.
[UNICODE] The Unicode Standard, Version 3.2.0 is defined by The
Unicode Standard, Version 3.0 (Reading, MA, Addison-
Wesley, 2000. ISBN 0-201-61633-5), as amended by the
Unicode Standard Annex #27: Unicode 3.1
(http://www.unicode.org/reports/tr27/) and by the
Unicode Standard Annex #28: Unicode 3.2
(http://www.unicode.org/reports/tr28/), The Unicode
Consortium, 2002.
9.2 Informative References
[CHARMOD] Character Model for the World Wide Web 1.0, W3C,
<http://www.w3.org/TR/charmod/>.
[FRAMEWORK] ISO/IEC TR 11017:1997(E). Information technology -
Framework for internationalization, prepared by ISO/IEC
JTC 1/SC 22/WG 20, 1997.
[ISO 639] ISO 639:2000 (E/F) - Code for the representation of
names of languages, 2000.
[ISO 3166] ISO 3166:1988 (E/F) - Codes for the representation of
names of countries, 2000.
[RFC2045] Freed, N. and N. Borenstein, "MIME Part One: Format of
Internet Message Bodies", November 1996.
[RFC2047] Moore, K., "MIME Part Three: Message Header Extensions
for Non-ASCII Text", RFC2047, November 1996.
[RFC2277] Alvestrand, H., "IETF Policy on Character Sets and
Languages", BCP 18, RFC2277, January 1998.
[RFC2279] Yergeau, F., "UTF-8, a transformation format of ISO
10646", RFC2279, January 1998.
[RFC2781] Hoffman, P. and F. Yergeau, "UTF-16, an encoding of ISO
10646", RFC2781, February 2000.
[RFC2822] Resnick, P., "Internet Message Format", RFC2822, April
2001.
[RFC3066] Alvestrand, H., "Tags for the Identification of
Languages", BCP 47, RFC3066, January 2001.
[US-ASCII] Coded Character Set -- 7-bit American Standard Code for
Information Interchange, ANSI X3.4-1986, 1986.
[UTN6] "BOCU-1: MIME-Compatible Unicode Compression", M.
Scherer & M. Davis, Unicode Technical Note #6.
[UTR6] "A Standard Compression Scheme for Unicode", M. Wolf,
et. al., Unicode Technical Report #6.
[UTR15] "Unicode Normalization Forms", M. Davis & M. Duerst,
Unicode Technical Report #15.
[UTR18] "Unicode Regular Expression Guidelines", M. Davis,
Unicode Technical Report #18.
[UTR19] "UTF-32", M. Davis, Unicode Technical Report #19.
[UTR22] "Character Mapping Markup Language", M. Davis, Unicode
Technical Report #22.
10. Additional Interesting Reading
ALA-LC Romanization Tables, Randall Barry (ed.), U.S. Library of
Congress, 1997, ISBN 0844409405
Blackwell Encyclopedia of Writing Systems, Florian Coulmas, Blackwell
Publishers, 1999, ISBN 063121481X
The World's Writing Systems, Peter Daniels and William Bright, Oxford
University Press, 1996, ISBN 0195079930
Writing Systems of the World, Akira Nakanishi, Charles E. Tuttle
Company, 1980, ISBN 0804816549
11. Index
alphabetic -- 4.1
ASCII-compatible encoding (ACE) -- 6
ASN.1 text formats -- 6
Base64 -- 6
Basic Multilingual Plane (BMP) -- 3.2
bidirectional display -- 5
BOCU-1 -- 3.2
case -- 4
character -- 2
character encoding form -- 2
character encoding scheme -- 2
charset -- 2
charset identification -- 6
CJK characters and Han characters -- 7
code chart -- 4
code table -- 4
coded character -- 2
coded character set -- 2
combining character -- 4
compatibility character -- 4.1
composite sequence -- 4
control character -- 4.1
diacritic -- 4.1
displaying and rendering text -- 2
font -- 5
formatting character -- 4.1
glyph -- 2
glyph code -- 2
graphic symbol -- 5
i18n, l10n -- 2
ideographic -- 4.1
input methods -- 5
internationalization -- 2
ISO -- 3.1
language -- 2
language identification -- 6
Latin characters -- 7
local and regional standards organizations -- 3.1
locale -- 7
localization -- 2
MIME -- 6
multilingual -- 2
name spaces -- 6
nonspacing character -- 4.1
normalization -- 4
on-the-wire encoding -- 6
parsed text -- 6
private use -- 7
protocol elements -- 6
punctuation -- 4.1
quoted printable -- 6
regular expressions -- 7
rendering rules -- 5
romanization -- 7
script -- 2
SCSU -- 3.2
sorting and collation -- 4
symbol -- 4.1
transcoding -- 2
transcription -- 7
transfer encoding syntax -- 6
translation -- 7
transliteration -- 7
UCS-2 and UCS-4 -- 3.2
undisplayable character -- 5
Unicode Consortium -- 3.1
UTF-32 -- 3.2
UTF-16, UTF-16BE, and UTF-16LE -- 3.2
UTF-8 -- 3.2
World Wide Web Consortium -- 3.1
XML -- 6
A. Acknowledgements
The definitions in this document come from many sources, including a
wide variety of IETF documents.
James Seng contributed to the initial outline of this document.
Harald Alvestrand and Martin Duerst made extensive useful comments on
early versions. Others who contributed to the development include:
Dan Kohn
Jacob Palme
Johan van Wingen
Peter Constable
Yuri Demchenko
Susan Harris
Zita Wenzel
John Klensin
Henning Schulzrinne
Leslie Daigle
Markus Scherer
Ken Whistler
B. Author's Address
Paul Hoffman
Internet Mail Consortium and VPN Consortium
127 Segre Place
Santa Cruz, CA 95060 USA
EMail: paul.hoffman@imc.org and paul.hoffman@vpnc.org
Full Copyright Statement
Copyright (C) The Internet Society (2003). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement
Funding for the RFCEditor function is currently provided by the
Internet Society.