elided and thus should be removed by scheme-based normalization. For
example, the second URI above is the normal form for the "http"
scheme.
Another case where normalization varies by scheme is in the handling
of an empty authority component or empty host subcomponent. For many
scheme specifications, an empty authority or host is considered an
error; for others, it is considered equivalent to "localhost" or the
end-user’s host. When a scheme defines a default for authority and a
URI reference to that default is desired, the reference should be
normalized to an empty authority for the sake of uniformity, brevity,
and internationalization. If, however, either the userinfo or port
subcomponents are non-empty, then the host should be given explicitly
even if it matches the default.
Normalization should not remove delimiters when their associated
component is empty unless licensed to do so by the scheme
specification. For example, the URI "http://example.com/?" cannot be
assumed to be equivalent to any of the examples above. Likewise, the
presence or absence of delimiters within a userinfo subcomponent is
usually significant to its interpretation. The fragment component is
not subject to any scheme-based normalization; thus, two URIs that
differ only by the suffix "#" are considered different regardless of
the scheme.
Some schemes define additional subcomponents that consist of case-
insensitive data, giving an implicit license to normalizers to
convert this data to a common case (e.g., all lowercase). For
example, URI schemes that define a subcomponent of path to contain an
Internet hostname, such as the "mailto" URI scheme, cause that
subcomponent to be case-insensitive and thus subject to case
normalization (e.g., "mailto:Joe@Example.COM" is equivalent to
"mailto:Joe@example.com", even though the generic syntax considers
the path component to be case-sensitive).
Other scheme-specific normalizations are possible.
6.2.4. Protocol-Based Normalization
Substantial effort to reduce the incidence of false negatives is
often cost-effective for web spiders. Therefore, they implement even
more aggressive techniques in URI comparison. For example, if they
observe that a URI such as
http://example.com/data
redirects to a URI differing only in the trailing slash
http://example.com/data/
they will likely regard the two as equivalent in the future. This
kind of technique is only appropriate when equivalence is clearly
indicated by both the result of accessing the resources and the
common conventions of their scheme’s dereference algorithm (in this
case, use of redirection by HTTP origin servers to avoid problems
with relative references).
7. Security Considerations
A URI does not in itself pose a security threat. However, as URIs
are often used to provide a compact set of instructions for access to
network resources, care must be taken to properly interpret the data
within a URI, to prevent that data from causing unintended access,
and to avoid including data that should not be revealed in plain
text.
7.1. Reliability and Consistency
There is no guarantee that once a URI has been used to retrieve
information, the same information will be retrievable by that URI in
the future. Nor is there any guarantee that the information
retrievable via that URI in the future will be observably similar to
that retrieved in the past. The URI syntax does not constrain how a
given scheme or authority apportions its namespace or maintains it
over time. Such guarantees can only be obtained from the person(s)
controlling that namespace and the resource in question. A specific
URI scheme may define additional semantics, such as name persistence,
if those semantics are required of all naming authorities for that
scheme.
7.2. Malicious Construction
It is sometimes possible to construct a URI so that an attempt to
perform a seemingly harmless, idempotent operation, such as the
retrieval of a representation, will in fact cause a possibly damaging
remote operation. The unsafe URI is typically constructed by
specifying a port number other than that reserved for the network
protocol in question. The client unwittingly contacts a site running
a different protocol service, and data within the URI contains
instructions that, when interpreted according to this other protocol,
cause an unexpected operation. A frequent example of such abuse has
been the use of a protocol-based scheme with a port component of
"25", thereby fooling user agent software into sending an unintended
or impersonating message via an SMTP server.
Applications should prevent dereference of a URI that specifies a TCP
port number within the "well-known port" range (0 - 1023) unless the
protocol being used to dereference that URI is compatible with the
protocol expected on that well-known port. Although IANA maintains a
registry of well-known ports, applications should make such
restrictions user-configurable to avoid preventing the deployment of
new services.
When a URI contains percent-encoded octets that match the delimiters
for a given resolution or dereference protocol (for example, CR and
LF characters for the TELNET protocol), these percent-encodings must
not be decoded before transmission across that protocol. Transfer of
the percent-encoding, which might violate the protocol, is less
harmful than allowing decoded octets to be interpreted as additional
operations or parameters, perhaps triggering an unexpected and
possibly harmful remote operation.
7.3. Back-End Transcoding
When a URI is dereferenced, the data within it is often parsed by
both the user agent and one or more servers. In HTTP, for example, a
typical user agent will parse a URI into its five major components,
access the authority’s server, and send it the data within the
authority, path, and query components. A typical server will take
that information, parse the path into segments and the query into
key/value pairs, and then invoke implementation-specific handlers to
respond to the request. As a result, a common security concern for
server implementations that handle a URI, either as a whole or split
into separate components, is proper interpretation of the octet data
represented by the characters and percent-encodings within that URI.
Percent-encoded octets must be decoded at some point during the
dereference process. Applications must split the URI into its
components and subcomponents prior to decoding the octets, as
otherwise the decoded octets might be mistaken for delimiters.
Security checks of the data within a URI should be applied after
decoding the octets. Note, however, that the "%00" percent-encoding
(NUL) may require special handling and should be rejected if the
application is not expecting to receive raw data within a component.
Special care should be taken when the URI path interpretation process
involves the use of a back-end file system or related system
functions. File systems typically assign an operational meaning to
special characters, such as the "/", "\", ":", "[", and "]"
characters, and to special device names like ".", "..", "...", "aux",
"lpt", etc. In some cases, merely testing for the existence of such
a name will cause the operating system to pause or invoke unrelated
system calls, leading to significant security concerns regarding
denial of service and unintended data transfer. It would be
impossible for this specification to list all such significant
characters and device names. Implementers should research the
reserved names and characters for the types of storage device that
may be attached to their applications and restrict the use of data
obtained from URI components accordingly.
7.4. Rare IP Address Formats
Although the URI syntax for IPv4address only allows the common
dotted-decimal form of IPv4 address literal, many implementations
that process URIs make use of platform-dependent system routines,
such as gethostbyname() and inet_aton(), to translate the string
literal to an actual IP address. Unfortunately, such system routines
often allow and process a much larger set of formats than those
described in Section 3.2.2.
For example, many implementations allow dotted forms of three
numbers, wherein the last part is interpreted as a 16-bit quantity
and placed in the right-most two bytes of the network address (e.g.,
a Class B network). Likewise, a dotted form of two numbers means
that the last part is interpreted as a 24-bit quantity and placed in
the right-most three bytes of the network address (Class A), and a
single number (without dots) is interpreted as a 32-bit quantity and
stored directly in the network address. Adding further to the
confusion, some implementations allow each dotted part to be
interpreted as decimal, octal, or hexadecimal, as specified in the C
language (i.e., a leading 0x or 0X implies hexadecimal; a leading 0
implies octal; otherwise, the number is interpreted as decimal).
These additional IP address formats are not allowed in the URI syntax
due to differences between platform implementations. However, they
can become a security concern if an application attempts to filter
access to resources based on the IP address in string literal format.
If this filtering is performed, literals should be converted to
numeric form and filtered based on the numeric value, and not on a
prefix or suffix of the string form.
7.5. Sensitive Information
URI producers should not provide a URI that contains a username or
password that is intended to be secret. URIs are frequently
displayed by browsers, stored in clear text bookmarks, and logged by
user agent history and intermediary applications (proxies). A
password appearing within the userinfo component is deprecated and
should be considered an error (or simply ignored) except in those
rare cases where the ’password’ parameter is intended to be public.
7.6. Semantic Attacks
Because the userinfo subcomponent is rarely used and appears before
the host in the authority component, it can be used to construct a
URI intended to mislead a human user by appearing to identify one
(trusted) naming authority while actually identifying a different
authority hidden behind the noise. For example
ftp://cnn.example.com&story=breaking_news@10.0.0.1/top_story.htm
might lead a human user to assume that the host is ’cnn.example.com’,
whereas it is actually ’10.0.0.1’. Note that a misleading userinfo
subcomponent could be much longer than the example above.
A misleading URI, such as that above, is an attack on the user’s
preconceived notions about the meaning of a URI rather than an attack
on the software itself. User agents may be able to reduce the impact
of such attacks by distinguishing the various components of the URI
when they are rendered, such as by using a different color or tone to
render userinfo if any is present, though there is no panacea. More
information on URI-based semantic attacks can be found in [Siedzik].
8. IANA Considerations
URI scheme names, as defined by <scheme> in Section 3.1, form a
registered namespace that is managed by IANA according to the
procedures defined in [BCP35]. No IANA actions are required by this
document.
9. Acknowledgements
This specification is derived from RFC 2396 [RFC2396], RFC 1808
[RFC1808], and RFC 1738 [RFC1738]; the acknowledgements in those
documents still apply. It also incorporates the update (with
corrections) for IPv6 literals in the host syntax, as defined by
Robert M. Hinden, Brian E. Carpenter, and Larry Masinter in
[RFC2732]. In addition, contributions by Gisle Aas, Reese Anschultz,
Daniel Barclay, Tim Bray, Mike Brown, Rob Cameron, Jeremy Carroll,
Dan Connolly, Adam M. Costello, John Cowan, Jason Diamond, Martin
Duerst, Stefan Eissing, Clive D.W. Feather, Al Gilman, Tony Hammond,
Elliotte Harold, Pat Hayes, Henry Holtzman, Ian B. Jacobs, Michael
Kay, John C. Klensin, Graham Klyne, Dan Kohn, Bruce Lilly, Andrew
Main, Dave McAlpin, Ira McDonald, Michael Mealling, Ray Merkert,
Stephen Pollei, Julian Reschke, Tomas Rokicki, Miles Sabin, Kai
Schaetzl, Mark Thomson, Ronald Tschalaer, Norm Walsh, Marc Warne,
Stuart Williams, and Henry Zongaro are gratefully acknowledged.
10. References
10.1. Normative References
[ASCII] American National Standards Institute, "Coded Character
Set -- 7-bit American Standard Code for Information
Interchange", ANSI X3.4, 1986.
[RFC2234] Crocker, D. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", RFC 2234, November 1997.
[STD63] Yergeau, F., "UTF-8, a transformation format of
ISO 10646", STD 63, RFC 3629, November 2003.
[UCS] International Organization for Standardization,
"Information Technology - Universal Multiple-Octet Coded
Character Set (UCS)", ISO/IEC 10646:2003, December 2003.
10.2. Informative References
[BCP19] Freed, N. and J. Postel, "IANA Charset Registration
Procedures", BCP 19, RFC 2978, October 2000.
[BCP35] Petke, R. and I. King, "Registration Procedures for URL
Scheme Names", BCP 35, RFC 2717, November 1999.
[RFC0952] Harrenstien, K., Stahl, M., and E. Feinler, "DoD Internet
host table specification", RFC 952, October 1985.
[RFC1034] Mockapetris, P., "Domain names - concepts and facilities",
STD 13, RFC 1034, November 1987.
[RFC1123] Braden, R., "Requirements for Internet Hosts - Application
and Support", STD 3, RFC 1123, October 1989.
[RFC1535] Gavron, E., "A Security Problem and Proposed Correction
With Widely Deployed DNS Software", RFC 1535,
October 1993.
[RFC1630] Berners-Lee, T., "Universal Resource Identifiers in WWW: A
Unifying Syntax for the Expression of Names and Addresses
of Objects on the Network as used in the World-Wide Web",
RFC 1630, June 1994.
[RFC1736] Kunze, J., "Functional Recommendations for Internet
Resource Locators", RFC 1736, February 1995.
[RFC1737] Sollins, K. and L. Masinter, "Functional Requirements for
Uniform Resource Names", RFC 1737, December 1994.
[RFC1738] Berners-Lee, T., Masinter, L., and M. McCahill, "Uniform
Resource Locators (URL)", RFC 1738, December 1994.
[RFC1808] Fielding, R., "Relative Uniform Resource Locators",
RFC 1808, June 1995.
[RFC2046] Freed, N. and N. Borenstein, "Multipurpose Internet Mail
Extensions (MIME) Part Two: Media Types", RFC 2046,
November 1996.
[RFC2141] Moats, R., "URN Syntax", RFC 2141, May 1997.
[RFC2396] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform
Resource Identifiers (URI): Generic Syntax", RFC 2396,
August 1998.
[RFC2518] Goland, Y., Whitehead, E., Faizi, A., Carter, S., and D.
Jensen, "HTTP Extensions for Distributed Authoring --
WEBDAV", RFC 2518, February 1999.
[RFC2557] Palme, J., Hopmann, A., and N. Shelness, "MIME
Encapsulation of Aggregate Documents, such as HTML
(MHTML)", RFC 2557, March 1999.
[RFC2718] Masinter, L., Alvestrand, H., Zigmond, D., and R. Petke,
"Guidelines for new URL Schemes", RFC 2718, November 1999.
[RFC2732] Hinden, R., Carpenter, B., and L. Masinter, "Format for
Literal IPv6 Addresses in URL’s", RFC 2732, December 1999.
[RFC3305] Mealling, M. and R. Denenberg, "Report from the Joint
W3C/IETF URI Planning Interest Group: Uniform Resource
Identifiers (URIs), URLs, and Uniform Resource Names
(URNs): Clarifications and Recommendations", RFC 3305,
August 2002.
[RFC3490] Faltstrom, P., Hoffman, P., and A. Costello,
"Internationalizing Domain Names in Applications (IDNA)",
RFC 3490, March 2003.
[RFC3513] Hinden, R. and S. Deering, "Internet Protocol Version 6
(IPv6) Addressing Architecture", RFC 3513, April 2003.
[Siedzik] Siedzik, R., "Semantic Attacks: What’s in a URL?",
April 2001, <http://www.giac.org/practical/gsec/
Richard_Siedzik_GSEC.pdf>.
Appendix A. Collected ABNF for URI
URI = scheme ":" hier-part [ "?" query ] [ "#" fragment ]
hier-part = "//" authority path-abempty
/ path-absolute
/ path-rootless
/ path-empty
URI-reference = URI / relative-ref
absolute-URI = scheme ":" hier-part [ "?" query ]
relative-ref = relative-part [ "?" query ] [ "#" fragment ]
relative-part = "//" authority path-abempty
/ path-absolute
/ path-noscheme
/ path-empty
scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )
authority = [ userinfo "@" ] host [ ":" port ]
userinfo = *( unreserved / pct-encoded / sub-delims / ":" )
host = IP-literal / IPv4address / reg-name
port = *DIGIT
IP-literal = "[" ( IPv6address / IPvFuture ) "]"
IPvFuture = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" )
IPv6address = 6( h16 ":" ) ls32
/ "::" 5( h16 ":" ) ls32
/ [ h16 ] "::" 4( h16 ":" ) ls32
/ [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32
/ [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
/ [ *3( h16 ":" ) h16 ] "::" h16 ":" ls32
/ [ *4( h16 ":" ) h16 ] "::" ls32
/ [ *5( h16 ":" ) h16 ] "::" h16
/ [ *6( h16 ":" ) h16 ] "::"
h16 = 1*4HEXDIG
ls32 = ( h16 ":" h16 ) / IPv4address
IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet
dec-octet = DIGIT ; 0-9
/ %x31-39 DIGIT ; 10-99
/ "1" 2DIGIT ; 100-199
/ "2" %x30-34 DIGIT ; 200-249
/ "25" %x30-35 ; 250-255
reg-name = *( unreserved / pct-encoded / sub-delims )
path = path-abempty ; begins with "/" or is empty
/ path-absolute ; begins with "/" but not "//"
/ path-noscheme ; begins with a non-colon segment
/ path-rootless ; begins with a segment
/ path-empty ; zero characters
path-abempty = *( "/" segment )
path-absolute = "/" [ segment-nz *( "/" segment ) ]
path-noscheme = segment-nz-nc *( "/" segment )
path-rootless = segment-nz *( "/" segment )
path-empty = 0<pchar>
segment = *pchar
segment-nz = 1*pchar
segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims / "@" )
; non-zero-length segment without any colon ":"
pchar = unreserved / pct-encoded / sub-delims / ":" / "@"
query = *( pchar / "/" / "?" )
fragment = *( pchar / "/" / "?" )
pct-encoded = "%" HEXDIG HEXDIG
unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"
reserved = gen-delims / sub-delims
gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
sub-delims = "!" / "$" / "&" / "’" / "(" / ")"
/ "*" / "+" / "," / ";" / "="
Appendix B. Parsing a URI Reference with a Regular Expression
As the "first-match-wins" algorithm is identical to the "greedy"
disambiguation method used by POSIX regular expressions, it is
natural and commonplace to use a regular expression for parsing the
potential five components of a URI reference.
The following line is the regular expression for breaking-down a
well-formed URI reference into its components.
^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?
12 3 4 5 6 7 8 9
The numbers in the second line above are only to assist readability;
they indicate the reference points for each subexpression (i.e., each
paired parenthesis). We refer to the value matched for subexpression
<n> as $<n>. For example, matching the above expression to
http://www.ics.uci.edu/pub/ietf/uri/#Related
results in the following subexpression matches:
$1 = http:
$2 = http
$3 = //www.ics.uci.edu
$4 = www.ics.uci.edu
$5 = /pub/ietf/uri/
$6 = <undefined>
$7 = <undefined>
$8 = #Related
$9 = Related
where <undefined> indicates that the component is not present, as is
the case for the query component in the above example. Therefore, we
can determine the value of the five components as
scheme = $2
authority = $4
path = $5
query = $7
fragment = $9
Going in the opposite direction, we can recreate a URI reference from
its components by using the algorithm of Section 5.3.
Appendix C. Delimiting a URI in Context
URIs are often transmitted through formats that do not provide a
clear context for their interpretation. For example, there are many
occasions when a URI is included in plain text; examples include text
sent in email, USENET news, and on printed paper. In such cases, it
is important to be able to delimit the URI from the rest of the text,
and in particular from punctuation marks that might be mistaken for
part of the URI.
In practice, URIs are delimited in a variety of ways, but usually
within double-quotes "http://example.com/", angle brackets
<http://example.com/>, or just by using whitespace:
http://example.com/
These wrappers do not form part of the URI.
In some cases, extra whitespace (spaces, line-breaks, tabs, etc.) may
have to be added to break a long URI across lines. The whitespace
should be ignored when the URI is extracted.
No whitespace should be introduced after a hyphen ("-") character.
Because some typesetters and printers may (erroneously) introduce a
hyphen at the end of line when breaking it, the interpreter of a URI
containing a line break immediately after a hyphen should ignore all
whitespace around the line break and should be aware that the hyphen
may or may not actually be part of the URI.