We define a "reuse" of R as as occurring when the proxy responds to a
request selecting R with a 304 (Not Modified) status, unless that
request is a Range request that does not specify byte #0 of the
entity.
5.3.1 Counting rules for hit-metering
A proxy participating in hit-metering for a cache response R
maintains two counters, CU and CR, associated with R. When a proxy
first stores R in its cache, it sets both CU and CR to 0 (zero).
When a subsequent client request results in a "use" of R, the proxy
increments CU. When a subsequent client request results in a "reuse"
of R, the proxy increments CR. When a subsequent client request
selecting R (i.e., including V) includes a "count" Meter directive,
the proxy increments CU and CR using the corresponding values in the
directive.
When the proxy sends a request selecting R (i.e., including V) to the
inbound server, it includes a "count" Meter directive with the
current CU and CR as the parameter values. If this request was
caused by the proxy's receipt of a request from a client, upon
receipt of the server's response, the proxy sets CU and CR to the
number of uses and reuses, respectively, that may have occurred while
the request was in progress. (These numbers are likely, but not
certain, to be zero.) If the proxy's request was a final HEAD-based
report, it need no longer maintain the CU and CR values, but it may
also set them to the number of intervening uses and reuses and retain
them.
5.3.2 Counting rules for usage-limiting
A proxy participating in usage-limiting for a response R maintains
either or both of two counters TU and TR, as appropriate, for that
resource. TU and TR are incremented in just the same way as CU and
CR, respectively. However, TU is zeroed only upon receipt of a
"max-uses" Meter directive for that response (including the initial
receipt). Similarly, TR is zeroed only upon receipt of a "max-
reuses" Meter directive for that response.
A proxy participating in usage-limiting for a response R also stores
values MU and/or MR associated with R. When it receives a response
including only a max-uses value, it sets MU to that value and MR to
infinity. When it receives a response including only a max-reuses
value, it sets MR to that value and MU to infinity. When it receives
a response including both max-reuses and max-reuses values, it sets
MU and MR to those values, respectively. When it receives a
subsequent response including neither max-reuses nor max-reuses
values, it sets both MU and MR to infinity.
If a proxy participating in usage-limiting for a response R receives
a request that would cause a "use" of R, and TU >= MU, it MUST
forward the request to the server. If it receives a request that
would cause a "reuse" of R, and TR >= MR, it MUST forward the request
to the server. If (in either case) the proxy has already forwarded a
previous request to the server and is waiting for the response, it
should delay further handling of the new request until the response
arrives (or times out); it SHOULD NOT have two revalidation requests
pending at once that select the same response, unless these are Range
requests selecting different subranges.
There is a special case of this rule for the "max-uses" directive: if
the proxy receives a response with "max-uses=0" and does not forward
it to a requesting client, the proxy should set a flag PF associated
with R. If R is true, then when a request arrives while if TU >= MU,
if the PF flag is set, then the request need not be forwarded to the
server (provided that this is not required by other caching rules).
However, the PF flag MUST be cleared on any use of the response.
Note: the "PF" flag is so named because this feature is useful
only for caches that could issue a "prefetch" request before an
actual client request for the response. A proxy not implementing
prefetching need not implement the PF flag.
5.3.3 Equivalent algorithms are allowed
Any other algorithm that exhibits the same external behavior (i.e.,
generates exactly the same requests from the proxy to the server) as
the one in this section is explicitly allowed.
Note: in most cases, TU will be equal to CU, and TR will be
equal to CR. The only two cases where they could differ are:
1. The proxy issues a non-conditional request for the
resource using V, while TU and/or TR are non-zero, and
the server's response includes a new "max-uses" and/or
"max-reuses" directive (thus zeroing TU and/or TR, but
not CU and CR).
2. The proxy issues a conditional request reporting the
hit-counts (and thus zeroing CU and CR, but not TU or
TR), but the server's response does not include a new
"max-uses" and/or "max-reuses" directive.
To solve the first case, the proxy has several implementation
options
- Always store TU and TR separately from CU and CR.
- Create "shadow" copies of TU and TR when this situation
arises (analogous to "copy on write").
- Generate a HEAD-based usage report when the
non-conditional request is sent (or when the
"max-uses=0" is received), causing CU and CR to be
zeroed (analogous in some ways to a "memory barrier"
instruction).
In the second case, the server implicitly has removed the
usage-limit(s) on the response (by setting MU and/or MR to
infinity), and so the fact that, say, TU is different from CU
is not significant.
Note: It may also be possible to eliminate the PF flag by
sending extra HEAD-based usage-report requests, but we
recommend against this; it is better to allocate an extra bit
per entry than to transmit extra requests.
5.4 Counting rules: interaction with Range requests
HTTP/1.1 allows a client to request sub-ranges of a resource. A
client might end up issuing several requests with the net effect of
receiving one copy of the resource. For uniformity of the results
seen by origin servers, proxies need to observe a rule for counting
these references, although it is not clear that one rule generates
accurate results in every case.
The rule established in this specification is that proxies count as a
"use" or "reuse" only those Range requests that result in the return
of byte #0 of the resource. The rationale for this rule is that in
almost every case, an end-client will retrieve the beginning of any
resource that it references at all, and that it will seldom retrieve
any portion more than once. Therefore, this rule appears to meet the
goal of a "best-efforts" approximation.
5.5 Implementation by non-caching proxies
A non-caching proxy may participate in the metering subtree; this is
strongly recommended.
A non-caching proxy (HTTP/1.1 or higher) that participates in the
metering subtree SHOULD forward Meter headers on both requests and
responses, with the appropriate Connection headers.
If a non-caching proxy forwards Meter headers, it MUST comply with
these restrictions:
1. If the proxy forwards Meter headers in responses, such a
response MUST NOT be returned to any request except the
one that elicited it.
2. Once a non-caching proxy starts forwarding Meter headers,
it should not arbitrarily stop forwarding them (or else
reports may be lost).
A proxy that caches some responses and not others, for whatever
reason, may choose to implement the Meter header as a caching proxy
for the responses that it caches, and as a non-caching proxy for the
responses that it does not cache, as long as its external behavior
with respect to any particularly response is fully consistent with
this specification.
5.6 Implementation by cooperating caches
Several HTTP cache implementations, most notably the Harvest/Squid
cache [2], create cooperative arrangements between several caches.
If such caches use a protocol other than HTTP to communicate between
themselves, such as the Internet Cache Protocol (ICP) [12], and if
they implement the Meter header, then they MUST act to ensure that
their cooperation does not violate the intention of this
specification.
In particular, if one member of a group of cooperating caches agrees
with a server to hit-meter a particular response, and then passes
this response via a non-HTTP protocol to a second cache in the group,
the caches MUST ensure that the server which requested the metering
receives reports that appropriately account for any uses or resues
made by the second cache. Similarly, if the first cache agreed to
usage-limit the response, the total number of uses by the group of
caches MUST be limited to the agreed-upon number.
6 Examples
6.1 Example of a complete set of exchanges
This example shows how the protocol is intended to be used most of
the time: for hit-metering without usage-limiting. Entity bodies are
omitted.
A client sends request to a proxy:
GET http://foo.com/bar.html HTTP/1.1
The proxy forwards request to the origin server:
GET /bar.html HTTP/1.1
Host: foo.com
Connection: Meter
thus offering (implicitly) "will-report-and-limit".
The server responds to the proxy:
HTTP/1.1 200 OK
Date: Fri, 06 Dec 1996 18:44:29 GMT
Cache-control: max-age=3600
Connection: meter
Etag: "abcde"
thus (implicitly) requiring "do-report" (but not requiring
usage-limiting).
The proxy responds to the client:
HTTP/1.1 200 OK
Date: Fri, 06 Dec 1996 18:44:29 GMT
Etag: "abcde"
Cache-control: max-age=3600, proxy-mustcheck
Age: 1
Since the proxy does not know if its client is an end-system, or a
proxy that doesn't do metering, it adds the "proxy-mustcheck"
directive.
Another client soon asks for the resource:
GET http://foo.com/bar.html HTTP/1.1
and the proxy sends the same response as it sent to the other client,
except (perhaps) for the Age value.
After an hour has passed, a third client asks for the response:
GET http://foo.com/bar.html HTTP/1.1
But now the response's max-age has been exceeded, so the proxy
revalidates the response with the origin server:
GET /bar.html HTTP/1.1
If-None-Match: "abcde"
Host: foo.com
Connection: Meter
Meter: count=1/0
thus simultaneously fulfilling its duties to validate the response
and to report the one "use" that wasn't forwarded.
The origin server responds:
HTTP/1.1 304 Not Modified
Date: Fri, 06 Dec 1996 19:44:29 GMT
Cache-control: max-age=3600
Etag: "abcde"
so the proxy can use the original response to reply to the new
client; the proxy also zeros the use-count it associates with that
response.
Another client soon asks for the resource:
GET http://foo.com/bar.html HTTP/1.1
and the proxy sends the appropriate response.
After another few hours, the proxy decides to remove the cache entry.
When it does so, it sends to the origin server:
HEAD /bar.html HTTP/1.1
If-None-Match: "abcde"
Host: foo.com
Connection: Meter
Meter: count=1/0
reporting that one more use of the response was satisfied from the
cache.
6.2 Protecting against HTTP/1.0 proxies
An origin server that does not want HTTP/1.0 caches to store the
response at all, and is willing to have HTTP/1.0 end-system clients
generate excess GETs (which will be forwarded by HTTP/1.0 proxies)
could send this for its reply:
HTTP/1.1 200 OK
Cache-control: max-age=3600
Connection: meter
Etag: "abcde"
Expires: Sun, 06 Nov 1994 08:49:37 GMT
HTTP/1.0 caches will see the ancient Expires header, but HTTP/1.1
caches will see the max-age directive and will ignore Expires.
Note: although most major HTTP/1.0 proxy implementations observe
the Expires header, it is possible that some are in use that do
not. Use of the Expires header to prevent caching by HTTP/1.0
proxies might not be entirely reliable.
6.3 More elaborate examples
Here is a request from a proxy that is willing to hit-meter but is
not willing to usage-limit:
GET /bar.html HTTP/1.1
Host: foo.com
Connection: Meter
Meter: wont-limit
Here is a response from an origin server that does not want hit
counting, but does want "uses" limited to 3, and "reuses" limited to
6:
HTTP/1.1 200 OK
Cache-control: max-age=3600
Connection: meter
Etag: "abcde"
Expires: Sun, 06 Nov 1994 08:49:37 GMT
Meter: max-uses=3, max-reuses=6, dont-report
Here is the same example with abbreviated Meter directive names:
HTTP/1.1 200 OK
Cache-control: max-age=3600
Connection: meter
Etag: "abcde"
Expires: Sun, 06 Nov 1994 08:49:37 GMT
Meter:u=3,r=6,e
7 Interactions with content negotiation
This section describes two aspects of the interaction between hit-
metering and "content-negotiated" resources:
1. treatment of responses carrying a Vary header (section
7.1).
2. treatment of responses that use the proposed Transparent
Content Negotiation mechanism (section 7.2).
7.1 Treatment of responses carrying a Vary header
Separate counts should be kept for each combination of the headers
named in the Vary header for the Request-URI (what [4] calls "the
selecting request-headers"), even if they map to the same entity-tag.
This rule has the effect of counting hits on each variant, if there
are multiple variants of a page available.
Note: This interaction between Vary and the hit-counting
directives allows the origin server a lot of flexibility in
specifying how hits should be counted. In essence, the origin
server uses the Vary mechanism to divide the requests for a
resource into arbitrary categories, based on the request- headers.
(We will call these categories "request-patterns".) Since a proxy
keeps its hit-counts for each request-pattern, rather than for
each resource, the origin server can obtain separate statistics
for many aspects of an HTTP request.
For example, if a page varied based on the value of the User-Agent
header in the requests, then hit counts would be kept for each
different flavor of browser. But it is in fact more general than
that; because multiple header combinations can map to the same
variant, it also enables the origin server to count the number of
times (e.g.) the Swahili version of a page was requested, even though
it is only available in English.
If a proxy does not support the Vary mechanism, then [4] says that it
MUST NOT cache any response that carries a Vary header, and hence
need not implement any aspect of this hit-counting or usage-limiting
design for varying resources.
Note: this also implies that if a proxy supports the Vary
mechanism but is not willing to maintain independent hit-counts
for each variant response in its cache, then it must follow at
least one of these rules:
1. It must not use the Meter header in a request to offer
to hit-meter or usage-limit responses.
2. If it does offer to hit-meter or usage-limit responses,
and then receives a response that includes both a Vary
header and a Meter header with a directive that it
cannot satisfy, then the proxy must not cache the
response.
In other words, a proxy is allowed to partially implement the
Vary mechanism with respect to hit-metering, as long as this has
no externally visible effect on its ability to comply with the
Meter specification.
This approach works for counting almost any aspect of the request
stream, without embedding any specific list of countable aspects in
the specification or proxy implementation.
7.2 Interaction with Transparent Content Negotiation
[A description of the interaction between this design and the
proposed Transparent Content Negotiation (TCN) design [6] will be
made available in a later document.]
8 A Note on Capturing Referrals
It is alleged that some advertisers want to pay content providers,
not by the "hit", but by the "nibble" -- the number of people who
actually click on the ad to get more information.
Now, HTTP already has a mechanism for doing this: the "Referer"
header. However, perhaps it ought to be disabled for privacy reasons
-- according the HTTP/1.1 spec:
"Because the source of the link may be private information or may
reveal an otherwise private information source, it is strongly
recommended that the user be able to select whether or not the
Referer field is sent."
However, in the case of ads, the source of the link actually wants to
let the referred-to page know where the reference came from.
This does not require the addition of any extra mechanism, but rather
can use schemes that embed the referrer in the URI in a manner
similar to this:
http://www.blah.com/ad-reference?from=site1
Such a URI should point to a resource (perhaps a CGI script) which
returns a 302 redirect to the real page
http://www.blah.com/ad-reference.html
Proxies which do not cache 302s will cause one hit on the redirection
page per use, but the real page will get cached. Proxies which do
cache 302s and report hits on the cached 302s will behave optimally.
This approach has the advantage that it works whether or not the
end-client has disabled the use of Referer. Combined with the rest
of the hit-metering proposal in this design, this approach allows,
for example, an advertiser to know how often a reference to an
advertisement was made from a particular page.
9 Alternative proposals
There might be a number of other ways of gathering demographic and
usage information; other mechanisms might respond to a different set
of needs than this proposal does. This proposal certainly does not
preclude the proposal or deployment of other such mechanisms, and
many of them may be complementary to and compatible with the
mechanism proposed here.
There has been some speculation that statistical sampling methods
might be used to gather reasonably accurate data. One such proposal
is to manipulate cache expiration times so that selected resources
are uncachable for carefully chosen periods, allowing servers to
accurately count accesses during those periods. The hit-metering
mechanism proposed here is entirely complementary to that approach,
since it could be used to reduce the cost of gathering those counts.
James Pitkow has written a paper comparing an earlier draft of this
hit-metering proposal with sampling approaches [9].
Phillip Hallam-Baker has proposed using a log-exchange protocol [5],
by which a server could request a proxy's logs by making an HTTP
request to the proxy. This proposal asserts that it is "believed to
operate correctly in configurations involving multiple proxies", but
it is not clear that this is true if an outer proxy is used as a
(one-way) firewall. The proposal also leaves a number of open
issues, such as how an origin server can be sure that all of the
proxies in the request subtree actually support log-exchange. It is
also not clear how this proposal couples a proxy's support of log-
exchange to a server's permission to cache a response.
For general background on the topic of Web measurement standards, see
the discussion by Thomas P. Novak and Donna L. Hoffman [8]. Also see
the "Privacy and Demographics Overview" page maintained by by the
World Wide Web Consortium [10], which includes a pointer to some
tentative proposals for gathering consumer demographics (not just
counting references) [3].
10 Security Considerations
Which outbound clients should a server (proxy or origin) trust to
report hit counts? A malicious proxy could easily report a large
number of hits on some page, and thus perhaps cause a large payment
to a content provider from an advertiser. To help avoid this
possibility, a proxy may choose to only relay usage counts received
from its outbound proxies to its inbound servers when the proxies
have authenticated themselves using Proxy-Authorization and/or they
are on a list of approved proxies.
It is not possible to enforce usage limits if a proxy is willing to
cheat (i.e., it offers to limit usage but then ignores a server's
Meter directive).
Regarding privacy: it appears that the design in this document does
not reveal any more information about individual users than would
already be revealed by implementation of the existing HTTP/1.1
support for "Cache-control: max-age=0, proxy-revalidate" or "Cache-
control: s-maxage=0". It may, in fact, help to conceal certain
aspects of the organizational structure on the outbound side of a
proxy. In any case, the conflict between user requirements for
anonymity and origin server requirements for demographic information
cannot be resolved by purely technical means.
11 Acknowledgments
We gratefully acknowledge the constructive comments received from
Anselm Baird-Smith, Ted Hardie, Koen Holtman (who suggested the
technique described in section 8), Dave Kristol, Ari Luotonen,
Patrick R. McManus, Ingrid Melve, and James Pitkow.
12 References
1. Bradner, S., "Key words for use in RFCs to Indicate Requirement
Levels", BCP 14, RFC2119, March 1997.
2. Anwat Chankhunthod, Peter B. Danzig, Chuck Neerdaels, Michael
F. Schwartz, and Kurt J. Worrell. A Hierarchical Internet Object
Cache. Proc. 1996 USENIX Technical Conf., San Diego, January,
1996, pp. 153-163.
3. Daniel W. Connolly. Proposals for Gathering Consumer
Demographics.
http://www.w3.org/pub/WWW/Demographics/Proposals.html.
4. Fielding, R., Gettys, J., Mogul, J., Nielsen, H. and T.
Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1," RFC2068,
January, 1997.
5. Phillip M. Hallam-Baker. Notification for Proxy Caches. W3C
Working Draft WD-proxy-960221, World Wide Web Consortium,
February, 1996. http://www.w3.org/pub/WWW/TR/WD-proxy.html.
6. Holtman, K., and A. Mutz, "Transparent Content Negotiation in
HTTP", Work in Progress.
7. Mogul, J., "Forcing HTTP/1.1 proxies to revalidate responses",
Work in Progress.
8. Thomas P. Novak and Donna L. Hoffman. New Metrics for New Media:
Toward the Development of Web Measurement Standards. This is a
draft paper, currently available at http://
www2000.ogsm.vanderbilt.edu/novak/web.standards/webstand.html.
Cited by permission of the author; do not quote or cite without
permission.
9. James Pitkow. In search of reliable usage data on the WWW.
Proc. Sixth International World Wide Web Conference, Santa Clara,
CA, April, 1997.
10. Joseph Reagle, Rohit Khare, Dan Connolly, and Tim Berners-Lee.
Privacy and Demographics Overview.
http://www.w3.org/pub/WWW/Demographics/.
11. Linda Tauscher and Saul Greenberg. Revisitation Patterns in
World Wide Web Navigation. Research Report 96/587/07, Department
of Computer Science, University of Calgary, March, 1996.
http://www.cpsc.ucalgary.ca/projects/grouplab/
papers/96WebReuse/TechReport96.html.
12. Wessels, D., and K. Claffy "Internet Cache Protocol (ICP),
version 2", RFC2186, September 1997.
13 Authors' Addresses
Jeffrey C. Mogul
Western Research Laboratory
Digital Equipment Corporation
250 University Avenue
Palo Alto, California, 94305, U.S.A.
EMail: mogul@wrl.dec.com
Phone: 1 415 617 3304 (email preferred)
Paul J. Leach
Microsoft
1 Microsoft Way
Redmond, Washington, 98052, U.S.A.
EMail: paulle@microsoft.com
14 Full Copyright Statement
Copyright (C) The Internet Society (1997). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implmentation may be prepared, copied, published
andand distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.