a significantly large subset of the log). With log files running
into hundreds of MB even after compression (for a cache dealing
with millions of transactions per day) this is a non-trivial task.
See Also
IP packet/header sniffing - it may be that individual transactions
are at a level of granularity which simply isn't sensible to be
attempting on extremely busy caches. There may also be legal
implications in some countries, e.g., if this analysis identifies
individuals.
Indications
Disks/memory full(!) Stats (using multiple programs) take too long
to run. Stats crunching must be distributed out to multiple
machines because of its high computational cost.
Solution(s)
Have the proxy produce a standardized summary of its activity
either automatically or via an external (e.g., third party) tool,
in a commonly agreed format. The format could be something like
XML or the Extended Common Logfile, but the format and contents
are subjects for discussion. Ideally this approach would permit
individual cache server products to supply subsets of the possible
summary info, since it may not be feasible for all servers to
provide all of the information which people would like to see.
Workaround
Devise a private summary format for your own personal use - but
this complicates or even precludes the exchange of summary info
with other interested parties.
References
See the web pages for the commonly used cache stats analysis
programs, e.g., Calamaris, squidtimes, squidclients, etc.
Contact
Martin Hamilton <martin@wwwcache.ja.net>
Full Copyright Statement
Copyright (C) The Internet Society (2001). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement
Funding for the RFCEditor function is currently provided by the
Internet Society.