latency measurements.
Number of objects requested.
Number of objects transferred.
Test duration, expressed in seconds.
Test instruments MUST generate packets with unique timestamp
signatures.
5.10.3 Network-layer procedure
A client will offer a unidirectional stream of unicast packets to a
server. The packets MUST use a connectionless protocol like IP or
UDP/IP.
The test instrument MUST offer packets in a steady state. As noted
in the latency discussion in RFC2544 [2], latency measurements MUST
be taken at the throughput level, that is, at the highest offered
load with zero packet loss. Measurements taken at the throughput
level are the only ones that can legitimately be termed latency.
It is RECOMMENDED that implementers use offered loads not only at the
throughput level, but also at load levels that are less than or
greater than the throughput level. To avoid confusion with existing
terminology, measurements from such tests MUST be labeled as delay
rather than latency.
It is RECOMMENDED to perform the latency measurements with different
packet sizes. When testing with different packet sizes the DUT/SUT
configuration MUST remain the same.
If desired, a step test MAY be used in which offered loads increment
or decrement through a range of load levels.
The duration of the test portion of each trial MUST be at least 30
seconds.
5.10.4 Application layer procedure
An HTTP 1.1 or higher client will request one or more objects from an
HTTP 1.1 or higher server using one or more HTTP GET requests. If
the test instrument makes multiple HTTP GET requests, it MUST request
the same-sized object each time. Multiple iterations of this test
may be performed with objects of different sizes.
Implementers MAY configure the test instrument to run for a fixed
duration. In this case, the test instrument MUST report the number
of objects requested and returned for the duration of the test. For
fixed-duration tests it is RECOMMENDED that the duration be at least
30 seconds.
5.10.5 Measurements
Minimum delay:
The smallest delay incurred by data traversing the DUT/SUT at the
network layer or application layer, as appropriate.
Maximum delay:
The largest delay incurred by data traversing the DUT/SUT at the
network layer or application layer, as appropriate.
Average delay:
The mean of all measurements of delay incurred by data traversing
the DUT/SUT at the network layer or application layer, as
appropriate.
Delay distribution:
A set of histograms of all delay measurements observed for data
traversing the DUT/SUT at the network layer or application layer,
as appropriate.
5.10.6 Network-layer reporting format
The test report MUST note the packet size(s), offered load(s) and
test duration used. In addition, the test report MUST conform to the
reporting requirements set in section 4, Test Setup.
The latency results SHOULD be reported in the format of a table with
a row for each of the tested packet sizes. There SHOULD be columns
for the packet size, the intended rate, the offered rate, and the
resultant latency or delay values for each test.
5.10.7 Application-layer reporting format
The test report MUST note the object size(s) and number of requests
and responses completed. If applicable, the report MUST note the
test duration if a fixed duration was used. In addition, the test
report MUST conform to the reporting requirements set in section 4,
Test Setup.
The latency results SHOULD be reported in the format of a table with
a row for each of the object sizes. There SHOULD be columns for the
object size, the number of completed requests, the number of
completed responses, and the resultant latency or delay values for
each test.
Failure analysis:
The test report SHOULD indicate the number and percentage of HTTP
GET request or responses that failed to complete within the test
duration.
Version information:
The test report MUST note the version of HTTP client and server.
6. References
6.1 Normative References
[1] Newman, D., "Benchmarking Terminology for Firewall Devices", RFC
2647, August 1999.
[2] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC2544, March 1999.
[3] Bradner, S., "Benchmarking Terminology for Network
Interconnection Devices", RFC1242, July 1991.
[4] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC2285, February 1998.
[5] Mandeville, R. and J. Perser, "Benchmarking Methodology for LAN
Switching Devices", RFC2889, August 2000.
6.2 Informative References
[6] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L.,
Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol -
HTTP/1.1", RFC2616, June 1999.
[7] Clark, D., "IP Datagram Reassembly Algorithm", RFC815, July
1982.
[8] Postel, J., "Transmission Control Protocol", STD 7, RFC793,
September 1981.
7. Security Considerations
The primary goal of this document is to provide methodologies in
benchmarking firewall performance. While there is some overlap
between performance and security issues, assessment of firewall
security is outside the scope of this document.
APPENDIX A: HTTP (HyperText Transfer Protocol)
The most common versions of HTTP in use today are HTTP/1.0 and
HTTP/1.1 with the main difference being in regard to persistent
connections. HTTP 1.0, by default, does not support persistent
connections. A separate TCP connection is opened up for each GET
request the client wants to initiate and closed after the requested
object transfer is completed. While some implementations HTTP/1.0
supports persistence through the use of a keep-alive, there is no
official specification for how the keep-alive operates. In addition,
HTTP 1.0 proxies do support persistent connection as they do not
recognize the connection header.
HTTP/1.1, by default, does support persistent connection and is
therefore the version that is referenced in this methodology. Proxy
based DUT/SUTs may monitor the TCP connection and after a timeout,
close the connection if no activity is detected. The duration of
this timeout is not defined in the HTTP/1.1 specification and will
vary between DUT/SUTs. If the DUT/SUT closes inactive connections,
the aging timer on the DUT SHOULD be configured for a duration that
exceeds the test time.
While this document cannot foresee future changes to HTTP and it
impact on the methodologies defined herein, such changes should be
accommodated for so that newer versions of HTTP may be used in
benchmarking firewall performance.
APPENDIX B: Connection Establishment Time Measurements
Some connection oriented protocols, such as TCP, involve an odd
number of messages when establishing a connection. In the case of
proxy based DUT/SUTs, the DUT/SUT will terminate the connection,
setting up a separate connection to the server. Since, in such
cases, the test instrument does not own both sides of the connection,
measurements will be made two different ways. While the following
describes the measurements with reference to TCP, the methodology may
be used with other connection oriented protocols which involve an odd
number of messages.
When testing non-proxy based DUT/SUTs , the establishment time shall
be directly measured and is considered to be from the time the first
bit of the first SYN packet is transmitted by the client to the time
the last bit of the final ACK in the three-way handshake is received
by the target server.
If the DUT/SUT is proxy based, the connection establishment time is
considered to be from the time the first bit of the first SYN packet
is transmitted by the client to the time the client transmits the
first bit of the first acknowledged TCP datagram (t4-t0 in the
following timeline).
t0: Client sends a SYN.
t1: Proxy sends a SYN/ACK.
t2: Client sends the final ACK.
t3: Proxy establishes separate connection with server.
t4: Client sends TCP datagram to server.
*t5: Proxy sends ACK of the datagram to client.
* While t5 is not considered part of the TCP connection
establishment, acknowledgement of t4 must be received for the
connection to be considered successful.
APPENDIX C: Connection Tear Down Time Measurements
While TCP connections are full duplex, tearing down of such
connections are performed in a simplex fashion, that is, FIN segments
are sent by each host/device terminating each side of the TCP
connection.
When making connection tear down times measurements, such
measurements will be made from the perspective of the entity, that
is, virtual client/server initiating the connection tear down
request. In addition, the measurement will be performed in the same
manner, independent of whether or not the DUT/SUT is proxy-based. The
connection tear down will be considered the interval between the
transmission of the first bit of the first TCP FIN packet transmitted
by the virtual client or server, whichever is applicable, requesting
a connection tear down to receipt of the last bit of the
corresponding ACK packet on the same virtual client/server interface.
Authors' Addresses
Brooks Hickman
Spirent Communications
26750 Agoura Road
Calabasas, CA 91302
USA
Phone: + 1 818 676 2412
EMail: brooks.hickman@spirentcom.com
David Newman
Network Test Inc.
31324 Via Colinas, Suite 113
Westlake Village, CA 91362-6761
USA
Phone: + 1 818 889-0011
EMail: dnewman@networktest.com
Saldju Tadjudin
Spirent Communications
26750 Agoura Road
Calabasas, CA 91302
USA
Phone: + 1 818 676 2468
EMail: Saldju.Tadjudin@spirentcom.com
Terry Martin
GVNW Consulting Inc.
8050 SW Warm Springs Road
Tualatin Or. 97062
USA
Phone: + 1 503 612 4422
EMail: tmartin@gvnw.com
Full Copyright Statement
Copyright (C) The Internet Society (2003). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement
Funding for the RFCEditor function is currently provided by the
Internet Society.