Request for Comments: 3918 Ixia
Category: Informational B. Hickman
Spirent Communications
October 2004
Methodology for IP Multicast Benchmarking
Status of this Memo
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2004).
Abstract
The purpose of this document is to describe methodology specific to
the benchmarking of multicast IP forwarding devices. It builds upon
the tenets set forth in RFC 2544, RFC 2432 and other IETF
Benchmarking Methodology Working Group (BMWG) efforts. This document
seeks to extend these efforts to the multicast paradigm.
The BMWG produces two major classes of documents: Benchmarking
Terminology documents and Benchmarking Methodology documents. The
Terminology documents present the benchmarks and other related terms.
The Methodology documents define the procedures required to collect
the benchmarks cited in the corresponding Terminology documents.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Key Words to Reflect Requirements. . . . . . . . . . . . . . . 3
3. Test Set Up. . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.1. Test Considerations. . . . . . . . . . . . . . . . . . . 4
3.1.1. IGMP Support. . . . . . . . . . . . . . . . . . . 5
3.1.2. Group Addresses . . . . . . . . . . . . . . . . . 5
3.1.3. Frame Sizes . . . . . . . . . . . . . . . . . . . 5
3.1.4. TTL . . . . . . . . . . . . . . . . . . . . . . . 6
3.1.5. Trial Duration. . . . . . . . . . . . . . . . . . 6
4. Forwarding and Throughput. . . . . . . . . . . . . . . . . . . 6
4.1. Mixed Class Throughput . . . . . . . . . . . . . . . . . 6
4.2. Scaled Group Forwarding Matrix . . . . . . . . . . . . . 8
4.3. Aggregated Multicast Throughput. . . . . . . . . . . . . 9
4.4. Encapsulation/Decapsulation (Tunneling) Throughput . . . 10
4.4.1. Encapsulation Throughput. . . . . . . . . . . . . 10
4.4.2. Decapsulation Throughput. . . . . . . . . . . . . 12
4.4.3. Re-encapsulation Throughput . . . . . . . . . . . 14
5. Forwarding Latency . . . . . . . . . . . . . . . . . . . . . . 15
5.1. Multicast Latency. . . . . . . . . . . . . . . . . . . . 16
5.2. Min/Max Multicast Latency. . . . . . . . . . . . . . . . 18
6. Overhead . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
6.1. Group Join Delay . . . . . . . . . . . . . . . . . . . . 20
6.2. Group Leave Delay. . . . . . . . . . . . . . . . . . . . 22
7. Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
7.1. Multicast Group Capacity . . . . . . . . . . . . . . . . 24
8. Interaction. . . . . . . . . . . . . . . . . . . . . . . . . . 25
8.1. Forwarding Burdened Multicast Latency. . . . . . . . . . 25
8.2. Forwarding Burdened Group Join Delay . . . . . . . . . . 27
9. Security Considerations. . . . . . . . . . . . . . . . . . . . 28
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 28
11. Contributions. . . . . . . . . . . . . . . . . . . . . . . . . 28
12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 28
12.1. Normative References . . . . . . . . . . . . . . . . . . 28
12.2. Informative References . . . . . . . . . . . . . . . . . 29
13. Authors’ Addresses . . . . . . . . . . . . . . . . . . . . . . 30
14. Full Copyright Statement . . . . . . . . . . . . . . . . . . . 31
1. Introduction
This document defines tests for measuring and reporting the
throughput, forwarding, latency and Internet Group Management
Protocol (IGMP) group membership characteristics of devices that
support IP multicast protocols. The results of these tests will
provide the user with meaningful data on multicast performance.
A previous document, "Terminology for IP Multicast Benchmarking"
[Du98], defined many of the terms that are used in this document.
The terminology document should be consulted before attempting to
make use of this document.
This methodology will focus on one source to many destinations,
although many of the tests described may be extended to use multiple
source to multiple destination topologies.
Subsequent documents may address IPv6 multicast and related multicast
routing protocol performance. Additional insight on IP and multicast
networking can be found in [Hu95], [Ka98] and [Mt98].
2. Key Words to Reflect Requirements
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119
[Br97]. RFC 2119 defines the use of these key words to help make the
intent of standards track documents as clear as possible. While this
document uses these keywords, this document is not a standards track
document.
3. Test set up
The set of methodologies presented in this document are for single
ingress, multiple egress multicast scenarios as exemplified by
Figures 1 and 2. Methodologies for multiple ingress and multiple
egress multicast scenarios are beyond the scope of this document.
Figure 1 shows a typical setup for an IP multicast test, with one
source to multiple destinations.
+------------+ +--------------+
| | | destination |
+--------+ | Egress(-)------->| test |
| source | | | | port(E1) |
| test |------>(|)Ingress | +--------------+
| port | | | +--------------+
+--------+ | Egress(-)------->| destination |
| | | test |
| | | port(E2) |
| DUT | +--------------+
| | . . .
| | +--------------+
| | | destination |
| Egress(-)------->| test |
| | | port(En) |
+------------+ +--------------+
Figure 1
If the multicast metrics are to be taken across multiple devices
forming a System Under Test (SUT), then test frames are offered to a
single ingress interface on a device of the SUT, subsequently
forwarded across the SUT topology, and finally forwarded to the test
apparatus’ frame-receiving components by the test egress interface(s)
of devices in the SUT. Figure 2 offers an example SUT test topology.
If a SUT is tested, the test topology and all relevant configuration
details MUST be disclosed with the corresponding test results.
*-----------------------------------------*
| |
+--------+ | +----------------+ | +--------+
| | | +------------+ |DUT B Egress E0(-)-|->| |
| | | |DUT A |--->| | | | |
| source | | | | | Egress E1(-)-|->| dest. |
| test |--|->(-)Ingress, I | +----------------+ | | test |
| port | | | | +----------------+ | | port |
| | | | |--->|DUT C Egress E2(-)-|->| |
| | | +------------+ | | | | |
| | | | Egress En(-)-|->| |
+--------+ | +----------------+ | +--------+
| |
*------------------SUT--------------------*
Figure 2
Generally, the destination test ports first join the desired number
of multicast groups by sending IGMP Group Report messages to the
DUT/SUT. To verify that all destination test ports successfully
joined the appropriate groups, the source test port MUST transmit IP
multicast frames destined for these groups. After test completion,
the destination test ports MAY send IGMP Leave Group messages to
clear the IGMP table of the DUT/SUT.
In addition, test equipment MUST validate the correct and proper
forwarding actions of the devices they test in order to ensure the
receipt of the frames that are involved in the test.
3.1. Test Considerations
The methodology assumes a uniform medium topology. Issues regarding
mixed transmission media, such as speed mismatch, headers
differences, etc., are not specifically addressed. Flow control, QoS
and other non-essential traffic or traffic-affecting mechanisms
affecting the variable under test MUST be disabled. Modifications to
the collection procedures might need to be made to accommodate the
transmission media actually tested. These accommodations MUST be
presented with the test results.
An actual flow of test traffic MAY be required to prime related
mechanisms, (e.g., process RPF events, build device caches, etc.) to
optimally forward subsequent traffic. Therefore, prior to running
any tests that require forwarding of multicast or unicast packets,
the test apparatus MUST generate test traffic utilizing the same
addressing characteristics to the DUT/SUT that will subsequently be
used to measure the DUT/SUT response. The test monitor should ensure
the correct forwarding of traffic by the DUT/SUT. The priming action
need only be repeated to keep the associated information current.
It is the intent of this memo to provide the methodology for basic
characterizations regarding the forwarding of multicast packets by a
device or simple system of devices. These characterizations may be
useful in illustrating the impact of device architectural features
(e.g., message passing versus shared memory; handling multicast
traffic as an exception by the general purpose processor versus the
by a primary data path, etc.) in the forwarding of multicast traffic.
It has been noted that the formation of the multicast distribution
tree may be a significant component of multicast performance. While
this component may be present in some of the measurements or
scenarios presented in this memo, this memo does not seek to
explicitly benchmark the formation of the multicast distribution
tree. The benchmarking of the multicast distribution tree formation
is left as future, more targeted work specific to a given tree
formation vehicle.
3.1.1. IGMP Support
All of the ingress and egress interfaces MUST support a version of
IGMP. The IGMP version on the ingress interface MUST be the same
version of IGMP that is being tested on the egress interfaces.
Each of the ingress and egress interfaces SHOULD be able to respond
to IGMP queries during the test.
Each of the ingress and egress interfaces SHOULD also send LEAVE
(running IGMP version 2 or later) [Ca02] [Fe97] after each test.
3.1.2. Group Addresses
There is no restriction to the use of multicast addresses [De89] to
compose the test traffic other than those assignments imposed by
IANA. The IANA assignments for multicast addresses [IANA1] MUST be
regarded for operational consistency. Address selection does not
need to be restricted to Administratively Scoped IP Multicast
addresses [Me98].
3.1.3. Frame Sizes
Each test SHOULD be run with different multicast frame sizes. For
Ethernet, the recommended sizes are 64, 128, 256, 512, 1024, 1280,
and 1518 byte frames.
Other link layer technologies MAY be used. The minimum and maximum
frame lengths of the link layer technology in use SHOULD be tested.
When testing with different frame sizes, the DUT/SUT configuration
MUST remain the same.
3.1.4. TTL
The data plane test traffic should have a TTL value large enough to
traverse the DUT/SUT.
The TTL in IGMP control plane messages MUST be in compliance with the
version of IGMP in use.
3.1.5. Trial Duration
The duration of the test portion of each trial SHOULD be at least 30
seconds. This parameter MUST be included as part of the results
reporting for each methodology.
4. Forwarding and Throughput
This section contains the description of the tests that are related
to the characterization of the frame forwarding of a DUT/SUT in a
multicast environment. Some metrics extend the concept of throughput
presented in RFC 1242. Forwarding Rate is cited in RFC 2285 [Ma98].
4.1. Mixed Class Throughput
Objective:
To determine the throughput of a DUT/SUT when both unicast class
frames and multicast class frames are offered simultaneously to a
fixed number of interfaces as defined in RFC 2432.
Procedure:
Multicast and unicast traffic are mixed together in the same
aggregated traffic stream in order to simulate a heterogeneous
networking environment.
The following events MUST occur before offering test traffic:
o All destination test ports configured to receive multicast
traffic MUST join all configured multicast groups;
o The DUT/SUT MUST learn the appropriate unicast and
multicast addresses; and
o Group membership and unicast address learning MUST be
verified through some externally observable method.
The intended load [Ma98] SHOULD be configured as alternating
multicast class frames and unicast class frames to a single ingress
interface. The unicast class frames MUST be configured to transmit
in an unweighted round-robin fashion to all of the destination ports.
For example, with six multicast groups and 3 destination ports with
one unicast addresses per port, the source test port will offer
frames in the following order:
m1 u1 m2 u2 m3 u3 m4 u1 m5 u2 m6 u3 m1 ...
Where:
m<Number> = Multicast Frame<Group>
u<Number> = Unicast Frame<Target Port>
Mixed class throughput measurement is defined in RFC 2432 [Du98]. A
search algorithm MUST be utilized to determine the Mixed Class
Throughput. The ratio of unicast to multicast frames MUST remain the
same when varying the intended load.
Reporting Format:
The following configuration parameters MUST be reflected in the test
report:
o Frame size(s)
o Number of tested egress interfaces on the DUT/SUT
o Test duration
o IGMP version
o Total number of multicast groups
o Traffic distribution for unicast and multicast traffic
classes
o The ratio of multicast to unicast class traffic
The following results MUST be reflected in the test report:
o Mixed Class Throughput as defined in RFC 2432 [Du98],
including: Throughput per unicast and multicast traffic
classes.
The Mixed Class Throughput results for each test SHOULD be reported
in the form of a table with a row for each of the tested frame sizes
per the recommendations in section 3.1.3. Each row SHOULD specify
the intended load, number of multicast frames offered, number of
unicast frames offered and measured throughput per class.
4.2. Scaled Group Forwarding Matrix
Objective:
To determine Forwarding Rate as a function of tested multicast groups
for a fixed number of tested DUT/SUT ports.
Procedure:
This is an iterative procedure. The destination test port(s) MUST
join an initial number of multicast groups on the first iteration.
All destination test ports configured to receive multicast traffic
MUST join all configured multicast groups. The recommended number of
groups to join on the first iteration is 10 groups. Multicast
traffic is subsequently transmitted to all groups joined on this
iteration and the forwarding rate is measured.
The number of multicast groups joined by each destination test port
is then incremented, or scaled, by an additional number of multicast
groups. The recommended granularity of additional groups to join per
iteration is 10, although the tester MAY choose a finer granularity.
Multicast traffic is subsequently transmitted to all groups joined
during this iteration and the forwarding rate is measured.
The total number of multicast groups joined MUST not exceed the
multicast group capacity of the DUT/SUT. The Group Capacity (Section
7.1) results MUST be known prior to running this test.
Reporting Format:
The following configuration parameters MUST be reflected in the test
report:
o Frame size(s)
o Number of tested egress interfaces on the DUT/SUT
o Test duration
o IGMP version
The following results MUST be reflected in the test report:
o The total number of multicast groups joined for that
iteration
o Forwarding rate determined for that iteration
The Scaled Group Forwarding results for each test SHOULD be reported
in the form of a table with a row representing each iteration of the
test. Each row or iteration SHOULD specify the total number of
groups joined for that iteration, offered load, total number of
frames transmitted, total number of frames received and the aggregate
forwarding rate determined for that iteration.
4.3. Aggregated Multicast Throughput
Objective:
To determine the maximum rate at which none of the offered frames to
be forwarded through N destination interfaces of the same multicast
groups are dropped.
Procedure:
Offer multicast traffic at an initial maximum offered load to a fixed
set of interfaces with a fixed number of groups at a fixed frame
length for a fixed duration of time. All destination test ports MUST
join all specified multicast groups.
If any frame loss is detected, the offered load is decreased and the
sender will transmit again. An iterative search algorithm MUST be
utilized to determine the maximum offered frame rate with a zero
frame loss.
Each iteration will involve varying the offered load of the multicast
traffic, while keeping the set of interfaces, number of multicast
groups, frame length and test duration fixed, until the maximum rate
at which none of the offered frames are dropped is determined.
Parameters to be measured MUST include the maximum offered load at
which no frame loss occurred. Other offered loads MAY be measured
for diagnostic purposes.
Reporting Format:
The following configuration parameters MUST be reflected in the test
report:
o Frame size(s)
o Number of tested egress interfaces on the DUT/SUT
o Test duration
o IGMP version
o Total number of multicast groups
The following results MUST be reflected in the test report:
o Aggregated Multicast Throughput as defined in RFC 2432
[Du98]
The Aggregated Multicast Throughput results SHOULD be reported in the
format of a table with a row for each of the tested frame sizes per
the recommendations in section 3.1.3. Each row or iteration SHOULD
specify offered load, total number of offered frames and the measured
Aggregated Multicast Throughput.
4.4. Encapsulation/Decapsulation (Tunneling) Throughput
This sub-section provides the description of tests related to the
determination of throughput measurements when a DUT/SUT or a set of
DUTs are acting as tunnel endpoints.
For this specific testing scenario, encapsulation or tunneling refers
to a packet that contains an unsupported protocol feature in a format
that is supported by the DUT/SUT.
4.4.1. Encapsulation Throughput
Objective:
To determine the maximum rate at which frames offered to one ingress
interface of a DUT/SUT are encapsulated and correctly forwarded on
one or more egress interfaces of the DUT/SUT without loss.
Procedure:
Source DUT/SUT Destination
Test Port Test Port(s)
+---------+ +-----------+ +---------+
| | | | | |
| | | Egress|--(Tunnel)-->| |