Request for Comments: 3955 SWITCH
Category: Informational October 2004
Evaluation of Candidate Protocols for
IP Flow Information Export (IPFIX)
Status of this Memo
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2004).
Abstract
This document contains an evaluation of the five candidate protocols
for an IP Flow Information Export (IPFIX) protocol, based on the
requirements document produced by the IPFIX Working Group. The
protocols are characterized and grouped in broad categories, and
evaluated against specific requirements. Finally, a recommendation
is made to select the NetFlow v9 protocol as the basis for the IPFIX
specification.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Protocol Summaries . . . . . . . . . . . . . . . . . . . . . . 2
2.1. CRANE. . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2. Diameter . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3. LFAP . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.4. NetFlow v9 . . . . . . . . . . . . . . . . . . . . . . . 5
2.5. Streaming IPDR . . . . . . . . . . . . . . . . . . . . . 6
3. Broad Classification of Candidate Protocols . . . . . . . . . 7
3.1. Design Goals . . . . . . . . . . . . . . . . . . . . . . 7
3.2. Data Representation. . . . . . . . . . . . . . . . . . . 8
3.3. Protocol Flow. . . . . . . . . . . . . . . . . . . . . . 9
4. Item-Level Compliance Evaluation . . . . . . . . . . . . . . . 10
4.1. Meter Reliability (5.1). . . . . . . . . . . . . . . . . 10
4.2. Sampling (5.2) . . . . . . . . . . . . . . . . . . . . . 11
4.3. Overload Behavior (5.3). . . . . . . . . . . . . . . . . 12
4.4. Timestamps (5.4) . . . . . . . . . . . . . . . . . . . . 12
4.5. Time Synchronization (5.5) . . . . . . . . . . . . . . . 12
4.6. Flow Expiration (5.6). . . . . . . . . . . . . . . . . . 13
4.7. Ignore Port Copy (5.9) . . . . . . . . . . . . . . . . . 13
4.8. Information Model (6.1). . . . . . . . . . . . . . . . . 13
4.9. Data Model (6.2) . . . . . . . . . . . . . . . . . . . . 13
4.10. Data Transfer (6.3). . . . . . . . . . . . . . . . . . . 14
5. Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.1. Recommendation . . . . . . . . . . . . . . . . . . . . . 19
6. Security Considerations. . . . . . . . . . . . . . . . . . . . 19
7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 19
8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20
8.1. Normative References . . . . . . . . . . . . . . . . . . 20
8.2. Informative References . . . . . . . . . . . . . . . . . 20
Appendix. A Note on References to the Candidate Protocol
Documents. . . . . . . . . . . . . . . . . . . . . . . 22
Author’s Address. . . . . . . . . . . . . . . . . . . . . . . . . 22
Full Copyright Statement. . . . . . . . . . . . . . . . . . . . . 23
1. Introduction
The IP Flow Information Export (IPFIX) Working Group has been
chartered to select a protocol for the export of flow information
from traffic-observing devices (such as routers or dedicated probes).
To this end, an evaluation team was formed to evaluate submitted
protocols. Each protocol was represented by an advocate, who
submitted a specific evaluation document for the respective protocol
against the requirements document [1]. The specification of each
protocol was itself available as one or several Internet-Drafts,
sometimes referring normatively to documents from outside the IETF.
This document contains an evaluation of the submitted protocols with
respect to the requirements document, and on a more general level, to
the working group charter.
The following IPFIX candidate protocol submissions were evaluated:
o CRANE [7], [8]
o Diameter [9], [10]
o LFAP [11], [12], [13]
o NetFlow v9 [2], [15], [16]
o Streaming IPDR [17], [18]
This document uses terminology defined in [1] intermixed with that
from submissions to explain the mapping between the two.
2. Protocol Summaries
In the following, each candidate protocol is described briefly,
highlighting its specific distinguishing features.
2.1. CRANE
XACCT’s Common Reliable Accounting for Network Element Protocol
Version 1.0 [7][8] is described as a protocol for the transmission of
accounting information from "Network Elements" to "mediation" and
"business support systems".
2.1.1. CRANE Protocol Operation
The exporting side is the CRANE client, the collecting side is the
CRANE server. Note that it is the server that is responsible for
initiating the connection to the client. A client can have multiple
simultaneous connections to different servers for robustness. Each
server has an associated priority. A client only exports to the
server with the highest priority that is perceived operational.
Clients and servers exchange messages over a reliable protocol such
as TCP [3] or (preferably) the Stream Control Transmission Protocol
(SCTP) [5]. The protocol uses application-layer acknowledgements as
an indication of successful processing by the server. Strong
authentication or data confidentiality aren’t supported by the
protocol, but can be supported by lower-layer mechanisms such as
IPsec [20] or TLS [21].
The protocol is bidirectional over the entire duration of a session.
There are 20 different message types. The protocol supports template
negotiation, not only at startup but also later on in a session, as
well as general status inquiries. There is a separate version
negotiation protocol defined over UDP.
2.1.2. CRANE Data Encoding
Data encoding is based on templates. Templates contain "keys"
representing items in data records. Clients (exporters) publish
templates to servers (collectors). Servers can then select the
subset of fields in a template that they are interested in. The
client will suppress keys that haven’t been selected by the server.
Data records contain references to template and configuration
instances. They also carry sequence numbers (DSNs for Data Sequence
Numbers). These sequence numbers can be used to de-duplicate data
records that have been delivered multiple times during
failover/fail-back in redundant configurations. A "duplicate" bit is
set in these situations as a hint for the de-duplication process.
The encoding of (flow information) data records themselves is very
compact. The client (exporter) can choose to send data in big-endian
(network byte order) or little-endian format. There are eighteen
fixed-size key types, as well as five variable-length string and
binary data (BLOB) types.
2.2. Diameter
Diameter [9][10] is an evolution of the Remote Authentication Dial In
User Service (RADIUS) protocol [22]. RADIUS is widely used to
outsource authentication and authorization in dialup access
environments. Diameter is a generalized and extensible protocol
intended to support Authentication, Authorization and Accounting
(AAA) requirements of different applications. Dialup and Mobile IPv4
are examples of such applications defined in the IETF.
2.2.1. Diameter Protocol Operation
Diameter is a peer-to-peer protocol. The base protocol defines
fourteen command codes, organized as seven request/response command
pairs. Presumably, only a subset of these would be used in a pure
IPFIX application. Diameter includes capability negotiation and
error notifications. Diameter operates over TCP or (preferred) SCTP.
There is a framework for end-to-end security, the mechanisms for
which are defined in a separate document. IPsec or TLS can be used
to provide authentication or encryption at the underlying layers.
2.2.2. Diameter Data Encoding
Diameter conveys data in the form of attribute/value pairs (AVPs).
An AVP consists of eight bytes of header plus the space to store the
data, which depends on the data format. There are numerous
predefined AVP data formats, including signed and unsigned integer
types, each in 32 and 64 bit variants, IPv4 and IPv6 addresses, as
well as others. The advocacy document [10] suggests that the
predefined data formats IPFilterRule and/or QoSFilterRule could be
extended to represent IP Flow Information. Such rules are
represented as readable UTF-8 strings. Alternatively, new AVPs could
be defined to represent flow information.
2.3. LFAP
LFAP [11][12][13] started out as the "Lightweight Flow Admission
Protocol" and was used to outsource shortcut creation decisions on
flow-based routers, as well as to provide per-flow statistics. Later
versions removed the admission function and changed the name to
"Lightweight Flow Accounting Protocol".
2.3.1. LFAP Protocol Operation
The exporter in LFAP is called the Connection Control Entity (CCE),
and the collector is the Flow Accounting Server (FAS). These
entities communicate with each other over a TCP connection. LFAP
knows thirteen message types, including operations for connection
management, version negotiation, flow information messages and
administrative requests. Authentication and encryption can be
provided by IPsec or TLS at lower layers. Additionally, the LFAP
protocol itself supports four levels of security using HMAC-MD5
authentication and DES-CBC encryption. Note that DES is now widely
regarded as not adequately secure, because its small key size makes
brute-force attacks viable.
A distinguishing feature is that LFAP has two different message types
for flow information: A Flow Accounting Request (FAR) message is sent
when a new flow is identified at the CCE (meter/exporter).
Accounting information is sent later in one or multiple Flow Update
Notification (FUN) messages. A collector must match each FUN to a
Flow ID previously sent in a FAR.
The LFAP document also defines a set of useful statistics about the
accounting process. A separate MIB document [14] is provided for
management of LFAP entities using SNMP.
2.3.2. LFAP Data Encoding
LFAP encodes data in a Type/Length/Value format with four bytes of
overhead per data item (two bytes for the type and two bytes for the
length field).
2.4. NetFlow v9
NetFlow v9 [2][15] is a generalized version of Cisco’s NetFlow
protocol. Previous versions of NetFlow, in particular version 5,
have been widely implemented and used for the exporting and
collecting of IP flow information.
2.4.1. NetFlow Protocol Operation
NetFlow uses a very simple protocol, with the exporter sending
template, options, and data "FlowSets" to the collector. FlowSets
are sequences of data records of similar format. NetFlow is the only
one of the candidate protocols that works over UDP [4]. Because of
the simple unidirectional nature of the protocol, it should be
relatively straightforward to add mappings to other transport
protocols such as SCTP or TCP.
The use of SCTP to transport NetFlow v9 has been suggested in [16].
The suggested mapping describes how control and data can be mapped to
different streams within a single SCTP connection, and suggests that
the Partial Reliability extension [23] be used on data streams. In
the proposed mapping, the exporter would initiate the connection.
2.4.2. NetFlow Data Encoding
NetFlow v9 uses a template facility to describe exported data. The
data itself is represented in a compact way using network byte order.
2.5. Streaming IPDR
Streaming IPDR [17][18] is an application of the Network Data
Management-Usage (NDM-U) for IP Services specification version 3.1
[19]. It has been developed by the Internet Protocol Detail Record
Organization (IPDR, Inc. or ipdr.org). The terminology used is
similar to CRANE’s, talking about Service Elements (SEs), mediation
systems and Business Support Systems (BSS).
2.5.1. Streaming IPDR Protocol Operation
Streaming IPDR operates over TCP. There is a "Trivial TCP Delivery"
mode as well as an "Acknowledged TCP Delivery" or "Reliable
Streaming" mode. The latter uses application-layer acknowledgements
for increased reliability.
The protocol is basically unidirectional. The exporter opens a
connection towards the collector, then sends a header followed by a
set of record descriptors. Then it can send "Usage Event" records
corresponding to these descriptors until the connection is
terminated. New record descriptors can be sent at any time.
Messages carry sequence numbers that are used for de-duplication
during failover. They are also referenced by application-level
acknowledgements when Reliable Streaming is used.
2.5.2. Streaming IPDR Data Encoding
IPDR uses an information modeling technique based on the XML-Schema
language [24]. Data can be represented in XML or in a streamlined
encoding based on the External Data Representation [25]. XDR forms
the basis of Sun’s Remote Procedure Call and Network File System
protocols, and has proven to be both space- and processing-efficient.
3. Broad Classification of Candidate Protocols
In order to evaluate the candidate protocols against the higher-level
requirements laid out in the IPFIX Working Group charter, it is
useful to group them into broader categories.
3.1. Design Goals
One way to look at the candidate protocols is to study the goals that
have directed their respective design. Note that the intention is
not to exclude protocols that have been designed with a different
class of applications in mind, but simply to better understand the
different tradeoffs that distinguish the protocols.
3.1.1. High-Performance Flow Metering (NetFlow, LFAP)
Of the candidate protocols, Cisco’s NetFlow is the purest example of
a highly specialized protocol that has been designed with the sole
objective of conveying accounting data from flow-aware routers at
high rates. Starting from a fixed set of accounting fields, it has
been extended a few times over the years to support additional fields
and various types of aggregation in the metering/exporting process.
Riverstone’s LFAP is similarly focused, except that it originated in
a protocol to outsource the decision whether to create shortcuts in
flow-based routers. This is still manifest in an increased emphasis
on reliable operation, and in the split reporting of flow information
using Flow Accounting Request (FAR) and Flow Update Notification
(FUN) messages.
It has been pointed out that split reporting as done by LFAP can
reduce memory requirements at the exporter. This concerns a subset
of attributes that are neither "key" attributes which define flows,
nor attributes such as packet or byte counters that must be updated
for each packet anyway. On the other hand, when there are many
short-lived flows, the number of flow export messages will be
significantly higher than with "unitary" flow export models, and the
collector will have to keep state about active flows until they are
terminated.
3.1.2. Carrier-Grade Multi-Purpose Accounting (IPDR, CRANE)
Streaming IPDR and CRANE describe themselves as protocols to
facilitate the reliable transfer of accounting information between
Network Elements (or more generally "Service Elements" in the case of
IPDR) and Mediation Systems or Business Support Systems (BSS). They
reflect a view of the accounting problem and of network system
architectures that originates in traditional "vertically integrated"
telecommunications.
Both protocols also emphasize extensibility with the goal of
applicability to a wide range of accounting tasks.
IPDR is based on NDM-U, which uses the XML-Schema language for
machine-readable specification of accounting data structures, while
using the efficient XDR encoding for the actual data transfer.
CRANE uses templates to describe exported data. These templates are
negotiated between collector and exporter and can change during a
session.
3.1.3. General-Purpose AAA (Diameter)
Diameter is another example of a broader-purpose protocol, in that it
covers aspects of authentication and authorization as well as
accounting. This explains its strong emphasis on security and
reliability. The design also takes into account various types of
intermediate agents.
3.2. Data Representation
IPFIX is intended to be deployed, among others, in high-speed routers
and to be used for exporting detailed flow data at high flow rates.
Therefore it is useful to look at the tradeoffs between the
efficiency of data representation and the extensibility of data
models. The two main efficiency goals should be (1) to minimize the
export data rate and (2) to minimize data encoding overhead in the
exporter. The overhead of decoding flow data at the collector is
deemed less critical, and is partly covered by efficiency target (2),
since an encoding that is easy on the encoder is often also easy on
the decoder.
3.2.1. Externally Described Encoding (CRANE, IPDR, NetFlow)
The protocols in this group use an external mechanism to fully
describe the format in which flow data is encoded. The mechanisms
are "templates" in the case of CRANE and NetFlow, and a subset of the
XML-Schema language, or alternatively XDR IDL, for IPDR.
A fully external data format description allows for very compact
encoding, with data components such as 32-bit integers taking up only
four octets. The XDR representation used in IPDR additionally
ensures that larger fields are always aligned on 32-bit boundaries,
which can reduce processing requirements at both the exporter and the
collector, at a slight cost of space (thus bandwidth) due to padding.
Most protocols specify "network byte order" or "big-endian" format in
the export data format. CRANE is the only protocol where the
exporter may choose the byte ordering. The principal benefit is that
this lowers the processing demand on exporters based on little-endian
architectures.
3.2.2. Partly Self-describing Encoding (Diameter, LFAP)
Diameter and LFAP represent flow data using Type/Length/Value
encodings. While this makes it possible to partly decode flow data
without full context information - possibly useful for debugging - it
does increase the encoding size and thus the bandwidth requirements
both on the wire and in the exporter and collector.
LFAP has a "multi-record" encoding which claims to provide similar
wire efficiency as the externally described encodings while still
supporting diagnostic tools.
3.3. Protocol Flow
Another criterion for classification is the flow of protocol messages
between exporter and collector.
3.3.1. Mainly Unidirectional Protocols (IPDR, NetFlow)
In IPDR and NetFlow, the data flow is essentially from exporter to
collector, with the collector only sending acknowledgements. The
protocols send data descriptions (templates) on session
establishment, and then start sending flow export data based on these
templates. "Meta-information" about the operational status of the
metering and exporting processes (for example about the sampling
parameters in force at a given moment) is conveyed using a special
type of "Option" template in NetFlow v9. IPDR currently doesn’t have
definitions for such "meta-data" types, but they could easily be
defined outside the protocol proper.
3.3.2. Bidirectional Protocols (CRANE, LFAP)
CRANE allows for negotiation of the templates used for data export at
the start of a session, and also allows negotiated template updates
later on. CRANE sessions include an exporter and potentially several
collectors, so these negotiations can involve more than two parties.
LFAP has an initial phase of version negotiation, followed by a phase
of "data negotiation". After these startup phases, the exporter
sends FAR and FUN messages to the collector. However, either party
may also send Administrative Request (AR) messages to the other, and
will normally receive Administrative Request Answers (ARA) in
response. Administrative Requests can be used for status inquiries,
including information about a specific active flow, or for
negotiation of the "Information Elements" that the collector wants
the exporter to export.
3.3.3. Unidirectional after Negotiation (Diameter)
Diameter has a general capabilities negotiation mechanism. The use
of Diameter for IPFIX hasn’t been described in sufficient detail to
determine how capabilities negotiation would be used. After
negotiation, the protocol would operate in essentially unidirectional
mode, with Accounting-Request (ACR) messages flowing from the
exporter to the collector, and Accounting-Answer (ACA) messages
flowing back.
4. Item-Level Compliance Evaluation
The template for protocol advocates noted that not all requirements
in [1] apply directly to the flow export protocol. In particular,
sections 4 (Distinguishing Flows) and 5 (Metering Process) mainly
specify requirements on the metering mechanism that "feeds" the
exporter. However, in some cases they require information about the
metering process to be reported to collectors, so the flow export
protocol must support conveying this information.
4.1. Meter Reliability (5.1)
CRANE, Diameter, IPDR consider requirement 5.1 (reliability of the
metering process or indication of "missing reliability") out of scope
for the IPFIX protocol, which presumably means that they assume the
metering process to be reliable.
The NetFlow v9 advocacy document takes a similar stance when it
claims "Total Compliance. The metering process is reliable."
(although this has been documented not to be true for all current
Cisco implementations of NetFlow v5).
LFAP is the only protocol that explicitly addresses the possibility
that data might be lost in the metering process, and provides useful
statistics for the collectors to estimate, not just the amount of
flow data that was lost, but also the amount of data that was not
unaccounted for.
Note that in the general case, it can be considered unrealistic to
assume total reliability of a flow-based metering process in all
situations, unless sampling or coarse flow definitions are used.
With the fine-grained flow classification mechanisms mandated by
IPFIX, it is easy to imagine traffic where each - possibly very small
- packet would create a new flow. This kind of traffic is in fact
encountered in practice during aggressive port scans, and will
eventually lead to table overflows or exceeding of memory bandwidth
at the meter.
While some of these situations can be handled by dropping data later
on in the exporter, data transfer, or collector, or by transitioning
the meter to sampling mode (or increasing the sampling interval), it
will sometimes be considered the lesser evil to simply report on the
data that couldn’t be accounted for. Currently LFAP is the only
protocol that supports this.
4.2. Sampling (5.2)
CRANE and IPDR don’t mention the possibility of sampling. This is
natural because they are targeted towards telco-grade accounting,
where sampling would be considered inadmissible. Since support for
sampling is a "MAY" requirement, its lack could be tolerated, but
severely restricts the applicability of these protocols in places of
high aggregation, where absolute precision is not necessary. This