SourceClass, DestClass, FlowClass,
SourceKind, DestKind, FlowKind:
These six attributes may be set by executing PushRuleto
actions. They allow the PME to save (in flow records)
information which has been built up during matching.
Since their values are only defined when matching is
complete (and the flow key is built) their values may
not be tested in rules.
4.5 Maintaining the Flow Table
The flow table may be thought of as a 1-origin array of flow records.
(A particular implementation may, of course, use whatever data
structure is most suitable). When the meter starts up there are no
known flows; all the flow records are in the 'inactive' state.
Each time a packet is seen for a flow which is not in the current
flow set a flow record is set up for it; the state of such a record
is 'current.' When selecting a record for the new flow the meter
searches the flow table for a 'inactive' record - there is no
particular significance in the ordering of records within the table.
Flow data may be collected by a 'meter reader' at any time. There is
no requirement for collections to be synchronized. The reader may
collect the data in any suitable manner, for example it could upload
a copy of the whole flow table using a file transfer protocol, or it
could read the records in the current flow set row by row using a
suitable data transfer protocol.
The meter keeps information about collections, in particular it
maintains a LastCollectTime variable which remembers the time the
last collection was made. A second variable, InactivityTime,
specifies the minimum time the meter will wait before considering
that a flow is idle.
The meter must recover records used for idle flows, if only to
prevent it running out of flow records. Recovered flow records are
returned to the 'inactive' state. A variety of recovery strategies
are possible, including the following:
One possible recovery strategy is to recover idle flow records as
soon as possible after their data has been collected. To implement
this the meter could run a background process which scans the flow
table looking for 'current' flows whose 'last packet' time is earlier
than the meter's LastCollectTime. This would be suitable for use
when one was interested in measuring flow lifetimes.
Another recovery strategy is to leave idle flows alone as long as
possible, which would be suitable if one was only interested in
measuring total traffic volumes. It could be implemented by having
the meter search for collected idle flows only when it ran out of
'inactive' flow records.
One further factor a meter should consider before recovering a flow
is the number of meter readers which have collected the flow's data.
If there are multiple meter readers operating, network Operations
personnel should be able to specify the minimum number of meters - or
perhaps a specific list of meters - which should collect a flow's
data before its memory can be recovered. This issue will be further
developed in the future.
4.6 Handling Increasing Traffic Levels
Under normal conditions the meter reader specifies which set of usage
records it wants to collect, and the meter provides them.
If memory usage rises above the high-water mark the meter should
switch to a STANDBY RULE SET so as to increase the granularity of
flow collection and decrease the rate at which new flows are created.
When the manager, usually as part of a regular poll, becomes aware
that the meter is using its standby rule set, it could decrease the
interval between collections. The meter should also increase its
efforts to recover flow memory so as to reduce the number of idle
flows in memory. When the situation returns to normal, the manager
may request the meter to switch back to its normal rule set.
5 Meter Readers
Usage data is accumulated by a meter (e.g. in a router) as memory
permits. It is collected at regular reporting intervals by meter
readers, as specified by a manager. The collected data is recorded
in a disk file called a FLOW DATA FILE, as a sequence of USAGE
RECORDS.
The following sections describe the contents of usage records and
flow data files. Note, however, that at this stage the details of
such records and files is not specified in the architecture.
Specifying a common format for them would be a worthwhile future
development.
5.1 Identifying Flows in Flow Records
Once a packet has been classified and is ready to be counted, an
appropriate flow data record must already exist in the flow table;
otherwise one must be created. The flow record has a flexible format
where unnecessary identification attributes may be omitted. The
determination of which attributes of the flow record to use, and of
what values to put in them, is specified by the current rule set.
Note that the combination of start time, rule set id and subscript
(row number in the flow table) provide a unique flow identifier,
regardless of the values of its other attributes.
The current rule set may specify additional information, e.g. a
computed attribute value such as FlowKind, which is to be placed in
the attribute section of the usage record. That is, if a particular
flow is matched by the rule set, then the corresponding flow record
should be marked not only with the qualifying identification
attributes, but also with the additional information. Using this
feature, several flows may each carry the same FlowKind value, so
that the resulting usage records can be used in post-processing or
between meter reader and meter as a criterion for collection.
5.2 Usage Records, Flow Data Files
The collected usage data will be stored in flow data files on the
meter reader, one file for each meter. As well as containing the
measured usage data, flow data files must contain information
uniquely identifiying the meter from which it was collected.
A USAGE RECORD contains the descriptions of and values for one or
more flows. Quantities are counted in terms of number of packets and
number of bytes per flow. Each usage record contains the entity
identifier of the meter (a network address), a time stamp and a list
of reported flows (FLOW DATA RECORDS). A meter reader will build up a
file of usage records by regularly collecting flow data from a meter,
using this data to build usage records and concatenating them to the
tail of a file. Such a file is called a FLOW DATA FILE.
A usage record contains the following information in some form:
+-------------------------------------------------------------------+
| RECORD IDENTIFIERS: |
| Meter Id (& digital signature if required) |
| Timestamp |
| Collection Rules ID |
+-------------------------------------------------------------------+
| FLOW IDENTIFIERS: | COUNTERS |
| Address List | Packet Count |
| Subscriber ID (Optional) | Byte Count |
| Attributes (Optional) | Flow Start/Stop Time |
+-------------------------------------------------------------------+
5.3 Meter to Meter Reader: Usage Record Transmission
The usage record contents are the raison d'etre of the system. The
accuracy, reliability, and security of transmission are the primary
concerns of the meter/meter reader exchange. Since errors may occur
on networks, and Internet packets may be dropped, some mechanism for
ensuring that the usage information is transmitted intact is needed.
Flow data is moved from meter to meter reader via a series of
protocol exchanges between them. This may be carried out in various
ways, moving individual attribute values, complete flows, or the
entire flow table (i.e. all the active flows). One possible method
of achieving this transfer is to use SNMP; the 'Traffic Flow
Measurement: Meter MIB' document [4] gives details. Note that this
is simply one example; the transfer of flow data from meter to meter
reader is not specified in this document.
The reliability of the data transfer method under light, normal, and
extreme network loads should be understood before selecting among
collection methods.
In normal operation the meter will be running a rule file which
provides the required degree of flow reporting granularity, and the
meter reader(s) will collect the flow data often enough to allow the
meter's garbage collection mechanism to maintain a stable level of
memory usage.
In the worst case traffic may increase to the point where the meter
is in danger of running completely out of flow memory. The meter
implementor must decide how to handle this, for example by switching
to a default (extremely coarse granularity) rule set, by sending a
trap to the manager, or by attempting to dump flow data to the meter
reader.
Users of the Traffic Flow Measurement system should analyse their
requirements carefully and assess for themselves whether it is more
important to attempt to collect flow data at normal granularity
(increasing the collection frequency as needed to keep up with
traffic volumes), or to accept flow data with a coarser granularity.
Similarly, it may be acceptable to lose flow data for a short time in
return for being sure that the meter keeps running properly, i.e. is
not overwhelmed by rising traffic levels.
6 Managers
A manager configures meters and controls meter readers. It does this
via the interactions described below.
6.1 Between Manager and Meter: Control Functions
- DOWNLOAD RULE SET: A meter may hold an array of rule sets. One of
these, the 'default' rule set, is built in to the meter and cannot
be changed; the others must be downloaded by the manager. A
manager may use any suitable protocol exchange to achieve this, for
example an FTP file transfer or a series of SNMP SETs, one for each
row of the rule set.
- SWITCH TO SPECIFIED RULE SET: Once the rule sets have been
downloaded, the manager must instruct the meter which rule set it
is to actually run (i.e. which is to be the current rule set), and
which is to be the standby rule set.
- SET HIGH WATER MARK: A percentage value interpreted by the meter
which tells the meter when to switch to its standby rule set, so as
to increase the granularity of the flows and conserve the meter's
flow memory. Once this has happened, the manager may also change
the polling frequency or the meter's control parameters (so as to
increase the rate at which the meter can recover memory from idle
flows).
If the high traffic levels persist, the meter's normal rule set may
have to be rewritten to permanently reduce the reporting
granularity.
- SET FLOW TERMINATION PARAMETERS: The meter should have the good
sense in situations where lack of resources may cause data loss to
purge flow records from its tables. Such records may include:
- Flows that have already been reported to at least one meter
reader, and show no activity since the last report,
- Oldest flows, or
- Flows with the smallest number of unreported packets.
- SET INACTIVITY TIMEOUT: This is a time in seconds since the last
packet was seen for a flow. Flow records may be reclaimed if they
have been idle for at least this amount of time, and have been
collected in accordance with the current collection criteria.
6.2 Between Manager and Meter Reader: Control Functions
Because there are a number of parameters that must be set for traffic
flow measurement to function properly, and viable settings may change
as a result of network traffic characteristics, it is desirable to
have dynamic network management as opposed to static meter
configurations. Many of these operations have to do with space
tradeoffs - if memory at the meter is exhausted, either the reporting
interval must be decreased or a coarser granularity of aggregation
must be used so that more data fits into less space.
Increasing the reporting interval effectively stores data in the
meter; usage data in transit is limited by the effective bandwidth of
the virtual link between the meter and the meter reader, and since
these limited network resources are usually also used to carry user
data (the purpose of the network), the level of traffic flow
measurement traffic should be kept to an affordable fraction of the
bandwidth. ("Affordable" is a policy decision made by the network
Operations personnel). At any rate, it must be understood that the
operations below do not represent the setting of independent
variables; on the contrary, each of the values set has a direct and
measurable effect on the behaviour of the other variables.
Network management operations follow:
- MANAGER and METER READER IDENTIFICATION: The manager should ensure
that meters report to the correct set of collection stations, and
take steps to prevent unauthorised access to usage information.
The collection stations so identified should be prepared to poll if
necessary and accept data from the appropriate meters. Alternate
collection stations may be identified in case both the primary
manager and the primary collection station are unavailable.
Similarly, alternate managers may be identified.
- REPORTING INTERVAL CONTROL: The usual reporting interval should be
selected to cope with normal traffic patterns. However, it may be
possible for a meter to exhaust its memory during traffic spikes
even with a correctly set reporting interval. Some mechanism must
be available for the meter to tell the manager that it is in danger
of exhausting its memory (by declaring a 'high water' condition),
and for the manager to arbitrate (by decreasing the polling
interval, letting nature take its course, or by telling the meter
to ask for help sooner next time).
- GRANULARITY CONTROL: Granularity control is a catch-all for all the
parameters that can be tuned and traded to optimise the system's
ability to reliably measure and store information on all the
traffic (or as close to all the traffic as an administration
requires). Granularity
- Controls flow-id granularities for each interface, and
- Determines the number of buckets into which user traffic will
be lumped together.
Since granularity is controlled by the meter's current rule set,
the manager can only change it by requesting the meter to switch to
a different rule set. The new rule set could be downloaded when
required, or it could have been downloaded as part of the meter's
initial configuration.
- FLOW LIFETIME CONTROL: Flow termination parameters include timeout
parameters for obsoleting inactive flows and removing them from
tables and maximum flow lifetimes. This is intertwined with
reporting interval and granularity, and must be set in accordance
with the other parameters.
6.3 Exception Conditions
Exception conditions must be handled, particularly occasions when the
meter runs out of buffer space. Since, to prevent counting any
packet twice, packets can only be counted in a single flow at any
given time, discarding records will result in the loss of
information. The mechanisms to deal with this are as follows:
- METER OUTAGES: In case of impending meter outages (controlled
crashes, etc.) the meter could send a trap to the manager. The
manager could then request one or more meter readers to pick up the
usage record from the meter.
Following an uncontrolled meter outage such as a power failure, the
meter could send a trap to the manager indicating that it has
restarted. The manager could then download the meter's correct
rule set and advise the meter reader(s) that the meter is running
again. Alternatively, the meter reader may discover from its
regular poll that a meter has failed and restarted. It could then
advise the manager of this, instead of relying on a trap from the
meter.
- METER READER OUTAGES: If the collection system is down or isolated,
the meter should try to inform the manager of its failure to
communicate with the collection system. Usage data is maintained
in the flows' rolling counters, and can be recovered when the meter
reader is restarted.
- MANAGER OUTAGES: If the manager fails for any reason, the meter
should continue measuring and the meter reader(s) should keep
gathering usage records.
- BUFFER PROBLEMS: The network manager may realise that there is a
'low memory' condition in the meter. This can usually be
attributed to the interaction between the following controls:
- The reporting interval is too infrequent,
- The reporting granularity is too fine, or
- The throughput/bandwidth of circuits carrying the usage
data is too low.
The manager may change any of these parameters in response to the
meter (or meter reader's) plea for help.
6.4 Standard Rule Sets
Although the rule table is a flexible tool, it can also become very
complex. It may be helpful to develop some rule sets for common
applications:
- PROTOCOL TYPE: The meter records packets by protocol type. This
will be the default rule table for Traffic Flow Meters.
- ADJACENT SYSTEMS: The meter records packets by the MAC address of
the Adjacent Systems (neighbouring originator or next-hop).
(Variants on this table are "report source" or "report sink" only.)
This strategy might be used by a regional or backbone network which
wants to know how much aggregate traffic flows to or from its
subscriber networks.
- END SYSTEMS: The meter records packets by the IP address pair
contained in the packet. (Variants on this table are "report
source" or "report sink" only.) This strategy might be used by an
End System network to get detailed host traffic matrix usage data.
- TRANSPORT TYPE: The meter records packets by transport address; for
IP packets this provides usage information for the various IP
services.
- HYBRID SYSTEMS: Combinations of the above, e.g. for one interface
report End Systems, for another interface report Adjacent Systems.
This strategy might be used by an enterprise network to learn
detail about local usage and use an aggregate count for the shared
regional network.
7 APPENDICES
7.1 Appendix A: Network Characterisation
Internet users have extraordinarily diverse requirements. Networks
differ in size, speed, throughput, and processing power, among other
factors. There is a range of traffic flow measurement capabilities
and requirements. For traffic flow measurement purposes, the
Internet may be viewed as a continuum which changes in character as
traffic passes through the following representative levels:
International |
Backbones/National ---------------
/ \
Regional/MidLevel ---------- ----------
/ \ \ / / \
Stub/Enterprise --- --- --- ---- ----
||| ||| ||| |||| ||||
End-Systems/Hosts xxx xxx xxx xxxx xxxx
Note that mesh architectures can also be built out of these
components, and that these are merely descriptive terms. The nature
of a single network may encompass any or all of the descriptions
below, although some networks can be clearly identified as a single
type.
BACKBONE networks are typically bulk carriers that connect other
networks. Individual hosts (with the exception of network management
devices and backbone service hosts) typically are not directly
connected to backbones.
REGIONAL networks are closely related to backbones, and differ only
in size, the number of networks connected via each port, and
geographical coverage. Regionals may have directly connected hosts,
acting as hybrid backbone/stub networks. A regional network is a
SUBSCRIBER to the backbone.
STUB/ENTERPRISE networks connect hosts and local area networks.
STUB/ENTERPRISE networks are SUBSCRIBERS to regional and backbone
networks.
END SYSTEMS, colloquially HOSTS, are SUBSCRIBERS to any of the above
networks.
Providing a uniform identification of the SUBSCRIBER in finer
granularity than that of end-system, (e.g. user/account), is beyond
the scope of the current architecture, although an optional attribute
in the traffic flow measurement record may carry system-specific
"accountable (billable) party" labels so that meters can implement
proprietary or non-standard schemes for the attribution of network
traffic to responsible parties.
7.2 Appendix B: Recommended Traffic Flow Measurement Capabilities
Initial recommended traffic flow measurement conventions are outlined
here according to the following Internet building blocks. It is
important to understand what complexity reporting introduces at each
network level. Whereas the hierarchy is described top-down in the
previous section, reporting requirements are more easily addressed
bottom-up.
End-Systems
Stub Networks
Enterprise Networks
Regional Networks
Backbone Networks
END-SYSTEMS are currently responsible for allocating network usage to
end-users, if this capability is desired. From the Internet Protocol
perspective, end-systems are the finest granularity that can be
identified without protocol modifications. Even if a meter violated
protocol boundaries and tracked higher-level protocols, not all
packets could be correctly allocated by user, and the definition of
user itself varies too widely from operating system to operating
system (e.g. how to trace network usage back to users from shared
processes).
STUB and ENTERPRISE networks will usually collect traffic data either
by end- system network address or network address pair if detailed
reporting is required in the local area network. If no local
reporting is required, they may record usage information in the exit
router to track external traffic only. (These are the only networks
which routinely use attributes to perform reporting at granularities
finer than end-system or intermediate-system network address.)
REGIONAL networks are intermediate networks. In some cases,
subscribers will be enterprise networks, in which case the
intermediate system network address is sufficient to identify the
regional's immediate subscriber. In other cases, individual hosts or
a disjoint group of hosts may constitute a subscriber. Then end-
system network address pairs need to be tracked for those
subscribers. When the source may be an aggregate entity (such as a
network, or adjacent router representing traffic from a world of
hosts beyond) and the destination is a singular entity (or vice
versa), the meter is said to be operating as a HYBRID system.
At the regional level, if the overhead is tolerable it may be
advantageous to report usage both by intermediate system network
address (e.g. adjacent router address) and by end-system network
address or end-system network address pair.
BACKBONE networks are the highest level networks operating at higher
link speeds and traffic levels. The high volume of traffic will in
most cases preclude detailed traffic flow measurement. Backbone
networks will usually account for traffic by adjacent routers'
network addresses.
7.3 Appendix C: List of Defined Flow Attributes
This Appendix provides a checklist of the attributes defined to date;
others will be added later as the Traffic Measurement Architecture is
further developed.
0 Null
1 Flow Subscript Integer Flow table info
2 Flow Status Integer
4 Source Interface Integer Source Address
5 Source Adjacent Type Integer
6 Source Adjacent Address String
7 Source Adjacent Mask String
8 Source Peer Type Integer
9 Source Peer Address String
10 Source Peer Mask String
11 Source Trans Type Integer
12 Source Trans Address String
13 Source Trans Mask String
14 Destination Interface Integer Destination Address
15 Destination Adjacent Type Integer
16 Destination Adjacent Address String
17 Destination AdjacentMask String
18 Destination PeerType Integer
19 Destination PeerAddress String
20 Destination PeerMask String
21 Destination TransType Integer
22 Destination TransAddress String
23 Destination TransMask String
24 Packet Scale Factor Integer 'Other' attributes
25 Byte Scale Factor Integer
26 Rule Set Number Integer
27 Forward Bytes Counter Source-to-Dest counters
28 Forward Packets Counter
29 Reverse Bytes Counter Dest-to-Source counters
30 Reverse Packets Counter
31 First Time TimeTicks Activity times
32 Last Active Time TimeTicks
33 Source Subscriber ID String Session attributes
34 Destination Subscriber ID String
35 Session ID String
36 Source Class Integer 'Computed' attributes
37 Destination Class Integer
38 Flow Class Integer
39 Source Kind Integer
40 Destination Kind Integer
41 Flow Kind Integer
51 V1 Integer Meter variables
52 V2 Integer
53 V3 Integer
54 V4 Integer
55 V5 Integer
7.4 Appendix D: List of Meter Control Variables
Current Rule Set Number Integer
Standby Rule Set Number Integer
High Water Mark Percentage
Flood Mark Percentage
Inactivity Timeout (seconds) Integer
Last Collect Time TimeTicks
8 Acknowledgments
This document was initially produced under the auspices of the IETF's
Internet Accounting Working Group with assistance from SNMP, RMON and
SAAG working groups. This version documents the implementation work
done by the Internet Accounting Working Group, and is intended to
provide a starting point for the Realtime Traffic Flow Measurement
Working Group. Particular thanks are due to Stephen Stibler (IBM
Research) for his patient and careful comments during the preparation
of this memo.
9 References
[1] Mills, C., Hirsch, G. and G. Ruth, "Internet Accounting
Background", RFC1272, Bolt Beranek and Newman Inc., Meridian
Technology Corporation, November 1991.
[2] International Standards Organisation (ISO), "Management
Framework," Part 4 of Information Processing Systems Open
Systems Interconnection Basic Reference Model, ISO 7498-4,
1994.
[3] IEEE 802.3/ISO 8802-3 Information Processing Systems -
Local Area Networks - Part 3: Carrier sense multiple access
with collision detection (CSMA/CD) access method and physical
layer specifications, 2nd edition, September 21, 1990.
[4] Brownlee, N., "Traffic Flow Measurement: Meter MIB",
RFC2064, The University of Auckland, January 1997.
10 Security Considerations
Security issues are not discussed in detail in this document. The
meter's management and collection protocols are responsible for
providing sufficient data integrity and confidentiality.
11 Authors' Addresses
Nevil Brownlee
Information Technology Systems & Services
The University of Auckland
Phone: +64 9 373 7599 x8941
EMail: n.brownlee @auckland.ac.nz
Cyndi Mills
BBN Systems and Technologies
Phone: +1 617 873 4143
EMail: cmills@bbn.com
Greg Ruth
GTE Laboratories, Inc
Phone: +1 617 466 2448
EMail: gruth@gte.com