#
from_ait:
v1 & 0 = DestPeerAddress: AssignAct, Next;
Null & 0 = 0: Gosub, classify;
Null & 0 = 0: NoMatch, 0; # 1 ait-ua
Null & 0 = 0: Ignore, 0; # 2 ait-ait
Null & 0 = 0: GotoAct, ok_pkt; # 3 ait-other
#
ok_pkt:
Null & 0 = 0: Count, 0;
The subroutine begins at the rule labelled classify (shown below).
It returns to the first, second or third rule after the invoking
Gosub rule, depending on whether the tested PeerAddress is in the UA,
AIT, or 'other' group of networks. In the listing below only one
network is tested in each of the groups - it is trivial to add more
rules (one per network) into either of the first two groups. In this
example the subroutine Pushes the network number from the packet into
the tested attribute before returning.
The first invocation of classify (above) begins at the rule labelled
ip_pkt. It Assigns SourcePeerAddress to V1 then executes a Gosub
action. Classify returns to one of the three following rules. They
will Goto from_ua or from_ait if the packet came from the UA or AIT
groups, otherwise the PME will retry the match. This means that
matched flows will have a UA or AIT network as their source, and
flows between other networks will be ignored.
The next two invocations of 'classify' test the packet's
DestPeerAddress. Packets from AIT to UA are Retried, forcing them to
be counted as AU to AIT flows. Packets from UA to UA are ignored, as
are packets from AIT to AIT.
classify:
v1 & 255.255.0.0 = 130.216.0.0: GotoAct, ua; # ua
v1 & 255.255.0.0 = 156.62.0.0: GotoAct, ait; # ait
Null & 0 = 0: Return, 3; # other
ua:
v1 & 255.255.0.0 = 0: PushPkttoAct, Next;
Null & 0 = 0: Return, 1;
ait:
v1 & 255.255.0.0 = 0: PushPkttoAct, Next;
Null & 0 = 0: Return, 2;
3.4 More complicated rule sets
The next example demonstrates a way of grouping IP flows together
depending on their Transport Address, i.e. their IP port number.
Simply Pushing every flow's SourceTransAddress and DestTransAddress
would produce a large number of flows, most of which differ only in
one of their transport addresses (the one which is not a well-known
port).
Instead we Push the well-known port number into each flow's
SourceTransAddress; its DestTransAddress will be zero by default.
SourcePeerType & 255 = dummy: Ignore, 0;
SourcePeerType & 255 = IP: Pushto, IP_pkt;
Null & 0 = 0: GotoAct, Next;
SourcePeerType & 255 = 0: PushPkttoAct, Next;
Null & 0 = 0: Count, 0; # Count others by protocol type
#
IP_pkt:
SourceTransType & 255 = tcp: Pushto, tcp_udp;
SourceTransType & 255 = udp: Pushto, tcp_udp;
SourceTransType & 255 = icmp: CountPkt, 0;
SourceTransType & 255 = ospf: CountPkt, 0;
Null & 0 = 0: GotoAct, c_unknown; # Unknown transport type
#
tcp_udp:
s_domain:
SourceTransAddress & 255.255 = domain: PushtoAct, c_well_known;
s_ftp:
SourceTransAddress & 255.255 = ftp: PushtoAct, c_well_known;
s_imap:
SourceTransAddress & 255.255 = 113: PushtoAct, c_well_known;
s_nfs
SourceTransAddress & 255.255 = 2049: PushtoAct, c_well_known;
s_pop:
SourceTransAddress & 255.255 = 110: PushtoAct, c_well_known;
s_smtp:
SourceTransAddress & 255.255 = smtp: PushtoAct, c_well_known;
s_telnet:
SourceTransAddress & 255.255 = telnet: PushtoAct, c_well_known;
s_www:
SourceTransAddress & 255.255 = www: PushtoAct, c_well_known;
s_xwin
SourceTransAddress & 255.255 = 6000: PushtoAct, c_well_known;
#
DestTransAddress & 255.255 = domain: GotoAct, s_domain;
DestTransAddress & 255.255 = ftp: GotoAct, s_ftp;
DestTransAddress & 255.255 = 113: GotoAct, s_imap;
DestTransAddress & 255.255 = 2049: GotoAct, s_nfs;
DestTransAddress & 255.255 = 110: GotoAct, s_pop;
DestTransAddress & 255.255 = smtp: GotoAct, s_smtp;
DestTransAddress & 255.255 = telnet: GotoAct, s_telnet;
DestTransAddress & 255.255 = www: GotoAct, s_www;
DestTransAddress & 255.255 = 6000: GotoAct, s_xwin;
#
Null & 0 = 0: GotoAct, c_unknown; # 'Unusual' port
#
c_unknown:
SourceTransType & 255 = 0: PushPkttoAct, Next;
DestTransType & 255 = 0: PushPkttoAct, Next;
SourceTransAddress & 255.255 = 0: PushPkttoAct, Next;
DestTransAddress & 255.255 = 0: CountPkt, 0;
#
c_well_known:
Null & 0 = 0: Count, 0
#
The first few rules ignore dummy packets, select IP packets for
further processing, and count packets for other protocols in a single
flow for each PeerType. TCP and UDP packets cause the PME to Push
their TransType and Goto tcp_udp. ICMP and OSPF packets are counted
in flows which have only their TransType Pushed.
At tcp_udp the packets' SourceTransAddress is tested to see whether
it is included in a set of 'interesting' port numbers. If it is, the
port number is pushed from the rule into the SourceTransAddress
attribute, and the packet is counted at c_well_known. (NeMaC accepts
Pushto as a synonym for PushRuleto).
This testing is repeated for the packet's DestTransAddress; if one of
these tests succeeds the PME Goes to the corresponding rule above and
Pushes the port number into the flow's SourceTransAddress. If these
tests fail the packet is counted at c_unknown, where all the flow's
Trans attributes are pushed. For production use more well-known
ports would need to be included in the tests above - c_unknown is
intended only for little-used exception flows!
Note that these rules only Push a value into a flow's
SourceTransAddress, and they don't contain any NoMatch actions. They
therefore don't specify a packet's direction, and they could be used
in other rule sets to group together flows for well-known ports.
The last example (below) meters flows from a remote router, and
demonstrates another approach to grouping well-known ports.
SourceAdjacentAddress & FF-FF-FF-FF-FF-FF =
00-60-3E-10-E0-A1: Goto, gateway; # tmkr2 router
DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-60-3E-10-E0-A1:
Goto, gateway; # Source is tmkr2
Null & 0 = 0: Ignore, 0;
#
gateway:
SourcePeerType & 255 = IP: GotoAct, IP_pkt;
Null & 0 = 0: GotoAct, Next;
SourcePeerType & 255 = 0: CountPkt, 0;
#
IP_pkt:
SourceTransType & 255 = tcp: PushRuleto, tcp_udp;
SourceTransType & 255 = udp: PushRuleto, tcp_udp;
Null & 0 = 0: GotoAct, not_wkp; # Unknown transport type
#
tcp_udp:
SourceTransAddress & FC-00 = 0: GotoAct, well_known_port;
DestTransAddress & FC-00 = 0: NoMatch, 0;
Null & 0 = 0: GotoAct, not_wkp;
#
not_wkp:
DestTransAddress & 255.255 = 0: PushPkttoAct, Next;
well_known_port:
SourcePeerType & 255 = 0: PushPkttoAct, Next;
DestPeerType & 255 = 0: PushPkttoAct, Next;
SourcePeerAddress & 255.255.255.0 = 0: PushPkttoAct, Next;
DestPeerAddress & 255.255.255.0 = 0: PushPkttoAct, Next;
SourceTransType & 255 = 0: PushPkttoAct, Next;
DestTransType & 255 = 0: PushPkttoAct, Next;
SourceTransAddress & 255.255 = 0: CountPkt, 0;
The first group of rules test incoming packet's AdjacentAddresses to
see whether they belong to a flow with an end point at the specified
router. Any which don't are ignored. Non-IP packets are counted in
flows which only have their PeerType Pushed; these will produce one
flow for each non-IP protocol. IP packets with TransTypes other than
UDP and TCP are counted at not_wkp, where all their address
attributes are pushed.
The high-order six bits of SourceTransAddress for UDP and TCP packets
are compared with zero. If this succeeds their source port number is
less than 1024, so they are from a well-known port. The port number
is pushed from the rule into the flow's SourceTransAddress attribute,
and the packet is counted at well_known_port. If the test fails, it
is repeated on the packet's DestTransAddress. If the destination is
a well-known port the match is Retried, and will succeed with the
well-known port as the flow's source.
If later analysis were to show that a high proportion of the observed
flows were from non-well-known ports, further pairs of rules could be
added to perform a test in each direction for other heavily-used
ports.
4 Flow data files
Although the Architecture document [1] specifies - in great detail -
how the Traffic Flow Meter works, and how a meter reader should
collect flow data from a meter, it does not say anything about how
the collected data should be stored. NeMaC uses a simple, self-
documenting file format, which has proved to be very effective in
use.
There are two kinds of records in a flow data file: flow records and
information records. Each flow record is simply a sequence of
attribute values with separators (these can be specified in a NeMaC
rule file) or spaces between them, terminated by a newline.
Information records all start with a cross-hatch. The file's first
record begins with ##, and identifies the file as being a file of
data from NeTraMet. It records NeMaC's parameters and the time this
collection was started. The file's second record begins with
#Format: and is a copy of the Format statement used by NeMaC to
collect the data.
The rest of the file is a sequence of collected data sets. Each of
these starts with a #Time: record, giving the time-of-day the
collection was started, the meter name, and the range of meter times
this collection represents. These from and to times are meter
UpTimes, i.e. they are times in hundredths of seconds since the meter
commenced operation. Most analysis applications have simply used the
collection start times (which are ASCII time-of-day values), but the
from and to times could be used to convert Uptime values to time-of-
day. The flow records which comprise a data set follow the #Time
record.
4.1 Sample flow data file
A sample flow data file appears below. Most of the flow records have
been deleted, but lines of dots show where they were.
##NeTraMet v3.2. -c300 -r rules.lan -e rules.default
test_meter -i eth0 4000 flows starting at 12:31:27 Wed 1 Feb 95
#Format: flowruleset flowindex firsttime sourcepeertype
sourcepeeraddress destpeeraddress topdus frompdus
tooctets fromoctets
#Time: 12:31:27 Wed 1 Feb 95 130.216.14.251 Flows
from 1 to 3642
1 2 13 5 31.32.0.0 33.34.0.0 1138 0 121824 0
1 3 13 2 11.12.0.0 13.14.0.0 4215 0 689711 0
1 4 13 7 41.42.0.0 43.34.0.0 1432 0 411712 0
1 5 13 6 21.22.0.0 23.24.0.0 8243 0 4338744 0
3 6 3560 2 130.216.14.0 130.216.3.0 0 10 0 1053
3 7 3560 2 130.216.14.0 130.216.76.0 59 65 4286 3796
3 8 3560 7 0.0.255.0 1.144.200.0 0 4 0 222
3 9 3560 2 130.216.14.0 130.216.14.0 118 1 32060 60
3 10 3560 6 130.216.0.28 130.216.0.192 782 1 344620 66
3 11 3560 7 0.0.255.0 0.128.113.0 0 1 0 73
3 12 3560 5 59.3.13.0 4.1.152.0 1 1 60 60
3 13 3560 7 0.128.94.0 0.129.27.0 2 2 120 158
3 14 3560 5 59.3.40.0 4.1.153.0 2 2 120 120
3 15 3560 5 0.0.0.0 4.1.153.0 0 1 0 60
3 16 3560 5 4.1.152.0 59.2.189.0 2 2 120 120
. . . . . . . . .
3 42 3560 7 0.128.42.0 0.129.34.0 0 1 0 60
3 43 3560 7 0.128.42.0 0.128.43.0 0 1 0 60
3 44 3560 7 0.128.42.0 0.128.41.0 0 1 0 60
3 45 3560 7 0.128.42.0 0.129.2.0 0 1 0 60
3 46 3560 5 4.1.152.0 59.2.208.0 2 2 120 120
3 47 3560 5 59.3.46.0 4.1.150.0 2 2 120 120
3 48 3560 5 4.1.152.0 59.2.198.0 2 2 120 120
3 49 3560 5 0.0.0.0 59.2.120.0 0 1 0 60
3 50 3664 5 4.1.152.0 59.2.214.0 0 1 0 60
3 51 3664 5 0.0.0.0 4.2.142.0 0 1 0 60
3 52 3664 5 4.1.153.0 59.3.45.0 4 4 240 240
#Time: 12:36:25 Wed 1 Feb 95 130.216.14.251 Flows
from 3641 to 33420
3 6 3560 2 130.216.14.0 130.216.3.0 0 21 0 2378
3 7 3560 2 130.216.14.0 130.216.76.0 9586 7148 1111118 565274
3 8 3560 7 0.0.255.0 1.144.200.0 0 26 0 1983
3 9 3560 2 130.216.14.0 130.216.14.0 10596 1 2792846 60
3 10 3560 6 130.216.0.28 130.216.0.192 16589 1 7878902 66
3 11 3560 7 0.0.255.0 0.128.113.0 0 87 0 16848
3 12 3560 5 59.3.13.0 4.1.152.0 20 20 1200 1200
3 13 3560 7 0.128.94.0 0.129.27.0 15 14 900 1144
3 14 3560 5 59.3.40.0 4.1.153.0 38 38 2280 2280
3 15 3560 5 0.0.0.0 4.1.153.0 0 30 0 1800
3 16 3560 5 4.1.152.0 59.2.189.0 20 20 1200 1200
3 17 3560 5 0.0.0.0 59.2.141.0 0 11 0 660
. . . . . . . . .
3 476 26162 7 0.129.113.0 0.128.37.0 0 1 0 82
3 477 27628 7 0.128.41.0 0.128.46.0 1 1 543 543
3 478 27732 7 0.128.211.0 0.128.46.0 1 1 543 543
3 479 31048 7 0.128.47.0 2.38.221.0 1 1 60 60
3 480 32717 2 202.14.100.0 130.216.76.0 0 4 0 240
3 481 32717 2 130.216.76.0 130.216.3.0 0 232 0 16240
#Time: 12:41:25 Wed 1 Feb 95 130.216.14.251 Flows
from 33419 to 63384
3 6 3560 2 130.216.14.0 130.216.3.0 51 180 3079 138195
3 7 3560 2 130.216.14.0 130.216.76.0 21842 18428 2467693 1356570
3 8 3560 7 0.0.255.0 1.144.200.0 0 30 0 2282
3 9 3560 2 130.216.14.0 130.216.14.0 24980 1 5051834 60
3 10 3560 6 130.216.0.28 130.216.0.192 20087 1 8800070 66
3 11 3560 7 0.0.255.0 0.128.113.0 0 164 0 32608
3 12 3560 5 59.3.13.0 4.1.152.0 41 41 2460 2460
3 14 3560 5 59.3.40.0 4.1.153.0 82 82 4920 4920
3 15 3560 5 0.0.0.0 4.1.153.0 0 60 0 3600
. . . . . . . . .
4.2 Flow data file features
Several features of NeMaC's flow data files (as indicated above) are
worthy of note:
- Collection times overlap slightly between samples. This allows for
flows which were created after the collection started, and makes
sure that flows are not missed from a collection.
- The rule set may change during a run. The above shows flows from
rule set 1 - the default set - in the first collection, followed by
the first flows created by rule set 3 (which has just been
downloaded by NeMaC).
- FlowIndexes may be reused by the meter once their flows have been
recovered by the garbage collector. The combination of
FlowRuleSet, FlowIndex and StartTime are needed to identify a flow
uniquely.
- Packet and Byte counters are 32-bit unsigned integers, and are
never reset by the meter. Computing the counts occurring within a
collection interval requires taking the difference between the
collected count and its value when the flow was last collected.
Note that counter wrap-around can be allowed for by simply
performing an unsigned subtraction and ignoring any carry.
- In the sample flow data file above I have used double spaces as
separators between the flow identifiers, peer addresses, pdu counts
and packet counts.
- The format of addresses in the flow data file depends on the type
of address. NeMaC always displays Adjacent addresses as six hex
bytes separated by hyphens, and Transport addresses as (16-bit)
integers. The format of a Peer address depends on its PeerType,
e.g. dotted decimal for IP. To facilitate this NeMaC needs to know
the PeerType for each flow; the user must request NeMaC to collect
it.
4.3 Terminating and restarting meter reading
When NeMaC first starts collecting from a meter, it reads the flow
data for all active flows. This provides a starting point for
analysis applications to compute the counts between successive
collections.
From time to time the user needs to terminate a flow data file and
begin a new one. For example, a user might need to generate a
separate file for each day of metering. NeMaC provides for this by
closing the file after each collection, then opening it and appending
the data from the next collection. To terminate a file the user
simply renames it. The Unix system will effect the name change
either immediately (if the file was closed) or as soon as the current
collection is complete (and the file is closed).
When NeMaC begins its next collection it observes that the file has
disappeared, so it creates a new one and writes the # header records
before writing the collected data.
There is one aspect of the above which requires some care on the
user's part. The last data set in a file is not duplicated as the
first data set of the next file. In other words, analysis
applications must either look ahead at the first data set of the next
file, or begin by reading the last data set of the previous file. If
they fail to do this they will loose one collection's worth of flow
data at each change of file.
5 Analysis applications
Most analysis applications will be unique, taking data produced by a
locally-developed rule set and producing reports to satisfy specific
local requirements. The NeTraMet distribution files include three
applications which are of general use, as follows:
- fd_filter computes data rates, i.e. the differences between
successive data sets in a flow data file. It also allows the user
to assign a 'tag' number to each flow; these are 'computed'
attributes similar to FlowClass and FlowKind - the only difference
is that they are computed from the collected data sets.
- fd_extract takes 'tagged' files from fd_filter and produces simple
'column list' files for use by other programs. One common use for
fd_extract is to produce time-series data files which can be plotted
by utilities like GNUPlot.
- nm_rc is a 'remote console' for a NeTraMet meter. It is a slightly
simplified version of NeMaC combined with fd_filter. It can be used
to monitor any meter, and will display (as lines of text
characters) information about the n busiest flows observed during
each collection interval.
- nifty is a traffic flow analyser, which (like nm_rc) displays data
from a NeTraMet meter. nifty is an X/Motif application, which
produces displays like 'Packet rate (pps) vs Flow lifetime
(minutes),' so as to highlight those flows which are 'interesting.'
These applications are useful in themselves, and they provide a good
starting point for users who wish to write their own analysis
applications.
6 Using NeTraMet in a measurement system
This section gives a brief summary of the steps involved in setting
up a traffic measurement system using NeTraMet. These are:
- Decide what is to be measured. One good way to approach this is to
specify exactly which flows are to be measured, and what reports
will be required. Specifying the flows should make it obvious
where meters will have to be placed so that the flows can be
observed, whether PCs will be adequate for the task, etc..
- Install meters. As well as actually placing the meter hosts this
includes making sure that they are configured correctly, with
appropriate IP addresses, SNMP community strings, etc.
- Develop the rule set (and a standby rule set). The degree of
difficulty here depends on how much is known in advance about the
traffic. One possible approach is to start with the meter default
rule set and measure how much traffic there is for each PeerType.
(This is a good way to verify that NeTraMet and NeMaC are working
properly). You can now add rules so as to increase the granularity
of the flows; this will of course increase the number of flows to
be collected, and force the meter's garbage collector to work
harder. Another approach is to try a rule set with very fine
granularity (i.e. one which Pushes all the address attributes),
then observing how many flows are collected every few minutes.
- Develop a strategy for controlling meter reader. This means
setting the meter's maximum number of flows, the collection
interval, how breaks between flow data files will be handled, how
often NeMaC should check that the meter is running, etc.
- Develop application(s) to process the collected flow data and
produce the required files and reports.
- Test run. Monitor the system, then refine the rule sets and meter
reading strategy until the overall system performance is
satisfactory.
This process can take quite a long time, but the overall result is
well worth the effort.
6.1 Examples of NeTraMet in production use
At the University of Auckland we run two sets of meters. One of
these measures the traffic entering and leaving our University
network, and generates usage reports for all our Internet users.
This has been in production since early 1994.
The other set consists of meters which are distributed at
Universities throughout New Zealand. They provide continuous traffic
flow measurements at five-minute intervals for all the links making
up the Universities' network (Kawaihiko); this system has been in
production since January 1996, and has already proved very useful in
planning the network's development.
The Kawaihiko Network provides IP connectivity for the New Zealand
Universities. They are linked via a Frame Relay cloud, using a
partial mesh of permanent virtual circuits. There is a NeTraMet
meter at each site, metering inward and outward traffic. All the
meters are managed from Auckland, and they all run copies of the same
rule set.
The rule set has about 650 rules, most of which are in a single
subroutine which classifies PeerAddresses into three categories -
'Kawaihiko network,' 'other New Zealand network' and 'non-New Zealand
network.' Inside New Zealand IP addresses lie within six CIDR
blocks, and there are about four hundred older networks which have
addresses outside those blocks. The rules are arranged in groups by
subnet size, i.e. all the /24 networks are tested first, then the /23
networks, etc, finishing with the /16 networks. This means that
although there are about 600 networks, any PeerAddress can be
classified with only nine tests.
The Kawaihiko rule set classifies flows, using computed attributes to
indicate the network 'kind' (Kawaihiko / New Zealand / international)
for each flow's SourcePeerAddress and DestPeerAddress, and to
indicate whether the flow is a 'network news' flow or not.
Flow data is collected from all of the meters every five minutes, and
is used to produce weekly reports, as follows:
- Traffic Plots. Plots of the 5-minute traffic rates for each site,
showing international traffic in and out, news traffic in and out,
and total traffic in and out of the site.
- Traffic Matrices. Two of these are produced, one for news traffic,
the other for total traffic. They show the traffic rates from
every site (including 'other New Zealand' and 'international') to
every other site. The mean, third quartile and maximum are printed
for every cell in the matrices.
7 Acknowledgments
This memo documents the implementation work on traffic flow
measurement here at the University of Auckland. Many of my
University colleagues have contributed significantly to this work,
especially Russell Fulton (who developed the rules sets, Perl scripts
and Cron jobs which produce our traffic usage reports automatically
week after week) and John White (for his patient help in documenting
the project).
8 References
[1] Brownlee, N., Mills, C., and G. Ruth, "Traffic Flow
Measurement: Architecture", RFC2063, The University of Auckland,
Bolt Beranek and Newman Inc., GTE Laboratories, Inc, January 1997.
[2] Brownlee, N., "Traffic Flow Measurement: Meter MIB",
RFC2064, The University of Auckland, January 1997.
[3] CRYNWR Packer Drivers distribution site:
http://www.crynwr.com/
[4] Case J., McCloghrie K., Rose M., and Waldbusser S.,
"Structure of Management Information for version 2 of the
Simple Network Managemenet Protocol", RFC1902, SNMP Research
Inc., Hughes LAN Systems, Dover Beach Consulting, Carnegie
Mellon University, April 1993.
[5] IBM Corporation, "IBM PC Technical Reference Manual," 1984.
[6] Waterloo TCP distribution site:
http://mvmpc9.ciw.uni-karlsruhe.de:80/d:/public/tcp_ip/wattcp
[7] CMU SNMP distribution site:
ftp://lancaster.andrew.cmu.edu/pub/snmp-dist
[8] libpcap distribution site:
ftp://ftp.ee.lbl.gov/libpcap-*.tar.gz
9 Security Considerations
Security issues are not discussed in detail in this document. The
meter's management and collection protocols are responsible for
providing sufficient data integrity and confidentiality.
10 Author's Address
Nevil Brownlee
The University of Auckland
Phone: +64 9 373 7599 x8941
Email: n.brownlee@auckland.ac.nz