protocol. When a connection is required by an application, it is
usually provided by TCP [RFC793], the Transmission Control Protocol,
running atop IP on an end-to-end basis.
Connection-oriented subnetworks can be (and are widely) used to carry
IP, but often with considerable complexity. Subnetworks consisting
of few nodes can simply open a permanent connection between each pair
of nodes. This is frequently done with ATM. However, the number of
connections increases as the square of the number of nodes, so this
is clearly impractical for large subnetworks. A "shim" layer between
IP and the subnetwork is therefore required to manage connections.
This is one of the most common functions of a Subnetwork Dependent
Convergence Function (SNDCF) sublayer between IP and a subnetwork.
SNDCFs typically open subnetwork connections as needed when an IP
packet is queued for transmission and close them after an idle
timeout. There is no relation between subnetwork connections and any
connections that may exist at higher layers (e.g., TCP).
Because Internet traffic is typically bursty and transaction-
oriented, it is often difficult to pick an optimal idle timeout. If
the timeout is too short, subnetwork connections are opened and
closed rapidly, possibly over-stressing the subnetwork connection
management system (especially if it was designed for voice traffic
call holding times). If the timeout is too long, subnetwork
connections are idle much of the time, wasting any resources
dedicated to them by the subnetwork.
Purely connectionless subnets (such as Ethernet), which have no state
and dynamically share resources, are optimal for supporting best-
effort IP, which is stateless and dynamically shares resources.
Connection-oriented packet networks (such as ATM and Frame Relay),
which have state and dynamically share resources, are less optimal,
since best-effort IP does not benefit from the overhead of creating
and maintaining state. Connection-oriented circuit-switched networks
(including the PSTN and ISDN) have state and statically allocate
resources for a call, and thus require state creation and maintenance
overhead, but do not benefit from the efficiencies of statistical
multiplexing sharing of capacity inherent in IP.
In any event, if an SNDCF that opens and closes subnet connections is
used to support IP, care should be taken to make sure that connection
processing in the subnet can keep up with relatively short holding
times.
5. Broadcasting and Discovery
Subnetworks fall into two categories: point-to-point and shared. A
point-to-point subnet has exactly two endpoint components (hosts or
routers); a shared link has more than two endpoint components, using
either an inherently broadcast medium (e.g., Ethernet, radio) or a
switching layer hidden from the network layer (e.g., switched
Ethernet, Myrinet [MYR95], ATM). Switched subnetworks handle
broadcast by copying broadcast packets, providing each interface that
supports one, or more, systems (hosts or routers) with a copy of each
packet.
Several Internet protocols for IPv4 make use of broadcast
capabilities, including link-layer address lookup (ARP), auto-
configuration (RARP, BOOTP, DHCP), and routing (RIP).
A lack of broadcast capability can impede the performance of these
protocols, or render them inoperable (e.g., DHCP). ARP-like link
address lookup can be provided by a centralized database, but at the
expense of potentially higher response latency and the need for nodes
to have explicit knowledge of the ARP server address. Shared links
should support native, link-layer subnet broadcast.
A corresponding set of IPv6 protocols uses multicasting (see next
section) instead of broadcasting to provide similar functions with
improved scaling in large networks.
6. Multicasting
The Internet model includes "multicasting", where IP packets are sent
to all the members of a multicast group [RFC1112] [RFC3376]
[RFC2710]. Multicast is an option in IPv4, but a standard feature of
IPv6. IPv4 multicast is currently used by multimedia,
teleconferencing, gaming, and file distribution (web, peer-to-peer
sharing) applications, as well as by some key network and host
protocols (e.g., RIPv2, OSPF, NTP). IPv6 additionally relies on
multicast for network configuration (DHCP-like autoconfiguration) and
link-layer address discovery [RFC2461] (replacing ARP). In the case
of IPv6, this can allow autoconfiguration and address discovery to
span across routers, whereas the IPv4 broadcast-based services cannot
without ad-hoc router support [RFC1812].
Multicast-enabled IP routers organize each multicast group into a
spanning tree, and route multicast packets by making copies of each
multicast packet and forwarding the copies to each output interface
that includes at least one downstream member of the multicast group.
Multicasting is considerably more efficient when a subnetwork
explicitly supports it. For example, a router relaying a multicast
packet onto an Ethernet segment need send only one copy of the
packet, no matter how many members of the multicast group are
connected to the segment. Without native multicast support, routers
and switches on shared links would need to use broadcast with
software filters, such that every multicast packet sent incurs
software overhead for every node on the subnetwork, even if a node is
not a member of the multicast group. Alternately, the router would
transmit a separate copy to every member of the multicast group on
the segment, as is done on multicast-incapable switched subnets.
Subnetworks using shared channels (e.g., radio LANs, Ethernets) are
especially suitable for native multicasting, and their designers
should make every effort to support it. This involves designating a
section of the subnetwork’s own address space for multicasting. On
these networks, multicast is basically broadcast on the medium, with
Layer-2 receiver filters.
Subnet interfaces also need to be designed to accept packets
addressed to some number of multicast addresses, in addition to the
unicast packets specifically addressed to them. The number of
multicast addresses that needs to be supported by a host depends on
the requirements of the associated host; at least several dozen will
meet most current needs.
On low-speed networks, the multicast address recognition function may
be readily implemented in host software, but on high-speed networks,
it should be implemented in subnetwork hardware. This hardware need
not be complete; for example, many Ethernet interfaces implement a
"hashing" function where the IP layer receives all of the multicast
(and unicast) traffic to which the associated host subscribes, plus
some small fraction of multicast traffic to which the host does not
subscribe. Host/router software then has to discard the unwanted
packets that pass the Layer-2 multicast address filter [RFC1112].
There does not need to be a one-to-one mapping between a Layer-2
multicast address and an IP multicast address. An address overlap
may significantly degrade the filtering capability of a receiver’s
hardware multicast address filter. A subnetwork supporting only
broadcast should use this service for multicast and must rely on
software filtering.
Switched subnetworks must also provide a mechanism for copying
multicast packets to ensure the packets reach at least all members of
a multicast group. One option is to "flood" multicast packets in the
same manner as broadcast. This can lead to unnecessary transmissions
on some subnetwork links (notably non-multicast-aware Ethernet
switches). Some subnetworks therefore allow multicast filter tables
to control which links receive packets belonging to a specific group.
To configure this automatically requires access to Layer-3 group
membership information (e.g., IGMP [RFC3376], or MLD [RFC2710]).
Various implementation options currently exist to provide a subnet
node with a list of mappings of multicast addresses to
ports/interfaces. These employ a range of approaches, including
signaling from end hosts (e.g., IEEE 802 GARP/GMRP [802.1p]),
signaling from switches (e.g., CGMP [CGMP] and RGMP [RFC3488]),
interception and proxy of IP group membership packets (e.g., IGMP/MLD
Proxy [MAGMA-PROXY]), and enabling Layer-2 devices to
snoop/inspect/peek into forwarded Layer-3 protocol headers (e.g.,
IGMP, MLD, PIM) so that they may infer Layer-3 multicast group
membership [MAGMA-SNOOP]. These approaches differ in their
complexity, flexibility, and ability to support new protocols.
7. Bandwidth on Demand (BoD) Subnets
Some subnets allow a number of subnet nodes to share a channel
efficiently by assigning transmission opportunities dynamically.
Transmission opportunities are requested by a subnet node when it has
packets to send. The subnet schedules and grants transmission
opportunities sufficient to allow the transmitting subnet node to
send one or more packets (or packet fragments). We call these
subnets Bandwidth on Demand (BoD) subnets. Examples of BoD subnets
include Demand Assignment Multiple Access (DAMA) satellite and
terrestrial wireless networks, IEEE 802.11 point coordination
function (PCF) mode, and DOCSIS. A connection-oriented network (such
as the PSTN, ATM or Frame Relay) reserves resources on a much longer
timescale, and is therefore not a BoD subnet in our taxonomy.
The design parameters for BoD are similar to those in connection-
oriented subnetworks, although the implementations may vary
significantly. In BoD, the user typically requests access to the
shared channel for some duration. Access may be allocated for a
period of time at a specific rate, for a certain number of packets,
or until the user releases the channel. Access may be coordinated
through a central management entity or with a distributed algorithm
amongst the users. Examples of the resource that may be shared
include a terrestrial wireless hop, an upstream channel in a cable
television system, a satellite uplink, and an end-to-end satellite
channel.
Long-delay BoD subnets pose problems similar to connection-oriented
subnets in anticipating traffic. While connection-oriented subnets
hold idle channels open expecting new data to arrive, BoD subnets
request channel access based on buffer occupancy (or expected buffer
occupancy) on the sending port. Poor performance will likely result
if the sender does not anticipate additional traffic arriving at that
port during the time it takes to grant a transmission request. It is
recommended that the algorithm have the capability to extend a hold
on the channel for data that has arrived after the original request
was generated (this may be done by piggybacking new requests on user
data).
There is a wide variety of BoD protocols available. However, there
has been relatively little comprehensive research on the interactions
between BoD mechanisms and Internet protocol performance. Research
on some specific mechanisms is available (e.g., [AR02]). One item
that has been studied is TCP’s retransmission timer [KY02]. BoD
systems can cause spurious timeouts when adjusting from a relatively
high data rate, to a relatively low data rate. In this case, TCP’s
transmitted data takes longer to get through the network than
predicted by the TCP sender’s computed retransmission timeout.
Therefore, the TCP sender is prone to resending a segment
prematurely.
8. Reliability and Error Control
In the Internet architecture, the ultimate responsibility for error
recovery is at the end points [SRC81]. The Internet may occasionally
drop, corrupt, duplicate, or reorder packets, and the transport
protocol (e.g., TCP) or application (e.g., if UDP is used as the
transport protocol) must recover from these errors on an end-to-end
basis [RFC3155]. Error recovery in the subnetwork is therefore
justifiable only to the extent that it can enhance overall
performance. It is important to recognize that a subnetwork can go
too far in attempting to provide error recovery services in the
Internet environment. Subnet reliability should be "lightweight",
i.e., it only has to be "good enough", *not* perfect.
In this section, we discuss how to analyze characteristics of a
subnetwork to determine what is "good enough". The discussion below
focuses on TCP, which is the most widely-used transport protocol in
the Internet. It is widely believed (and is a stated goal within the
IETF) that non-TCP transport protocols should attempt to be "TCP-
friendly" and have many of the same performance characteristics.
Thus, the discussion below should be applicable, even to portions of
the Internet where TCP may not be the predominant protocol.
8.1. TCP vs Link-Layer Retransmission
Error recovery involves the generation and transmission of redundant
information computed from user data. Depending on how much redundant
information is sent and how it is generated, the receiver can use it
to reliably detect transmission errors, correct up to some maximum
number of transmission errors, or both. The general approach is
known as Error Control Coding, or ECC.
The use of ECC to detect transmission errors so that retransmissions
(hopefully without errors) can be requested is widely known as "ARQ"
(Automatic Repeat Request).
When enough ECC information is available to permit the receiver to
correct some transmission errors without a retransmission, the
approach is known as Forward Error Correction (FEC). Due to the
greater complexity of the required ECC and the need to tailor its
design to the characteristics of a specific modem and channel, FEC
has traditionally been implemented in special-purpose hardware
integral to a modem. This effectively makes it part of the physical
layer.
Unlike ARQ, FEC was rarely used for telecommunications outside of
space links prior to the 1990s. It is now nearly universal in
telephone, cable and DSL modems, digital satellite links, and digital
mobile telephones. FEC is also heavily used in optical and magnetic
storage where "retransmissions" are not possible.
Some systems use hybrid combinations of ARQ layered atop FEC; V.90
dialup modems (in the upstream direction) with V.42 error control are
one example. Most errors are corrected by the trellis (FEC) code
within the V.90 modem, and most remaining errors are detected and
corrected by the ARQ mechanisms in V.42.
Work is now underway to apply FEC above the physical layer, primarily
in connection with reliable multicasting [RFC3048] [RFC3450-RFC3453]
where conventional ARQ mechanisms are inefficient or difficult to
implement. However, in this discussion, we will assume that if FEC
is present, it is implemented within the physical layer.
Depending on the layer in which it is implemented, error control can
operate on an end-to-end basis or over a shorter span, such as a
single link. TCP is the most important example of an end-to-end
protocol that uses an ARQ strategy.
Many link-layer protocols use ARQ, usually some flavor of HDLC
[ISO3309]. Examples include the X.25 link layer, the AX.25 protocol
used in amateur packet radio, 802.11 wireless LANs, and the reliable
link layer specified in IEEE 802.2.
Only end-to-end error recovery can ensure reliable service to the
application (see Section 8). However, some subnetworks (e.g., many
wireless links) also have link-layer error recovery as a performance
enhancement [RFC3366]. For example, many cellular links have small
physical frame sizes (< 100 bytes) and relatively high frame loss
rates. Relying solely on end-to-end error recovery can clearly yield
a performance degradation, as retransmissions across the end-to-end
path take much longer to be received than when link layer
retransmissions are used. Thus, link-layer error recovery can often
increase end-to-end performance. As a result, link-layer and end-
to-end recovery often co-exist; this can lead to the possibility of
inefficient interactions between the two layers of ARQ protocols.
This inter-layer "competition" might lead to the following wasteful
situation. When the link layer retransmits (parts of) a packet, the
link latency momentarily increases. Since TCP bases its
retransmission timeout on prior measurements of total end-to-end
latency, including that of the link in question, this sudden increase
in latency may trigger an unnecessary retransmission by TCP of a
packet that the link layer is still retransmitting. Such spurious
end-to-end retransmissions generate unnecessary load and reduce end-
to-end throughput. As a result, the link layer may even have
multiple copies of the same packet in the same link queue at the same
time. In general, one could say the competing error recovery is
caused by an inner control loop (link-layer error recovery) reacting
to the same signal as an outer control loop (end-to-end error
recovery) without any coordination between the loops. Note that this
is solely an efficiency issue; TCP continues to provide reliable
end-to-end delivery over such links.
This raises the question of how persistent a link-layer sender should
be in performing retransmission [RFC3366]. We define the link-layer
(LL) ARQ persistency as the maximum time that a particular link will
spend trying to transfer a packet before it can be discarded. This
deliberately simplified definition says nothing about the maximum
number of retransmissions, retransmission strategies, queue sizes,
queuing disciplines, transmission delays, or the like. The reason we
use the term LL ARQ persistency, instead of a term such as "maximum
link-layer packet holding time," is that the definition closely
relates to link-layer error recovery. For example, on links that
implement straightforward error recovery strategies, LL ARQ
persistency will often correspond to a maximum number of
retransmissions permitted per link-layer frame.
For link layers that do not or cannot differentiate between flows
(e.g., due to network layer encryption), the LL ARQ persistency
should be small. This avoids any harmful effects or performance
degradation resulting from indiscriminate high persistence. A
detailed discussion of these issues is provided in [RFC3366].
However, when a link layer can identify individual flows and apply
ARQ selectively [LKJK02], then the link ARQ persistency should be
high for a flow using reliable unicast transport protocols (e.g.,
TCP) and must be low for all other flows. Setting the link ARQ
persistency larger than the largest link outage allows TCP to rapidly
restore transmission without needing to wait for a retransmission
time out. This generally improves TCP performance in the face of
transient outages. However, excessively high persistence may be
disadvantageous; a practical upper limit of 30-60 seconds may be
desirable. Implementation of such schemes remains a research issue.
(See also the following section "Recovery from Subnetwork Outages").
Many subnetwork designers have opportunities to reduce the
probability of packet loss, e.g., with FEC, ARQ, and interleaving, at
the cost of increased delay. TCP performance improves with
decreasing loss but worsens with increasing end-to-end delay, so it
is important to find the proper balance through analysis and
simulation.
8.2. Recovery from Subnetwork Outages
Some types of subnetworks, particularly mobile radio, are subject to
frequent temporary outages. For example, an active cellular data
user may drive or walk into an area (such as a tunnel) that is out of
range of any base station. No packets will be delivered successfully
until the user returns to an area with coverage.
The Internet protocols currently provide no standard way for a
subnetwork to explicitly notify an upper layer protocol (e.g., TCP)
that it is experiencing an outage rather than severe congestion.
Under these circumstances TCP will, after each unsuccessful
retransmission, wait even longer before trying again; this is its
"exponential back-off" algorithm. Furthermore, TCP will not discover
that the subnetwork outage has ended until its next retransmission
attempt. If TCP has backed off, this may take some time. This can
lead to extremely poor TCP performance over such subnetworks.
It is therefore highly desirable that a subnetwork subject to outages
does not silently discard packets during an outage. Ideally, the
subnetwork should define an interface to the next higher layer (i.e.,
IP) that allows it to refuse packets during an outage, and to
automatically ask IP for new packets when it is again able to deliver
them. If it cannot do this, then the subnetwork should hold onto at
least some of the packets it accepts during an outage and attempt to
deliver them when the outage ends. When packets are discarded, IP
should be notified so that the appropriate ICMP messages can be sent.
Note that it is *not* necessary to completely avoid dropping packets
during an outage. The purpose of holding onto a packet during an
outage, either in the subnetwork or at the IP layer, is so that its
eventual delivery will implicitly notify TCP that the subnetwork is
again operational. This is to enhance performance, not to ensure
reliability -- reliability, as discussed earlier, can only be ensured
on an end-to-end basis.
Only a few packets per TCP connection, including ACKs, need be held
in this way to cause the TCP sender to recover from the additional
losses once the flow resumes [RFC3366].
Because it would be a layering violation (and possibly a performance
hit) for IP or a subnetwork layer to look at TCP headers (which would
in any event be impossible if IPsec encryption [RFC2401] is in use),
it would be reasonable for the IP or subnetwork layers to choose, as
a design parameter, some small number of packets that will be
retained during an outage.
8.3. CRCs, Checksums and Error Detection
The TCP [RFC793], UDP [RFC768], ICMP, and IPv4 [RFC791] protocols all
use the same simple 16-bit 1’s complement checksum algorithm
[RFC1071] to detect corrupted packets. The IPv4 header checksum
protects only the IPv4 header, while the TCP, ICMP, and UDP checksums
provide end-to-end error detection for both the transport pseudo
header (including network and transport layer information) and the
transport payload data. Protection of the data is optional for
applications using UDP [RFC768] for IPv4, but is required for IPv6.
The Internet checksum is not very strong from a coding theory
standpoint, but it is easy to compute in software, and various
proposals to replace the Internet checksums with stronger checksums
have failed. However, it is known that undetected errors can and do
occur in packets received by end hosts [SP2000].
To reduce processing costs, IPv6 has no IP header checksum. The
destination host detects "important" errors in the IP header, such as
the delivery of the packet to the wrong destination. This is done by
including the IP source and destination addresses (pseudo header) in
the computation of the checksum in the TCP or UDP header, a practice
already performed in IPv4. Errors in other IPv6 header fields may go
undetected within the network; this was considered a reasonable price
to pay for a considerable reduction in the processing required by
each router, and it was assumed that subnetworks would use a strong
link CRC.
One way to provide additional protection for an IPv4 or IPv6 header
is by the authentication and packet integrity services of the IP
Security (IPsec) protocol [RFC2401]. However, this may not be a
choice available to the subnetwork designer.
Most subnetworks implement error detection just above the physical
layer. Packets corrupted in transmission are detected and discarded
before delivery to the IP layer. A 16-bit cyclic redundancy check
(CRC) is usually the minimum for error detection. This is
significantly more robust against most patterns of errors than the
16-bit Internet checksum. Note that the error detection properties
of a specific CRC code diminish with increasing frame size. The
Point-to-Point Protocol [RFC1662] requires support of a 16-bit CRC
for each link frame, with a 32-bit CRC as an option. (PPP is often
used in conjunction with a dialup modem, which provides its own error
control). Other subnetworks, including 802.3/Ethernet, AAL5/ATM,
FDDI, Token Ring, and PPP over SONET/SDH all use a 32-bit CRC. Many
subnetworks can also use other mechanisms to enhance the error
detection capability of the link CRC (e.g., FEC in dialup modems,
mobile radio and satellite channels).
Any new subnetwork designed to carry IP should therefore provide
error detection for each IP packet that is at least as strong as the
32-bit CRC specified in [ISO3309]. While this will achieve a very
low undetected packet error rate due to transmission errors, it will
not (and need not) achieve a very low packet loss rate as the
Internet protocols are better suited to dealing with lost packets