than to dealing with corrupted packets [SRC81].
Packet corruption may be, and is, also caused by bugs in host and
router hardware and software. Even if every subnetwork implemented
strong error detection, it is still essential that end-to-end
checksums are used at the receiving end host [SP2000].
Designers of complex subnetworks consisting of internal links and
packet switches should consider implementing error detection on an
edge-to-edge basis to cover an entire SNDU (or IP packet). A CRC
would be generated at the entry point to the subnetwork and checked
at the exit endpoint. This may be used instead of, or in combination
with, error detection at the interface to each physical link. An
edge-to-edge check has the significant advantage of protecting
against errors introduced anywhere within the subnetwork, not just
within its transmission links. Examples of this approach include the
way in which the Ethernet CRC-32 is handled by LAN bridges [802.1D].
ATM AAL5 [ITU-I363] also uses an edge-to-edge CRC-32.
Some specific applications may be tolerant of residual errors in the
data they exchange, but removal of the link CRC may expose the
network to an undesirable increase in undetected errors in the IP and
transport headers. Applications may also require a high level of
error protection for control information exchanged by protocols
acting above the transport layer. One example is a voice codec,
which is robust against bit errors in the speech samples. For such
mechanisms to work, the receiving application must be able to
tolerate receiving corrupted data. This also requires that an
application uses a mechanism to signal that payload corruption is
permitted and to indicate the coverage (headers and data) required to
be protected by the subnetwork CRC. The UDP-Lite protocol [RFC3828]
is the first Internet standards track transport protocol supporting
partial payload protection. Receipt of corrupt data by arbitrary
application protocols carries a serious danger that a subnet delivers
data with errors that remain undetected by the application and hence
corrupt the communicated data [SRC81].
8.4. How TCP Works
One of TCP’s functions is end-host based congestion control for the
Internet. This is a critical part of the overall stability of the
Internet, so it is important that link-layer designers understand
TCP’s congestion control algorithms.
TCP assumes that, at the most abstract level, the network consists of
links and queues. Queues provide output-buffering on links that are
momentarily oversubscribed. They smooth instantaneous traffic bursts
to fit the link bandwidth. When demand exceeds link capacity long
enough to fill the queue, packets must be dropped. The traditional
action of dropping the most recent packet ("tail dropping") is no
longer recommended [RFC2309] [RFC2914], but it is still widely
practiced.
TCP uses sequence numbering and acknowledgments (ACKs) on an
end-to-end basis to provide reliable, sequenced delivery. TCP ACKs
are cumulative, i.e., each implicitly ACKs every segment received so
far. If a packet with an unexpected sequence number is received, the
ACK field in the packets returned by the receiver will cease to
advance. Using an optional enhancement, TCP can send selective
acknowledgments (SACKs) [RFC2018] to indicate which segments have
arrived at the receiver.
Since the most common cause of packet loss is congestion, TCP treats
packet loss as an indication of potential Internet congestion along
the path between TCP end hosts. This happens automatically, and the
subnetwork need not know anything about IP or TCP. A subnetwork node
simply drops packets whenever it must, though some packet-dropping
strategies (e.g., RED) are more fair to competing flows than others.
TCP recovers from packet losses in two different ways. The most
important mechanism is the retransmission timeout. If an ACK fails
to arrive after a certain period of time, TCP retransmits the oldest
unacked packet. Taking this as a hint that the network is congested,
TCP waits for the retransmission to be ACKed before it continues, and
it gradually increases the number of packets in flight as long as a
timeout does not occur again.
A retransmission timeout can impose a significant performance
penalty, as the sender is idle during the timeout interval and
restarts with a congestion window of one TCP segment following the
timeout. To allow faster recovery from the occasional lost packet in
a bulk transfer, an alternate scheme, known as "fast recovery", was
introduced [RFC2581] [RFC2582] [RFC2914] [TCPF98].
Fast recovery relies on the fact that when a single packet is lost in
a bulk transfer, the receiver continues to return ACKs to subsequent
data packets that do not actually acknowledge any newly-received
data. These are known as "duplicate acknowledgments" or "dupacks".
The sending TCP can use dupacks as a hint that a packet has been lost
and retransmit it without waiting for a timeout. Dupacks effectively
constitute a negative acknowledgment (NAK) for the packet sequence
number in the acknowledgment field. TCP waits until a certain number
of dupacks (currently 3) are seen prior to assuming a loss has
occurred; this helps avoid an unnecessary retransmission during
out-of-sequence delivery.
A technique called "Explicit Congestion Notification" (ECN) [RFC3168]
allows routers to directly signal congestion to hosts without
dropping packets. This is done by setting a bit in the IP header.
Since ECN support is likely to remain optional, the lack of an ECN
bit must *never* be interpreted as a lack of congestion. Thus, for
the foreseeable future, TCP must interpret a lost packet as a signal
of congestion.
The TCP "congestion avoidance" [RFC2581] algorithm maintains a
congestion window (cwnd) controlling the amount of data TCP may have
in flight at any moment. Reducing cwnd reduces the overall bandwidth
obtained by the connection; similarly, raising cwnd increases
performance, up to the limit of the available capacity.
TCP probes for available network capacity by initially setting cwnd
to one or two packets and then increasing cwnd by one packet for each
ACK returned from the receiver. This is TCP’s "slow start"
mechanism. When a packet loss is detected (or congestion is signaled
by other mechanisms), cwnd is reset to one and the slow start process
is repeated until cwnd reaches one half of its previous setting
before the reset. Cwnd continues to increase past this point, but at
a much slower rate than before. If no further losses occur, cwnd
will ultimately reach the window size advertised by the receiver.
This is an "Additive Increase, Multiplicative Decrease" (AIMD)
algorithm. The steep decrease of cwnd in response to congestion
provides for network stability; the AIMD algorithm also provides for
fairness between long running TCP connections sharing the same path.
8.5. TCP Performance Characteristics
Caveat
Here we present a current "state-of-the-art" understanding of TCP
performance. This analysis attempts to characterize the performance
of TCP connections over links of varying characteristics.
Link designers may wish to use the techniques in this section to
predict what performance TCP/IP may achieve over a new link-layer
design. Such analysis is encouraged. Because this is a relatively
new analysis, and the theory is based on single-stream TCP
connections under "ideal" conditions, it should be recognized that
the results of such analysis may differ from actual performance in
the Internet. That being said, we have done our best to provide the
designers with helpful information to get an accurate picture of the
capabilities and limitations of TCP under various conditions.
8.5.1. The Formulae
The performance of TCP’s AIMD Congestion Avoidance algorithm has been
extensively analyzed. The current best formula for the performance
of the specific algorithms used by Reno TCP (i.e., the TCP specified
in [RFC2581]) is given by Padhye, et al. [PFTK98]. This formula is:
MSS
BW = --------------------------------------------------------
RTT*sqrt(1.33*p) + RTO*p*[1+32*p^2]*min[1,3*sqrt(.75*p)]
where
BW is the maximum TCP throughout achievable by an
individual TCP flow
MSS is the TCP segment size being used by the connection
RTT is the end-to-end round trip time of the TCP connection
RTO is the packet timeout (based on RTT)
p is the packet loss rate for the path
(i.e., .01 if there is 1% packet loss)
Note that the speed of the links making up the Internet path does not
explicitly appear in this formula. Attempting to send faster than
the slowest link in the path causes the queue to grow at the
transmitter driving the bottleneck. This increases the RTT, which in
turn reduces the achievable throughput.
This is currently considered to be the best approximate formula for
Reno TCP performance. A further simplification of this formula is
generally made by assuming that RTO is approximately 5*RTT.
TCP is constantly being improved. A simpler formula, which gives an
upper bound on the performance of any AIMD algorithm which is likely
to be implemented in TCP in the future, was derived by Ott, et al.
[MSMO97].
MSS 1
BW = C --- -------
RTT sqrt(p)
where C is 0.93.
8.5.2. Assumptions
Both formulae assume that the TCP Receiver Window is not limiting the
performance of the connection. Because the receiver window is
entirely determined by end-hosts, we assume that hosts will maximize
the announced receiver window to maximize their network performance.
Both of these formulae allow BW to become infinite if there is no
loss. However, an Internet path will drop packets at bottlenecked
queues if the load is too high. Thus, a completely lossless TCP/IP
network can never occur (unless the network is being underutilized).
The RTT used is the arithmetic average, including queuing delays.
The formulae are for a single TCP connection. If a path carries many
TCP connections, each will follow the formulae above independently.
The formulae assume long-running TCP connections. For connections
that are extremely short (<10 packets) and don’t lose any packets,
performance is driven by the TCP slow-start algorithm. For
connections of medium length, where on average only a few segments
are lost, single connection performance will actually be slightly
better than given by the formulae above.
The difference between the simple and complex formulae above is that
the complex formula includes the effects of TCP retransmission
timeouts. For very low levels of packet loss (significantly less
than 1%), timeouts are unlikely to occur, and the formulae lead to
very similar results. At higher packet losses (1% and above), the
complex formula gives a more accurate estimate of performance (which
will always be significantly lower than the result from the simple
formula).
Note that these formulae break down as p approaches 100%.
8.5.3. Analysis of Link-Layer Effects on TCP Performance
Consider the following example:
A designer invents a new wireless link layer which, on average, loses
1% of IP packets. The link layer supports packets of up to 1040
bytes, and has a one-way delay of 20 msec.
If this link were to be used on an Internet path with a round trip
time greater than 80ms, the upper bound may be computed by:
For MSS, use 1000 bytes to exclude the 40 bytes of minimum IPv4 and
TCP headers.
For RTT, use 120 msec (80 msec for the Internet part, plus 20 msec
each way for the new wireless link).
For p, use .01. For C, assume 1.
The simple formula gives:
BW = (1000 * 8 bits) / (.120 sec * sqrt(.01)) = 666 kbit/sec
The more complex formula gives:
BW = 402.9 kbit/sec
If this were a 2 Mb/s wireless LAN, the designers might be somewhat
disappointed.
Some observations on performance:
1. We have assumed that the packet losses on the link layer are
interpreted as congestion by TCP. This is a "fact of life" that
must be accepted.
2. The equations for TCP performance are all expressed in terms of
packet loss, but many subnetwork designers think in terms of
bit-error ratio. *If* channel bit errors are independent, then
the probability of a packet being corrupted is:
p = 1 - ([1 - BER]^[FRAME_SIZE*8])
Here we assume FRAME_SIZE is in bytes and "^" represents
exponentiation. It includes the user data and all headers
(TCP,IP and subnetwork). (Note: this analysis assumes the
subnetwork does not perform ARQ or transparent fragmentation
[RFC3366].) If the inequality
BER * [FRAME_SIZE*8] << 1
holds, the packet loss probability p can be approximated by:
p = BER * [FRAME_SIZE*8]
These equations can be used to apply BER to the performance
equations above.
Note that FRAME_SIZE can vary from one packet to the next. Small
packets (such as TCP acks) generally have a smaller probability
of packet error than, say, a TCP packet carrying one MSS (maximum
segment size) of user data. A flow of small TCP acks can be
expected to be slightly more reliable than a stream of larger TCP
data segments.
It bears repeating that the above analysis assumes that bit
errors are statistically independent. Because this is not true
for many real links, our computation of p is actually an upper
bound, not the exact probability of packet loss.
There are many reasons why bit errors are not independent on real
links. Many radio links are affected by propagation fading or by
interference that lasts over many bit times. Also, links with
Forward Error Correction (FEC) generally have very non-uniform
bit error distributions that depend on the type of FEC, but in
general the uncorrected errors tend to occur in bursts even when
channel symbol errors are independent. In all such cases, our
computation of p from BER can only place an upper limit on the
packet loss rate.
If the distribution of errors under the FEC scheme is known, one
could apply the same type of analysis as above, using the correct
distribution function for the BER. It is more likely in these
FEC cases, however, that empirical methods are needed to
determine the actual packet loss rate.
3. Note that the packet size plays an important role. If the
subnetwork loss characteristics are such that large packets have
the same probability of loss as smaller packets, then larger
packets will yield improved performance.
4. We have chosen a specific RTT that might occur on a wide-area
Internet path within the USA. It is important to recognize that
a variety of RTT values are experienced in the Internet.
For example, RTTs are typically less than 10 msec in a wired LAN
environment when communicating with a local host. International
connections may have RTTs of 200 msec or more. Modems and other
low-capacity links can add considerable delay due to their long
packet transmission (serialisation) times.
Links over geostationary repeater satellites have one-way speed-
of-light delays of around 250ms, a minimum of 125ms propagation
delay up to the satellite and 125ms down. The RTT of an end-to-
end TCP connection that includes such a link can be expected to
be greater than 250ms.
Queues on heavily-congested links may back up, increasing RTTs.
Finally, virtual private networks (VPNs) and other forms of
encryption and tunneling can add significant end-to-end delay to
network connections.
9. Quality-of-Service (QoS) considerations
It is generally recognized that specific service guarantees are
needed to support real-time multimedia, toll-quality telephony, and
other performance-critical applications. The provision of such
Quality of Service guarantees in the Internet is an active area of
research and standardization. The IETF has not converged on a single
service model, set of services, or single mechanism that will offer
useful guarantees to applications and be scalable to the Internet.
Indeed, the IETF does not have a single definition of Quality of
Service. [RFC2990] represents a current understanding of the
challenges in architecting QoS for the Internet.
There are presently two architectural approaches to providing
mechanisms for QoS support in the Internet.
IP Integrated Services (Intserv) [RFC1633] provides fine-grained
service guarantees to individual flows. Flows are identified by a
flow specification (flowspec), which creates a stateful association
between individual packets by matching fields in the packet header.
Capacity is reserved for the flow, and appropriate traffic
conditioning and scheduling is installed in routers along the path.
The ReSerVation Protocol (RSVP) [RFC2205] [RFC2210] is usually, but
need not necessarily be, used to install the flow QoS state. Intserv
defines two services, in addition to the Default (best effort)
service.
1. Guaranteed Service (GS) [RFC2212] offers hard upper bounds on
delay to flows that conform to a traffic specification (TSpec).
It uses a fluid-flow model to relate the TSpec and reserved
bandwidth (RSpec) to variable delay. Non-conforming packets are
forwarded on a best-effort basis.
2. Controlled Load Service (CLS) [RFC2211] offers delay and packet
loss equivalent to that of an unloaded network to flows that
conform to a TSpec, but no hard bounds. Non-conforming packets
are forwarded on a best-effort basis.
Intserv requires installation of state information in every
participating router. Performance guarantees cannot be made unless
this state is present in every router along the path. This, along
with RSVP processing and the need for usage-based accounting, is
believed to have scalability problems, particularly in the core of
the Internet [RFC2208].
IP Differentiated Services (Diffserv) [RFC2475] provides a "toolkit"
offering coarse-grained controls to aggregates of flows. Diffserv in
itself does *not* provide QoS guarantees, but can be used to
construct services with QoS guarantees across a Diffserv domain.
Diffserv attempts to address the scaling issues associated with
Intserv by requiring state awareness only at the edge of a Diffserv
domain. At the edge, packets are classified into flows, and the
flows are conditioned (marked, policed, or shaped) to a traffic
conditioning specification (TCS). A Diffserv Codepoint (DSCP),
identifying a per-hop behavior (PHB), is set in each packet header.
The DSCP is carried in the DS-field, subsuming six bits of the former
Type-of-Service (ToS) byte [RFC791] of the IP header [RFC2474]. The
PHB denotes the forwarding behavior to be applied to the packet in
each node in the Diffserv domain. Although there is a "recommended"
DSCP associated with each PHB, the mappings from DSCPs to PHBs are
defined by the DS-domain. In fact, there can be several DSCPs
associated with the same PHB. Diffserv presently defines three PHBs.
1. The class selector PHB [RFC2474] replaces the IP precedence field
of the former ToS byte. It offers relative forwarding
priorities.
2. The Expedited Forwarding (EF) PHB [RFC3246] [RFC3248] guarantees
that packets will have a well-defined minimum departure rate
which, if not exceeded, ensures that the associated queues are
short or empty. EF is intended to support services that offer
tightly-bounded loss, delay, and delay jitter.
3. The Assured Forwarding (AF) PHB group [RFC2597] offers different
levels of forwarding assurance for each aggregated flow of
packets. Each AF group is independently allocated forwarding
resources. Packets are marked with one of three drop
precedences; those with the highest drop precedence are dropped
with lower probability than those marked with the lowest drop
precedence. DSCPs are recommended for four independent AF
groups, although a DS domain can have more or fewer AF groups.
Ongoing work in the IETF is addressing ways to support Intserv with
Diffserv. There is some belief (e.g., as expressed in [RFC2990])
that such an approach will allow individual flows to receive service
guarantees and scale to the global Internet.
The QoS guarantees that can be offered by the IP layer are a product
of two factors:
1. the concatenation of the QoS guarantees offered by the subnets
along the path of a flow. This implies that a subnet may wish to
offer multiple services (with different QoS guarantees) to the IP
layer, which can then determine which flows use which subnet
service. To put it another way, forwarding behavior in the
subnet needs to be "clued" by the forwarding behavior (service or
PHB) at the IP layer, and
2. the operation of a set of cooperating mechanisms, such as
bandwidth reservation and admission control, policy management,
traffic classification, traffic conditioning (marking, policing
and/or shaping), selective discard, queuing, and scheduling.
Note that support for QoS in subnets may require similar
mechanisms, especially when these subnets are general topology
subnets (e.g., ATM, frame relay, or MPLS) or shared media
subnets.
Many subnetwork designers face inherent tradeoffs between delay,
throughput, reliability, and cost. Other subnetworks have parameters
that manage bandwidth, internal connection state, and the like.
Therefore, the following subnetwork capabilities may be desirable,
although some might be trivial or moot if the subnet is a dedicated
point-to-point link.
1. The subnetwork should have the ability to reserve bandwidth for a
connection or flow and schedule packets accordingly.
2. Bandwidth reservations should be based on a one- or two-token
bucket model, depending on whether the service is intended to
support constant-rate or bursty traffic.
3. If a connection or flow does not use its reserved bandwidth at a
given time, the unused bandwidth should be available for other
flows.
4. Packets in excess of a connection or flow’s agreed rate should be
forwarded as best-effort or discarded, depending on the service
offered by the subnet to the IP layer.
5. If a subnet contains error control mechanisms (retransmission
and/or FEC), it should be possible for the IP layer to influence
the inherent tradeoffs between uncorrected errors, packet losses,
and delay. These capabilities at the subnet/IP layer service
boundary correspond to selection of more or less error control
and/or to selection of particular error control mechanisms within
the subnetwork.
6. The subnet layer should know, and be able to inform the IP layer,
how much fixed delay and delay jitter it offers for a flow or
connection. If the Intserv model is used, the delay jitter
component may be best expressed in terms of the TSpec/RSpec model