Request for Comments: 3449 MIT LCS
BCP: 69 V. N. Padmanabhan
Category: Best Current Practice Microsoft Research
G. Fairhurst
M. Sooriyabandara
University of Aberdeen, U.K.
December 2002
TCP Performance Implications
of Network Path Asymmetry
Status of this Memo
This document specifies an Internet Best Current Practices for the
Internet Community, and requests discussion and suggestions for
improvements. Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2002). All Rights Reserved.
Abstract
This document describes TCP performance problems that arise because
of asymmetric effects. These problems arise in several access
networks, including bandwidth-asymmetric networks and packet radio
subnetworks, for different underlying reasons. However, the end
result on TCP performance is the same in both cases: performance
often degrades significantly because of imperfection and variability
in the ACK feedback from the receiver to the sender.
The document details several mitigations to these effects, which have
either been proposed or evaluated in the literature, or are currently
deployed in networks. These solutions use a combination of local
link-layer techniques, subnetwork, and end-to-end mechanisms,
consisting of: (i) techniques to manage the channel used for the
upstream bottleneck link carrying the ACKs, typically using header
compression or reducing the frequency of TCP ACKs, (ii) techniques to
handle this reduced ACK frequency to retain the TCP sender's
acknowledgment-triggered self-clocking and (iii) techniques to
schedule the data and ACK packets in the reverse direction to improve
performance in the presence of two-way traffic. Each technique is
described, together with known issues, and recommendations for use.
A summary of the recommendations is provided at the end of the
document.
Table of Contents
1. Conventions used in this Document ...............................3
2. Motivation ....................................................4
2.1 Asymmetry due to Differences in Transmit
and Receive Capacity .........................................4
2.2 Asymmetry due to Shared Media in the Reverse Direction .......5
2.3 The General Problem ..........................................5
3. How does Asymmetry Degrade TCP Performance? .....................5
3.1 Asymmetric Capacity ..........................................5
3.2 MAC Protocol Interactions ....................................7
3.3 Bidirectional Traffic ........................................8
3.4 Loss in Asymmetric Network Paths ............................10
4. Improving TCP Performance using Host Mitigations ...............10
4.1 Modified Delayed ACKs .......................................11
4.2 Use of Large MSS ............................................12
4.3 ACK Congestion Control ......................................13
4.4 Window Prediction Mechanism .................................14
4.5 Acknowledgement based on Cwnd Estimation. ...................14
4.6 TCP Sender Pacing ...........................................14
4.7 TCP Byte Counting ...........................................15
4.8 Backpressure ................................................16
5. Improving TCP performance using Transparent Modifications ......17
5.1 TYPE 0: Header Compression ..................................18
5.1.1 TCP Header Compression ..................................18
5.1.2 Alternate Robust Header Compression Algorithms ..........19
5.2 TYPE 1: Reverse Link Bandwidth Management ...................19
5.2.1 ACK Filtering ...........................................20
5.2.2 ACK Decimation ..........................................21
5.3 TYPE 2: Handling Infrequent ACKs ............................22
5.3.1 ACK Reconstruction ......................................23
5.3.2 ACK Compaction and Companding ...........................25
5.3.3 Mitigating TCP packet bursts generated by
Infrequent ACKs .........................................26
5.4 TYPE 3: Upstream Link Scheduling ............................27
5.4.1 Per-Flow queuing at the Upstream Bottleneck Link ........27
5.4.2 ACKs-first Scheduling ...................................28
6. Security Considerations ........................................29
7. Summary ........................................................30
8. Acknowledgments ................................................32
9. References .....................................................32
10. IANA Considerations ...........................................37
Appendix: Examples of Subnetworks Exhibiting Network Path
Asymmetry ...............................................38
Authors' Addresses ................................................40
Full Copyright Statement ..........................................41
1. Conventions used in this Document
FORWARD DIRECTION: The dominant direction of data transfer over an
asymmetric network path. It corresponds to the direction with better
characteristics in terms of capacity, latency, error rate, etc. Data
transfer in the forward direction is called "forward transfer".
Packets travelling in the forward direction follow the forward path
through the IP network.
REVERSE DIRECTION: The direction in which acknowledgments of a
forward TCP transfer flow. Data transfer could also happen in this
direction (and is termed "reverse transfer"), but it is typically
less voluminous than that in the forward direction. The reverse
direction typically exhibits worse characteristics than the forward
direction. Packets travelling in the reverse direction follow the
reverse path through the IP network.
UPSTREAM LINK: The specific bottleneck link that normally has much
less capability than the corresponding downstream link. Congestion
is not confined to this link alone, and may also occur at any point
along the forward and reverse directions (e.g., due to sharing with
other traffic flows).
DOWNSTREAM LINK: A link on the forward path, corresponding to the
upstream link.
ACK: A cumulative TCP acknowledgment [RFC791]. In this document,
this term refers to a TCP segment that carries a cumulative
acknowledgement (ACK), but no data.
DELAYED ACK FACTOR, d: The number of TCP data segments acknowledged
by a TCP ACK. The minimum value of d is 1, since at most one ACK
should be sent for each data packet [RFC1122, RFC2581].
STRETCH ACK: Stretch ACKs are acknowledgements that cover more than 2
segments of previously unacknowledged data (d>2) [RFC2581]. Stretch
ACKs can occur by design (although this is not standard), due to
implementation bugs [All97b, RFC2525], or due to ACK loss [RFC2760].
NORMALIZED BANDWIDTH RATIO, k: The ratio of the raw bandwidth
(capacity) of the forward direction to the return direction, divided
by the ratio of the packet sizes used in the two directions [LMS97].
SOFTSTATE: Per-flow state established in a network device that is
used by the protocol [Cla88]. The state expires after a period of
time (i.e., is not required to be explicitly deleted when a session
expires), and is continuously refreshed while a flow continues (i.e.,
lost state may be reconstructed without needing to exchange
additional control messages).
2. Motivation
Asymmetric characteristics are exhibited by several network
technologies, including cable data networks, (e.g., DOCSIS cable TV
networks [DS00, DS01]), direct broadcast satellite (e.g., an IP
service using Digital Video Broadcast, DVB, [EN97] with an
interactive return channel), Very Small Aperture satellite Terminals
(VSAT), Asymmetric Digital Subscriber Line (ADSL) [ITU02, ANS01], and
several packet radio networks. These networks are increasingly being
deployed as high-speed Internet access networks, and it is therefore
highly desirable to achieve good TCP performance. However, the
asymmetry of the network paths often makes this challenging.
Examples of some networks that exhibit asymmetry are provided in the
Appendix.
Asymmetry may manifest itself as a difference in transmit and receive
capacity, an imbalance in the packet loss rate, or differences
between the transmit and receive paths [RFC3077]. For example, when
capacity is asymmetric, such that there is reduced capacity on
reverse path used by TCP ACKs, slow or infrequent ACK feedback
degrades TCP performance in the forward direction. Similarly,
asymmetry in the underlying Medium Access Control (MAC) and Physical
(PHY) protocols could make it expensive to transmit TCP ACKs
(disproportionately to their size), even when capacity is symmetric.
2.1 Asymmetry due to Differences in Transmit and Receive Capacity
Network paths may be asymmetric because the upstream and downstream
links operate at different rates and/or are implemented using
different technologies.
The asymmetry in capacity may be substantially increased when best
effort IP flows carrying TCP ACKs share the available upstream
capacity with other traffic flows, e.g., telephony, especially flows
that have reserved upstream capacity. This includes service
guarantees at the IP layer (e.g., the Guaranteed Service [RFC2212])
or at the subnet layer (e.g., support of Voice over IP [ITU01] using
the Unsolicited Grant service in DOCSIS [DS01], or CBR virtual
connections in ATM over ADSL [ITU02, ANS01]).
When multiple upstream links exist the asymmetry may be reduced by
dividing upstream traffic between a number of available upstream
links.
2.2 Asymmetry due to Shared Media in the Reverse Direction
In networks employing centralized multiple access control, asymmetry
may be a fundamental consequence of the hub-and-spokes architecture
of the network (i.e., a single base node communicating with multiple
downstream nodes). The central node often incurs less transmission
overhead and does not incur latency in scheduling its own downstream
transmissions. In contrast, upstream transmission is subject to
additional overhead and latency (e.g., due to guard times between
transmission bursts, and contention intervals). This can produce
significant network path asymmetry.
Upstream capacity may be further limited by the requirement that each
node must first request per-packet bandwidth using a contention MAC
protocol (e.g., DOCSIS 1.0 MAC restricts each node to sending at most
a single packet in each upstream time-division interval [DS00]). A
satellite network employing dynamic Bandwidth on Demand (BoD), also
consumes MAC resources for each packet sent (e.g., [EN00]). In these
schemes, the available uplink capacity is a function of the MAC
algorithm. The MAC and PHY schemes also introduce overhead per
upstream transmission which could be so significant that transmitting
short packets (including TCP ACKs) becomes as costly as transmitting
MTU-sized data packets.
2.3 The General Problem
Despite the technological differences between capacity-dependent and
MAC-dependent asymmetries, both kinds of network path suffer reduced
TCP performance for the same fundamental reason: the imperfection and
variability of ACK feedback. This document discusses the problem in
detail and describes several techniques that may reduce or eliminate
the constraints.
3. How does Asymmetry Degrade TCP Performance?
This section describes the implications of network path asymmetry on
TCP performance. The reader is referred to [BPK99, Bal98, Pad98,
FSS01, Sam99] for more details and experimental results.
3.1 Asymmetric Capacity
The problems that degrade unidirectional transfer performance when
the forward and return paths have very different capacities depend on
the characteristics of the upstream link. Two types of situations
arise for unidirectional traffic over such network paths: when the
upstream bottleneck link has sufficient queuing to prevent packet
(ACK) losses, and when the upstream bottleneck link has a small
buffer. Each is considered in turn.
If the upstream bottleneck link has deep queues, so that this does
not drop ACKs in the reverse direction, then performance is a strong
function of the normalized bandwidth ratio, k. For example, for a 10
Mbps downstream link and a 50 Kbps upstream link, the raw capacity
ratio is 200. With 1000-byte data packets and 40-byte ACKs, the
ratio of the packet sizes is 25. This implies that k is 200/25 = 8.
Thus, if the receiver acknowledges more frequently than one ACK every
8 (k) data packets, the upstream link will become saturated before
the downstream link, limiting the throughput in the forward
direction. Note that, the achieved TCP throughput is determined by
the minimum of the receiver advertised window or TCP congestion
window, cwnd [RFC2581].
If ACKs are not dropped (at the upstream bottleneck link) and k > 1
or k > 0.5 when delayed ACKs are used [RFC1122], TCP ACK-clocking
breaks down. Consider two data packets transmitted by the sender in
quick succession. En route to the receiver, these packets get spaced
apart according to the capacity of the smallest bottleneck link in
the forward direction. The principle of ACK clocking is that the
ACKs generated in response to receiving these data packets reflects
this temporal spacing all the way back to the sender, enabling it to
transmit new data packets that maintain the same spacing [Jac88]. ACK
clocking with delayed ACKs, reflects the spacing between data packets
that actually trigger ACKs. However, the limited upstream capacity
and queuing at the upstream bottleneck router alters the inter-ACK
spacing of the reverse path, and hence that observed at the sender.
When ACKs arrive at the upstream bottleneck link at a faster rate
than the link can support, they get queued behind one another. The
spacing between them when they emerge from the link is dilated with
respect to their original spacing, and is a function of the upstream
bottleneck capacity. Thus the TCP sender clocks out new data packets
at a slower rate than if there had been no queuing of ACKs. The
performance of the connection is no longer dependent on the
downstream bottleneck link alone; instead, it is throttled by the
rate of arriving ACKs. As a side effect, the sender's rate of cwnd
growth also slows down.
A second side effect arises when the upstream bottleneck link on the
reverse path is saturated. The saturated link causes persistent
queuing of packets, leading to an increasing path Round Trip Time
(RTT) [RFC2998] observed by all end hosts using the bottleneck link.
This can impact the protocol control loops, and may also trigger
false time out (underestimation of the path RTT by the sending host).
A different situation arises when the upstream bottleneck link has a
relatively small amount of buffer space to accommodate ACKs. As the
transmission window grows, this queue fills, and ACKs are dropped. If
the receiver were to acknowledge every packet, only one of every k
ACKs would get through to the sender, and the remaining (k-1) are
dropped due to buffer overflow at the upstream link buffer (here k is
the normalized bandwidth ratio as before). In this case, the reverse
bottleneck link capacity and slow ACK arrival rate are not directly
responsible for any degraded performance. However, the infrequency
of ACKs leads to three reasons for degraded performance:
1. The sender transmits data in large bursts of packets, limited only
by the available cwnd. If the sender receives only one ACK in k,
it transmits data in bursts of k (or more) packets because each
ACK shifts the sliding window by at least k (acknowledged) data
packets (TCP data segments). This increases the likelihood of
data packet loss along the forward path especially when k is
large, because routers do not handle large bursts of packets well.
2. Current TCP sender implementations increase their cwnd by counting
the number of ACKs they receive and not by how much data is
actually acknowledged by each ACK. The later approach, also known
as byte counting (section 4.7), is a standard implementation
option for cwnd increase during the congestion avoidance period
[RFC2581]. Thus fewer ACKs imply a slower rate of growth of the
cwnd, which degrades performance over long-delay connections.
3. The sender TCP's Fast Retransmission and Fast Recovery algorithms
[RFC2581] are less effective when ACKs are lost. The sender may
possibly not receive the threshold number of duplicate ACKs even
if the receiver transmits more than the DupACK threshold (> 3
DupACKs) [RFC2581]. Furthermore, the sender may possibly not
receive enough duplicate ACKs to adequately inflate its cwnd
during Fast Recovery.
3.2 MAC Protocol Interactions
The interaction of TCP with MAC protocols may degrade end-to-end
performance. Variable round-trip delays and ACK queuing are the main
symptoms of this problem.
One example is the impact on terrestrial wireless networks [Bal98]. A
high per-packet overhead may arise from the need for communicating
link nodes to first synchronise (e.g., via a Ready To Send / Clear to
Send (RTS/CTS) protocol) before communication and the significant
turn-around time for the wireless channel. This overhead is
variable, since the RTS/CTS exchange may need to back-off
exponentially when the remote node is busy (e.g., engaged in a
conversation with a different node). This leads to large and
variable communication latencies in packet-radio networks.
An asymmetric workload (more downstream than upstream traffic) may
cause ACKs to be queued in some wireless nodes (especially in the end
host modems), exacerbating the variable latency. Queuing may also
occur in other shared media, e.g., cable modem uplinks, BoD access
systems often employed on shared satellite channels.
Variable latency and ACK queuing reduces the smoothness of the TCP
data flow. In particular, ACK traffic can interfere with the flow of
data packets, increasing the traffic load of the system.
TCP measures the path RTT, and from this calculates a smoothed RTT
estimate (srtt) and a linear deviation, rttvar. These are used to
estimate a path retransmission timeout (RTO) [RFC2988], set to srtt +
4*rttvar. For most wired TCP connections, the srtt remains constant
or has a low linear deviation. The RTO therefore tracks the path
RTT, and the TCP sender will respond promptly when multiple losses
occur in a window. In contrast, some wireless networks exhibit a
high variability in RTT, causing the RTO to significantly increase
(e.g., on the order of 10 seconds). Paths traversing multiple
wireless hops are especially vulnerable to this effect, because this
increases the probability that the intermediate nodes may already be
engaged in conversation with other nodes. The overhead in most MAC
schemes is a function of both the number and size of packets.
However, the MAC contention problem is a significant function of the
number of packets (e.g., ACKs) transmitted rather than their size.
In other words, there is a significant cost to transmitting a packet
regardless of packet size.
Experiments conducted on the Ricochet packet radio network in 1996
and 1997 demonstrated the impact of radio turnarounds and the
corresponding increased RTT variability, resulting in degraded TCP
performance. It was not uncommon for TCP connections to experience
timeouts of 9 - 12 seconds, with the result that many connections
were idle for a significant fraction of their lifetime (e.g.,
sometimes 35% of the total transfer time). This leads to under-
utilization of the available capacity. These effects may also occur
in other wireless subnetworks.
3.3 Bidirectional Traffic
Bidirectional traffic arises when there are simultaneous TCP
transfers in the forward and reverse directions over an asymmetric
network path, e.g., a user who sends an e-mail message in the reverse
direction while simultaneously receiving a web page in the forward
direction. To simplify the discussion, only one TCP connection in
each direction is considered. In many practical cases, several
simultaneous connections need to share the available capacity,
increasing the level of congestion.
Bidirectional traffic makes the effects discussed in section 3.1 more
pronounced, because part of the upstream link bandwidth is consumed
by the reverse transfer. This effectively increases the degree of
bandwidth asymmetry. Other effects also arise due to the interaction
between data packets of the reverse transfer and ACKs of the forward
transfer. Suppose at the time the forward TCP connection is
initiated, the reverse TCP connection has already saturated the
bottleneck upstream link with data packets. There is then a high
probability that many ACKs of the new forward TCP connection will
encounter a full upstream link buffer and hence get dropped. Even
after these initial problems, ACKs of the forward connection could
get queued behind large data packets of the reverse connection. The
larger data packets may have correspondingly long transmission times
(e.g., it takes about 280 ms to transmit a 1 Kbyte data packet over a
28.8 kbps line). This causes the forward transfer to stall for long
periods of time. It is only at times when the reverse connection
loses packets (due to a buffer overflow at an intermediate router)
and slows down, that the forward connection gets the opportunity to
make rapid progress and build up its cwnd.
When ACKs are queued behind other traffic for appreciable periods of
time, the burst nature of TCP traffic and self-synchronizing effects
can result in an effect known as ACK Compression [ZSC91], which
reduces the throughput of TCP. It occurs when a series of ACKs, in
one direction are queued behind a burst of other packets (e.g., data
packets traveling in the same direction) and become compressed in
time. This results in an intense burst of data packets in the other
direction, in response to the burst of compressed ACKs arriving at
the server. This phenomenon has been investigated in detail for
bidirectional traffic, and recent analytical work [LMS97] has
predicted ACK Compression may also result from bi-directional
transmission with asymmetry, and was observed in practical asymmetric
satellite subnetworks [FSS01]. In the case of extreme asymmetry
(k>>1), the inter-ACK spacing can increase due to queuing (section
3.1), resulting in ACK dilation.
In summary, sharing of the upstream bottleneck link by multiple flows
(e.g., IP flows to the same end host, or flows to a number of end
hosts sharing a common upstream link) increases the level of ACK
Congestion. The presence of bidirectional traffic exacerbates the
constraints introduced by bandwidth asymmetry because of the adverse
interaction between (large) data packets of a reverse direction
connection and the ACKs of a forward direction connection.
3.4 Loss in Asymmetric Network Paths
Loss may occur in either the forward or reverse direction. For data
transfer in the forward direction this results respectively in loss
of data packets and ACK packets. Loss of ACKs is less significant
than loss of data packets, because it generally results in stretch
ACKs [CR98, FSS01].
In the case of long delay paths, a slow upstream link [RFC3150] can
lead to another complication when the end host uses TCP large windows
[RFC1323] to maximize throughput in the forward direction. Loss of
data packets on the forward path, due to congestion, or link loss,
common for some wireless links, will generate a large number of
back-to-back duplicate ACKs (or TCP SACK packets [RFC2018]), for each
correctly received data packet following a loss. The TCP sender
employs Fast Retransmission and Recovery [RFC2581] to recover from
the loss, but even if this is successful, the ACK to the
retransmitted data segment may be significantly delayed by other
duplicate ACKs still queued at the upstream link buffer. This can
ultimately lead to a timeout [RFC2988] and a premature end to the TCP
Slow Start [RFC2581]. This results in poor forward path throughput.
Section 5.3 describes some mitigations to counter this.
4. Improving TCP Performance using Host Mitigations
There are two key issues that need to be addressed to improve TCP
performance over asymmetric networks. The first is to manage the
capacity of the upstream bottleneck link, used by ACKs and possibly
other traffic. A number of techniques exist which work by reducing
the number of ACKs that flow in the reverse direction. This has the
side effect of potentially destroying the desirable self-clocking
property of the TCP sender where transmission of new data packets is
triggered by incoming ACKs. Thus, the second issue is to avoid any
adverse impact of infrequent ACKs.
Each of these issues can be handled by local link-layer solutions
and/or by end-to-end techniques. This section discusses end-to-end
modifications. Some techniques require TCP receiver changes
(sections 4.1 4.4, 4.5), some require TCP sender changes (sections
4.6, 4.7), and a pair requires changes to both the TCP sender and
receiver (sections 4.2, 4.3). One technique requires a sender
modification at the receiving host (section 4.8). The techniques may
be used independently, however some sets of techniques are
complementary, e.g., pacing (section 4.6) and byte counting (section
4.7) which have been bundled into a single TCP Sender Adaptation
scheme [BPK99].
It is normally envisaged that these changes would occur in the end
hosts using the asymmetric path, however they could, and have, been
used in a middle-box or Protocol Enhancing Proxy (PEP) [RFC3135]
employing split TCP. This document does not discuss the issues
concerning PEPs. Section 4 describes several techniques, which do
not require end-to-end changes.
4.1 Modified Delayed ACKs
There are two standard methods that can be used by TCP receivers to
generate acknowledgments. The method outlined in [RFC793] generates
an ACK for each incoming data segment (i.e., d=1). [RFC1122] states
that hosts should use "delayed acknowledgments". Using this
algorithm, an ACK is generated for at least every second full-sized
segment (d=2), or if a second full-sized segment does not arrive
within a given timeout (which must not exceed 500 ms [RFC1122], and
is typically less than 200 ms). Relaxing the latter constraint
(i.e., allowing d>2) may generate Stretch ACKs [RFC2760]. This
provides a possible mitigation, which reduces the rate at which ACKs
are returned by the receiver. An implementer should only deviate
from this requirement after careful consideration of the implications
[RFC2581].
Reducing the number of ACKs per received data segment has a number of
undesirable effects including:
(i) Increased path RTT
(ii) Increased time for TCP to open the cwnd
(iii) Increased TCP sender burst size, since cwnd opens in larger
steps
In addition, a TCP receiver is often unable to determine an optimum
setting for a large d, since it will normally be unaware of the
details of the properties of the links that form the path in the
reverse direction.
RECOMMENDATION: A TCP receiver must use the standard TCP algorithm
for sending ACKs as specified in [RFC2581]. That is, it may delay
sending an ACK after it receives a data segment [RFC1122]. When ACKs
are delayed, the receiver must generate an ACK within 500 ms and the
ACK should be generated for at least every second full sized segment
(MSS) of received data [RFC2581]. This will result in an ACK delay
factor (d) that does not exceed a value of 2. Changing the algorithm
would require a host modification to the TCP receiver and awareness
by the receiving host that it is using a connection with an
asymmetric path. Such a change has many drawbacks in the general
case and is currently not recommended for use within the Internet.
4.2 Use of Large MSS
A TCP sender that uses a large Maximum Segment Size (MSS) reduces the
number of ACKs generated per transmitted byte of data.
Although individual subnetworks may support a large MTU, the majority
of current Internet links employ an MTU of approx 1500 bytes (that of
Ethernet). By setting the Don't Fragment (DF) bit in the IP header,
Path MTU (PMTU) discovery [RFC1191] may be used to determine the
maximum packet size (and hence MSS) a sender can use on a given
network path without being subjected to IP fragmentation, and
provides a way to automatically select a suitable MSS for a specific
path. This also guarantees that routers will not perform IP
fragmentation of normal data packets.
By electing not to use PMTU Discovery, an end host may choose to use
IP fragmentation by routers along the path in the forward direction
[RFC793]. This allows an MSS larger than smallest MTU along the
path. However, this increases the unit of error recovery (TCP
segment) above the unit of transmission (IP packet). This is not
recommended, since it can increase the number of retransmitted
packets following loss of a single IP packet, leading to reduced
efficiency, and potentially aggravating network congestion [Ken87].
Choosing an MSS larger than the forward path minimum MTU also permits
the sender to transmit more initial packets (a burst of IP fragments
for each TCP segment) when a session starts or following RTO expiry,
increasing the aggressiveness of the sender compared to standard TCP
[RFC2581]. This can adversely impact other standard TCP sessions
that share a network path.
RECOMMENDATION:
A larger forward path MTU is desirable for paths with bandwidth
asymmetry. Network providers may use a large MTU on links in the
forward direction. TCP end hosts using Path MTU discovery may be
able to take advantage of a large MTU by automatically selecting an
appropriate larger MSS, without requiring modification. The use of
Path MTU discovery [RFC1191] is therefore recommended.
Increasing the unit of error recovery and congestion control (MSS)
above the unit of transmission and congestion loss (the IP packet) by
using a larger end host MSS and IP fragmentation in routers is not
recommended.
4.3 ACK Congestion Control
ACK Congestion Control (ACC) is an experimental technique that
operates end to end. ACC extends congestion control to ACKs, since
they may make non-negligible demands on resources (e.g., packet
buffers, and MAC transmission overhead) at an upstream bottleneck
link. It has two parts: (a) a network mechanism indicating to the
receiver that the ACK path is congested, and (b) the receiver's
response to such an indication.
A router feeding an upstream bottleneck link may detect incipient
congestion, e.g., using an algorithm based on RED (Random Early
Detection) [FJ93]. This may track the average queue size over a time
window in the recent past. If the average exceeds a threshold, the
router may select a packet at random. If the packet IP header has
the Explicit Congestion Notification Capable Transport (ECT) bit set,
the router may mark the packet, i.e., sets an Explicit Congestion
Notification (ECN) [RFC3168] bit(s) in the IP header, otherwise the
packet is normally dropped. The ECN notification received by the end
host is reflected back to the sending TCP end host, to trigger
congestion avoidance [RFC3168]. Note that routers implementing RED
with ECN, do not eliminate packet loss, and may drop a packet (even
when the ECT bit is set). It is also possible to use an algorithm
other than RED to decide when to set the ECN bit.
ACC extends ECN so that both TCP data packets and ACKs set the ECT
bit and are thus candidates for being marked with an ECN bit.
Therefore, upon receiving an ACK with the ECN bit set [RFC3168], a
TCP receiver reduces the rate at which it sends ACKs. It maintains a
dynamically varying delayed-ACK factor, d, and sends one ACK for
every d data packets received. When it receives a packet with the
ECN bit set, it increases d multiplicatively, thereby
multiplicatively decreasing the frequency of ACKs. For each
subsequent RTT (e.g., determined using the TCP RTTM option [RFC1323])
during which it does not receive an ECN, it linearly decreases the
factor d, increasing the frequency of ACKs. Thus, the receiver
mimics the standard congestion control behavior of TCP senders in the
manner in which it sends ACKs.
The maximum value of d is determined by the TCP sender window size,
which could be conveyed to the receiver in a new (experimental) TCP
option. The receiver should send at least one ACK (preferably more)
for each window of data from the sender (i.e., d < (cwnd/mss)) to
prevent the sender from stalling until the receiver's delayed ACK
timer triggers an ACK to be sent.
RECOMMENDATION: ACK Congestion Control (ACC) is an experimental
technique that requires TCP sender and receiver modifications. There
is currently little experience of using such techniques in the
Internet. Future versions of TCP may evolve to include this or
similar techniques. These are the subject of ongoing research. ACC
is not recommended for use within the Internet in its current form.
4.4 Window Prediction Mechanism
The Window Prediction Mechanism (WPM) is a TCP receiver side
mechanism [CLP98] that uses a dynamic ACK delay factor (varying d)
resembling the ACC scheme (section 4.3). The TCP receiver
reconstructs the congestion control behavior of the TCP sender by
predicting a cwnd value. This value is used along with the allowed
window to adjust the receiver's value of d. WPM accommodates for
unnecessary retransmissions resulting from losses due to link errors.
RECOMMENDATION: Window Prediction Mechanism (WPM) is an experimental
TCP receiver side modification. There is currently little experience
of using such techniques in the Internet. Future versions of TCP may
evolve to include this or similar techniques. These are the subjects
of ongoing research. WPM is not recommended for use within the
Internet in its current form.
4.5 Acknowledgement based on Cwnd Estimation.
Acknowledgement based on Cwnd Estimation (ACE) [MJW00] attempts to
measure the cwnd at the TCP receiver and maintain a varying ACK delay
factor (d). The cwnd is estimated by counting the number of packets
received during a path RTT. The technique may improve accuracy of
prediction of a suitable cwnd.
RECOMMENDATION: Acknowledgement based on Cwnd Estimation (ACE) is an
experimental TCP receiver side modification. There is currently
little experience of using such techniques in the Internet. Future
versions of TCP may evolve to include this or similar techniques.
These are the subject of ongoing research. ACE is not recommended
for use within the Internet in its current form.
4.6 TCP Sender Pacing
Reducing the frequency of ACKs may alleviate congestion of the
upstream bottleneck link, but can lead to increased size of TCP
sender bursts (section 4.1). This may slow the growth of cwnd, and
is undesirable when used over shared network paths since it may
significantly increase the maximum number of packets in the
bottleneck link buffer, potentially resulting in an increase in
network congestion. This may also lead to ACK Compression [ZSC91].
TCP Pacing [AST00], generally referred to as TCP Sender pacing,
employs an adapted TCP sender to alleviating transmission burstiness.
A bound is placed on the maximum number of packets the TCP sender can
transmit back-to-back (at local line rate), even if the window(s)
allow the transmission of more data. If necessary, more bursts of
data packets are scheduled for later points in time computed based on
the transmission rate of the TCP connection. The transmission rate
may be estimated from the ratio cwnd/srtt. Thus, large bursts of
data packets get broken up into smaller bursts spread over time.
A subnetwork may also provide pacing (e.g., Generic Traffic Shaping
(GTS)), but implies a significant increase in the per-packet
processing overhead and buffer requirement at the router where
shaping is performed (section 5.3.3).
RECOMMENDATIONS: TCP Sender Pacing requires a change to
implementation of the TCP sender. It may be beneficial in the
Internet and will significantly reduce the burst size of packets
transmitted by a host. This successfully mitigates the impact of
receiving Stretch ACKs. TCP Sender Pacing implies increased
processing cost per packet, and requires a prediction algorithm to
suggest a suitable transmission rate. There are hence performance
trade-offs between end host cost and network performance.
Specification of efficient algorithms remains an area of ongoing
research. Use of TCP Sender Pacing is not expected to introduce new
problems. It is an experimental mitigation for TCP hosts that may
control the burstiness of transmission (e.g., resulting from Type 1
techniques, section 5.1.2), however it is not currently widely
deployed. It is not recommended for use within the Internet in its
current form.
4.7 TCP Byte Counting
The TCP sender can avoid slowing growth of cwnd by taking into
account the volume of data acknowledged by each ACK, rather than
opening the cwnd based on the number of received ACKs. So, if an ACK
acknowledges d data packets (or TCP data segments), the cwnd would
grow as if d separate ACKs had been received. This is called TCP
Byte Counting [RFC2581, RFC2760]. (One could treat the single ACK as
being equivalent to d/2, instead of d ACKs, to mimic the effect of
the TCP delayed ACK algorithm.) This policy works because cwnd
growth is only tied to the available capacity in the forward
direction, so the number of ACKs is immaterial.
This may mitigate the impact of asymmetry when used in combination
with other techniques (e.g., a combination of TCP Pacing
(section4.6), and ACC (section 4.3) associated with a duplicate ACK
threshold at the receiver.)
The main issue is that TCP byte counting may generate undesirable
long bursts of TCP packets at the sender host line rate. An
implementation must also consider that data packets in the forward
direction and ACKs in the reverse direction may both travel over
network paths that perform some amount of packet reordering.
Reordering of IP packets is currently common, and may arise from
various causes [BPS00].
RECOMMENDATION: TCP Byte Counting requires a small TCP sender
modification. In its simplest form, it can generate large bursts of
TCP data packets, particularly when Stretch ACKs are received.
Unlimited byte counting is therefore not allowed [RFC2581] for use
within the Internet.
It is therefore strongly recommended [RFC2581, RFC2760] that any byte
counting scheme should include a method to mitigate the potentially
large bursts of TCP data packets the algorithm can cause (e.g., TCP
Sender Pacing (section 4.6), ABC [abc-ID]). If the burst size or
sending rate of the TCP sender can be controlled then the scheme may
be beneficial when Stretch ACKs are received. Determining safe
algorithms remain an area of ongoing research. Further
experimentation will then be required to assess the success of these
safeguards, before they can be recommended for use in the Internet.
4.8 Backpressure
Backpressure is a technique to enhance the performance of
bidirectional traffic for end hosts directly connected to the
upstream bottleneck link [KVR98]. A limit is set on how many data
packets of upstream transfers can be enqueued at the upstream
bottleneck link. In other words, the bottleneck link queue exerts
'backpressure' on the TCP (sender) layer. This requires a modified
implementation, compared to that currently deployed in many TCP
stacks. Backpressure ensures that ACKs of downstream connections do
not get starved at the upstream bottleneck, thereby improving
performance of the downstream connections. Similar generic schemes
that may be implemented in hosts/routers are discussed in section
5.4.
Backpressure can be unfair to a reverse direction connection and make
its throughput highly sensitive to the dynamics of the forward
connection(s).
RECOMMENDATION: Backpressure requires an experimental modification to
the sender protocol stack of a host directly connected to an upstream
bottleneck link. Use of backpressure is an implementation issue,
rather than a network protocol issue. Where backpressure is
implemented, the optimizations described in this section could be
desirable and can benefit bidirectional traffic for hosts.
Specification of safe algorithms for providing backpressure is still
a subject of ongoing research. The technique is not recommended for
use within the Internet in its current form.
5. Improving TCP performance using Transparent Modifications
Various link and network layer techniques have been suggested to
mitigate the effect of an upstream bottleneck link. These techniques
may provide benefit without modification to either the TCP sender or
receiver, or may alternately be used in conjunction with one or more
of the schemes identified in section 4. In this document, these
techniques are known as "transparent" [RFC3135], because at the
transport layer, the TCP sender and receiver are not necessarily
aware of their existence. This does not imply that they do not
modify the pattern and timing of packets as observed at the network
layer. The techniques are classified here into three types based on
the point at which they are introduced.
Most techniques require the individual TCP connections passing over
the bottleneck link(s) to be separately identified and imply that
some per-flow state is maintained for active TCP connections. A link
scheduler may also be employed (section 5.4). The techniques (with
one exception, ACK Decimation (section 5.2.2) require:
(i) Visibility of an unencrypted IP and TCP packet header (e.g., no
use of IPSec with payload encryption [RFC2406]).
(ii) Knowledge of IP/TCP options and ability to inspect packets with
tunnel encapsulations (e.g., [RFC2784]) or to suspend
processing of packets with unknown formats.
(iii) Ability to demultiplex flows (by using address/class/port
number, or an explicit flow-id).
[RFC3135] describes a class of network device that provides more than
forwarding of packets, and which is known as a Protocol Enhancing
Proxy (PEP). A large spectrum of PEP devices exists, ranging from
simple devices (e.g., ACK filtering) to more sophisticated devices
(e.g., stateful devices that split a TCP connection into two separate
parts). The techniques described in section 5 of this document
belong to the simpler type, and do not inspect or modify any TCP or
UDP payload data. They also do not modify port numbers or link
addresses. Many of the risks associated with more complex PEPs do
not exist for these schemes. Further information about the operation
and the risks associated with using PEPs are described in [RFC3135].
5.1 TYPE 0: Header Compression
A client may reduce the volume of bits used to send a single ACK by
using compression [RFC3150, RFC3135]. Most modern dial-up modems
support ITU-T V.42 bulk compression. In contrast to bulk
compression, header compression is known to be very effective at
reducing the number of bits sent on the upstream link [RFC1144]. This
relies on the observation that most TCP packet headers vary only in a
few bit positions between successive packets in a flow, and that the
variations can often be predicted.
5.1.1 TCP Header Compression
TCP header compression [RFC1144] (sometimes known as V-J compression)
is a Proposed Standard describing use over low capacity links running
SLIP or PPP [RFC3150]. It greatly reduces the size of ACKs on the
reverse link when losses are infrequent (a situation that ensures
that the state of the compressor and decompressor are synchronized).
However, this alone does not address all of the asymmetry issues:
(i) In some (e.g., wireless) subnetworks there is a significant
per-packet MAC overhead that is independent of packet size
(section 3.2).
(ii) A reduction in the size of ACKs does not prevent adverse
interaction with large upstream data packets in the presence
of bidirectional traffic (section 3.3).
(iii) TCP header compression cannot be used with packets that have
IP or TCP options (including IPSec [