Request for Comments: 3649 ICSI
Category: Experimental December 2003
HighSpeed TCP for Large Congestion Windows
Status of this Memo
This memo defines an Experimental Protocol for the Internet
community. It does not specify an Internet standard of any kind.
Discussion and suggestions for improvement are requested.
Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2003). All Rights Reserved.
Abstract
The proposals in this document are experimental. While they may be
deployed in the current Internet, they do not represent a consensus
that this is the best method for high-speed congestion control. In
particular, we note that alternative experimental proposals are
likely to be forthcoming, and it is not well understood how the
proposals in this document will interact with such alternative
proposals.
This document proposes HighSpeed TCP, a modification to TCP’s
congestion control mechanism for use with TCP connections with large
congestion windows. The congestion control mechanisms of the current
Standard TCP constrains the congestion windows that can be achieved
by TCP in realistic environments. For example, for a Standard TCP
connection with 1500-byte packets and a 100 ms round-trip time,
achieving a steady-state throughput of 10 Gbps would require an
average congestion window of 83,333 segments, and a packet drop rate
of at most one congestion event every 5,000,000,000 packets (or
equivalently, at most one congestion event every 1 2/3 hours). This
is widely acknowledged as an unrealistic constraint. To address this
limitation of TCP, this document proposes HighSpeed TCP, and solicits
experimentation and feedback from the wider community.
Table of Contents
1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 2
2. The Problem Description.. . . . . . . . . . . . . . . . . . . . 3
3. Design Guidelines.. . . . . . . . . . . . . . . . . . . . . . . 4
4. Non-Goals.. . . . . . . . . . . . . . . . . . . . . . . . . . . 5
5. Modifying the TCP Response Function.. . . . . . . . . . . . . . 6
6. Fairness Implications of the HighSpeed Response
Function. . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
7. Translating the HighSpeed Response Function into
Congestion Control Parameters . . . . . . . . . . . . . . . . . 12
8. An alternate, linear response functions.. . . . . . . . . . . . 13
9. Tradeoffs for Choosing Congestion Control Parameters. . . . . . 16
9.1. The Number of Round-Trip Times between Loss Events . . . . 17
9.2. The Number of Packet Drops per Loss Event, with Drop-Tail. 17
10. Related Issues . . . . . . . . . . . . . . . . . . . . . . . . 18
10.1. Slow-Start. . . . . . . . . . . . . . . . . . . . . . . . 18
10.2. Limiting burstiness on short time scales. . . . . . . . . 19
10.3. Other limitations on window size. . . . . . . . . . . . . 19
10.4. Implementation issues.. . . . . . . . . . . . . . . . . . 19
11. Deployment issues. . . . . . . . . . . . . . . . . . . . . . . 20
11.1. Deployment issues of HighSpeed TCP. . . . . . . . . . . . 20
11.2. Deployment issues of Scalable TCP . . . . . . . . . . . . 22
12. Related Work in HighSpeed TCP. . . . . . . . . . . . . . . . . 23
13. Relationship to other Work.. . . . . . . . . . . . . . . . . . 25
14. Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . 25
15. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 25
16. Normative References . . . . . . . . . . . . . . . . . . . . . 26
17. Informative References . . . . . . . . . . . . . . . . . . . . 26
18. Security Considerations. . . . . . . . . . . . . . . . . . . . 28
19. IANA Considerations. . . . . . . . . . . . . . . . . . . . . . 28
A. TCP’s Loss Event Rate in Steady-State. . . . . . . . . . . . . 29
B. A table for a(w) and b(w). . . . . . . . . . . . . . . . . . . 30
C. Exploring the time to converge to fairness . . . . . . . . . . 32
Author’s Address . . . . . . . . . . . . . . . . . . . . . . . 33
Full Copyright Statement . . . . . . . . . . . . . . . . . . . 34
1. Introduction
This document proposes HighSpeed TCP, a modification to TCP’s
congestion control mechanism for use with TCP connections with large
congestion windows. In a steady-state environment, with a packet
loss rate p, the current Standard TCP’s average congestion window is
roughly 1.2/sqrt(p) segments. This places a serious constraint on
the congestion windows that can be achieved by TCP in realistic
environments. For example, for a Standard TCP connection with 1500-
byte packets and a 100 ms round-trip time, achieving a steady-state
throughput of 10 Gbps would require an average congestion window of
83,333 segments, and a packet drop rate of at most one congestion
event every 5,000,000,000 packets (or equivalently, at most one
congestion event every 1 2/3 hours). The average packet drop rate of
at most 2*10^(-10) needed for full link utilization in this
environment corresponds to a bit error rate of at most 2*10^(-14),
and this is an unrealistic requirement for current networks.
To address this fundamental limitation of TCP and of the TCP response
function (the function mapping the steady-state packet drop rate to
TCP’s average sending rate in packets per round-trip time), this
document describes a modified TCP response function for regimes with
higher congestion windows. This document also solicits
experimentation and feedback on HighSpeed TCP from the wider
community.
Because HighSpeed TCP’s modified response function would only take
effect with higher congestion windows, HighSpeed TCP does not modify
TCP behavior in environments with heavy congestion, and therefore
does not introduce any new dangers of congestion collapse. However,
if relative fairness between HighSpeed TCP connections is to be
preserved, then in our view any modification to the TCP response
function should be addressed in the IETF, rather than made as ad hoc
decisions by individual implementors or TCP senders. Modifications
to the TCP response function would also have implications for
transport protocols that use TFRC and other forms of equation-based
congestion control, as these congestion control mechanisms directly
use the TCP response function [RFC3448].
This proposal for HighSpeed TCP focuses specifically on a proposed
change to the TCP response function, and its implications for TCP.
This document does not address what we view as a separate fundamental
issue, of the mechanisms required to enable best-effort connections
to *start* with large initial windows. In our view, while HighSpeed
TCP proposes a somewhat fundamental change to the TCP response
function, at the same time it is a relatively simple change to
implement in a single TCP sender, and presents no dangers in terms of
congestion collapse. In contrast, in our view, the problem of
enabling connections to *start* with large initial windows is
inherently more risky and structurally more difficult, requiring some
form of explicit feedback from all of the routers along the path.
This is another reason why we would propose addressing the problem of
starting with large initial windows separately, and on a separate
timetable, from the problem of modifying the TCP response function.
2. The Problem Description
This section describes the number of round-trip times between
congestion events required for a Standard TCP flow to achieve an
average throughput of B bps, given packets of D bytes and a round-
trip time of R seconds. A congestion event refers to a window of
data with one or more dropped or ECN-marked packets (where ECN stands
for Explicit Congestion Notification).
From Appendix A, achieving an average TCP throughput of B bps
requires a loss event at most every BR/(12D) round-trip times. This
is illustrated in Table 1, for R = 0.1 seconds and D = 1500 bytes.
The table also gives the average congestion window W of BR/(8D), and
the steady-state packet drop rate P of 1.5/W^2.
TCP Throughput (Mbps) RTTs Between Losses W P
--------------------- ------------------- ---- -----
1 5.5 8.3 0.02
10 55.5 83.3 0.0002
100 555.5 833.3 0.000002
1000 5555.5 8333.3 0.00000002
10000 55555.5 83333.3 0.0000000002
Table 1: RTTs Between Congestion Events for Standard TCP, for
1500-Byte Packets and a Round-Trip Time of 0.1 Seconds.
This document proposes HighSpeed TCP, a minimal modification to TCP’s
increase and decrease parameters, for TCP connections with larger
congestion windows, to allow TCP to achieve high throughput with more
realistic requirements for the steady-state packet drop rate.
Equivalently, HighSpeed TCP has more realistic requirements for the
number of round-trip times between loss events.
3. Design Guidelines
Our proposal for HighSpeed TCP is motivated by the following
requirements:
* Achieve high per-connection throughput without requiring
unrealistically low packet loss rates.
* Reach high throughput reasonably quickly when in slow-start.
* Reach high throughput without overly long delays when recovering
from multiple retransmit timeouts, or when ramping-up from a
period with small congestion windows.
* No additional feedback or support required from routers:
For example, the goal is for acceptable performance in both ECN-
capable and non-ECN-capable environments, and with Drop-Tail as well
as with Active Queue Management such as RED in the routers.
* No additional feedback required from TCP receivers.
* TCP-compatible performance in environments with moderate or high
congestion (e.g., packet drop rates of 1% or higher):
Equivalently, the requirement is that there be no additional load on
the network (in terms of increased packet drop rates) in environments
with moderate or high congestion.
* Performance at least as good as Standard TCP in environments with
moderate or high congestion.
* Acceptable transient performance, in terms of increases in the
congestion window in one round-trip time, responses to severe
congestion, and convergence times to fairness.
Currently, users wishing to achieve throughputs of 1 Gbps or more
typically open up multiple TCP connections in parallel, or use MulTCP
[CO98,GRK99], which behaves roughly like the aggregate of N virtual
TCP connections. While this approach suffices for the occasional
user on well-provisioned links, it leaves the parameter N to be
determined by the user, and results in more aggressive performance
and higher steady-state packet drop rates if used in environments
with periods of moderate or high congestion. We believe that a new
approach is needed that offers more flexibility, more effectively
scales to a wide range of available bandwidths, and competes more
fairly with Standard TCP in congested environments.
4. Non-Goals
The following are explicitly *not* goals of our work:
* Non-goal: TCP-compatible performance in environments with very low
packet drop rates.
We note that our proposal does not require, or deliver, TCP-
compatible performance in environments with very low packet drop
rates, e.g., with packet loss rates of 10^-5 or 10^-6. As we discuss
later in this document, we assume that Standard TCP is unable to make
effective use of the available bandwidth in environments with loss
rates of 10^-6 in any case, so that it is acceptable and appropriate
for HighSpeed TCP to perform more aggressively than Standard TCP in
such an environment.
* Non-goal: Ramping-up more quickly than allowed by slow-start.
It is our belief that ramping-up more quickly than allowed by slow-
start would necessitate more explicit feedback from routers along the
path. The proposal for HighSpeed TCP is focused on changes to TCP
that could be effectively deployed in the current Internet
environment.
* Non-goal: Avoiding oscillations in environments with only one-way,
long-lived flows all with the same round-trip times.
While we agree that attention to oscillatory behavior is useful,
avoiding oscillations in aggregate throughput has not been our
primary consideration, particularly for simplified environments
limited to one-way, long-lived flows all with the same, large round-
trip times. Our assessment is that some oscillatory behavior in
these extreme environments is an acceptable price to pay for the
other benefits of HighSpeed TCP.
5. Modifying the TCP Response Function
The TCP response function, w = 1.2/sqrt(p), gives TCP’s average
congestion window w in MSS-sized segments, as a function of the
steady-state packet drop rate p [FF98]. This TCP response function
is a direct consequence of TCP’s Additive Increase Multiplicative
Decrease (AIMD) mechanisms of increasing the congestion window by
roughly one segment per round-trip time in the absence of congestion,
and halving the congestion window in response to a round-trip time
with a congestion event. This response function for Standard TCP is
reflected in the table below. In this proposal we restrict our
attention to TCP performance in environments with packet loss rates
of at most 10^-2, and so we can ignore the more complex response
functions that are required to model TCP performance in more
congested environments with retransmit timeouts. From Appendix A, an
average congestion window of W corresponds to an average of 2/3 W
round-trip times between loss events for Standard TCP (with the
congestion window varying from 2/3 W to 4/3 W).
Packet Drop Rate P Congestion Window W RTTs Between Losses
------------------ ------------------- -------------------
10^-2 12 8
10^-3 38 25
10^-4 120 80
10^-5 379 252
10^-6 1200 800
10^-7 3795 2530
10^-8 12000 8000
10^-9 37948 25298
10^-10 120000 80000
Table 2: TCP Response Function for Standard TCP. The average
congestion window W in MSS-sized segments is given as a function of
the packet drop rate P.
To specify a modified response function for HighSpeed TCP, we use
three parameters, Low_Window, High_Window, and High_P. To ensure TCP
compatibility, the HighSpeed response function uses the same response
function as Standard TCP when the current congestion window is at
most Low_Window, and uses the HighSpeed response function when the
current congestion window is greater than Low_Window. In this
document we set Low_Window to 38 MSS-sized segments, corresponding to
a packet drop rate of 10^-3 for TCP.
To specify the upper end of the HighSpeed response function, we
specify the packet drop rate needed in the HighSpeed response
function to achieve an average congestion window of 83000 segments.
This is roughly the window needed to sustain 10 Gbps throughput, for
a TCP connection with the default packet size and round-trip time
used earlier in this document. For High_Window set to 83000, we
specify High_P of 10^-7; that is, with HighSpeed TCP a packet drop
rate of 10^-7 allows the HighSpeed TCP connection to achieve an
average congestion window of 83000 segments. We believe that this
loss rate sets an achievable target for high-speed environments,
while still allowing acceptable fairness for the HighSpeed response
function when competing with Standard TCP in environments with packet
drop rates of 10^-4 or 10^5.
For simplicity, for the HighSpeed response function we maintain the
property that the response function gives a straight line on a log-
log scale (as does the response function for Standard TCP, for low to
moderate congestion). This results in the following response
function, for values of the average congestion window W greater than
Low_Window:
W = (p/Low_P)^S Low_Window,
for Low_P the packet drop rate corresponding to Low_Window, and for S
as following constant [FRS02]:
S = (log High_Window - log Low_Window)/(log High_P - log Low_P).
(In this paper, "log x" refers to the log base 10.) For example, for
Low_Window set to 38, we have Low_P of 10^-3 (for compatibility with
Standard TCP). Thus, for High_Window set to 83000 and High_P set to
10^-7, we get the following response function:
W = 0.12/p^0.835. (1)
This HighSpeed response function is illustrated in Table 3 below.
For HighSpeed TCP, the number of round-trip times between losses,
1/(pW), equals 12.7 W^0.2, for W > 38 segments.
Packet Drop Rate P Congestion Window W RTTs Between Losses
------------------ ------------------- -------------------
10^-2 12 8
10^-3 38 25
10^-4 263 38
10^-5 1795 57
10^-6 12279 83
10^-7 83981 123
10^-8 574356 180
10^-9 3928088 264
10^-10 26864653 388
Table 3: TCP Response Function for HighSpeed TCP. The average
congestion window W in MSS-sized segments is given as a function of
the packet drop rate P.
We believe that the problem of backward compatibility with Standard
TCP requires a response function that is quite close to that of
Standard TCP for loss rates of 10^-1, 10^-2, or 10^-3. We believe,
however, that such stringent TCP-compatibility is not required for
smaller loss rates, and that an appropriate response function is one
that gives a plausible packet drop rate for a connection throughput
of 10 Gbps. This also gives a slowly increasing number of round-trip
times between loss events as a function of a decreasing packet drop
rate.
Another way to look at the HighSpeed response function is to consider
that HighSpeed TCP is roughly emulating the congestion control
response of N parallel TCP connections, where N is initially one, and
where N increases as a function of the HighSpeed TCP’s congestion
window. Thus for the HighSpeed response function in Equation (1)
above, the response function can be viewed as equivalent to that of
N(W) parallel TCP connections, where N(W) varies as a function of the
congestion window W. Recall that for a single standard TCP
connection, the average congestion window equals 1.2/sqrt(p). For N
parallel TCP connections, the aggregate congestion window for the N
connections equals N*1.2/sqrt(p). From the HighSpeed response
function in Equation (1) and the relationship above, we can derive
the following:
N(W) = 0.23*W^(0.4)
for N(W) the number of parallel TCP connections emulated by the
HighSpeed TCP response function, and for N(W) >= 1. This is shown in
Table 4 below.
Congestion Window W Number N(W) of Parallel TCPs
------------------- -------------------------
1 1
10 1
100 1.4
1,000 3.6
10,000 9.2
100,000 23.0
Table 4: Number N(W) of parallel TCP connections roughly emulated by
the HighSpeed TCP response function.
In this document, we do not attempt to seriously evaluate the
HighSpeed response function for congestion windows greater than
100,000 packets. We believe that we will learn more about the
requirements for sustaining the throughput of best-effort connections
in that range as we gain more experience with HighSpeed TCP with
congestion windows of thousands and tens of thousands of packets.
There also might be limitations to the per-connection throughput that
can be realistically achieved for best-effort traffic, in terms of
congestion window of hundreds of thousands of packets or more, in the
absence of additional support or feedback from the routers along the
path.
6. Fairness Implications of the HighSpeed Response Function
The Standard and Highspeed Response Functions can be used directly to
infer the relative fairness between flows using the two response
functions. For example, given a packet drop rate P, assume that
Standard TCP has an average congestion window of W_Standard, and
HighSpeed TCP has a higher average congestion window of W_HighSpeed.
In this case, a single HighSpeed TCP connection is receiving