HighSpeed TCP connection to have some method for limiting bursts.
In this document, we do not specify TCP mechanisms for reducing the
short-term burstiness. One possible mechanism is to use some form of
rate-based pacing, and another possibility is to use maxburst, which
limits the number of packets that are sent in response to a single
acknowledgement. We would caution, however, against a permanent
reduction in the congestion window as a mechanism for limiting
short-term bursts. Such a mechanism has been deployed in some TCP
stacks, and our view would be that using permanent reductions of the
congestion window to reduce transient bursts would be a bad idea
[Fl03].
10.3. Other limitations on window size
The TCP header uses a 16-bit field to report the receive window size
to the sender. Unmodified, this allows a window size of at most
2**16 = 65K bytes. With window scaling, the maximum window size is
2**30 = 1073M bytes [RFC 1323]. Given 1500-byte packets, this allows
a window of up to 715,000 packets.
10.4. Implementation issues
One implementation issue that has been raised with HighSpeed TCP is
that with congestion windows of 4MB or more, the handling of
successive SACK packets after a packet is dropped becomes very time-
consuming at the TCP sender [S03]. Tom Kelly’s Scalable TCP includes
a "SACK Fast Path" patch that addresses this problem.
The issues addressed in the Web100 project, the Net100 project, and
related projects about the tuning necessary to achieve high bandwidth
data rates with TCP apply to HighSpeed TCP as well [Net100, Web100].
11. Deployment issues
11.1. Deployment issues of HighSpeed TCP
We do not claim that the HighSpeed TCP modification to TCP described
in this paper is an optimal transport protocol for high-bandwidth
environments. Based on our experiences with HighSpeed TCP in the NS
simulator [NS], on simulation studies [SA03], and on experimental
reports [ABLLS03,D02,CC03,F03], we believe that HighSpeed TCP
improves the performance of TCP in high-bandwidth environments, and
we are documenting it for the benefit of the IETF community. We
encourage the use of HighSpeed TCP, and of its underlying response
function, and we further encourage feedback about operational
experiences with this or related modifications.
We note that in environments typical of much of the current Internet,
HighSpeed TCP behaves exactly as does Standard TCP today. This is
the case any time the congestion window is less than 38 segments.
Bandwidth Avg Cwnd w (pkts) Increase a(w) Decrease b(w)
--------- ----------------- ------------- -------------
1.5 Mbps 12.5 1 0.50
10 Mbps 83 1 0.50
100 Mbps 833 6 0.35
1 Gbps 8333 26 0.22
10 Gbps 83333 70 0.10
Table 9: Performance of a HighSpeed TCP connection
To help calibrate, Table 9 considers a TCP connection with 1500-byte
packets, an RTT of 100 ms (including average queueing delay), and no
competing traffic, and shows the average congestion window if that
TCP connection had a pipe all to itself and fully used the link
bandwidth, for a range of bandwidths for the pipe. This assumes that
the TCP connection would use Table 12 in determining its increase and
decrease parameters. The first column of Table 9 gives the
bandwidth, and the second column gives the average congestion window
w needed to utilize that bandwidth. The third column shows the
increase a(w) in segments per RTT for window w. The fourth column
shows the decrease b(w) for that window w (where the TCP sender
decreases the congestion window from w to w(1-b(w)) segments after a
loss event). When a loss occurs we note that the actual congestion
window is likely to be greater than the average congestion window w
in column 2, so the decrease parameter used could be slightly smaller
than the one given in column 4 of Table 9.
Table 9 shows that a HighSpeed TCP over a 10 Mbps link behaves
exactly the same as a Standard TCP connection, even in the absence of
competing traffic. One can think of the congestion window staying
generally in the range of 55 to 110 segments, with the HighSpeed TCP
behavior being exactly the same as the behavior of Standard TCP. (If
the congestion window is ever 128 segments or more, then the
HighSpeed TCP increases by two segments per RTT instead of by one,
and uses a decrease parameter of 0.44 instead of 0.50.)
Table 9 shows that for a HighSpeed TCP connection over a 100 Mbps
link, with no competing traffic, HighSpeed TCP behaves roughly as
aggressively as six parallel TCP connections, increasing its
congestion window by roughly six segments per round-trip time, and
with a decrease parameter of roughly 1/3 (corresponding to decreasing
down to 2/3-rds of its old congestion window, rather than to half, in
response to a loss event).
For a Standard TCP connection in this environment, the congestion
window could be thought of as generally varying in the range of 550
to 1100 segments, with an average packet drop rate of 2.2 * 10^-6
(corresponding to a bit error rate of 1.8 * 10^-10), or equivalently,
roughly 55 seconds between congestion events. While a Standard TCP
connection could sustain such a low packet drop rate in a carefully
controlled environment with minimal competing traffic, we would
contend that in an uncontrolled best-effort environment with even a
small amount of competing traffic, the occasional congestion events
from smaller competing flows could easily be sufficient to prevent a
Standard TCP flow with no lower-speed bottlenecks from fully
utilizing the available bandwidth of the underutilized 100 Mbps link.
That is, we would contend that in the environment of 100 Mbps links
with a significant amount of available bandwidth, Standard TCP would
sometimes be unable to fully utilize the link bandwidth, and that
HighSpeed TCP would be an improvement in this regard. We would
further contend that in this environment, the behavior of HighSpeed
TCP is sufficiently close to that of Standard TCP that HighSpeed TCP
would be safe to deploy in the current Internet. We note that
HighSpeed TCP can only use high congestion windows if allowed by the
receiver’s advertised window size. As a result, even if HighSpeed
TCP was ubiquitously deployed in the Internet, the impact would be
limited to those TCP connections with an advertised window from the
receiver of 118 MSS or larger.
We do not believe that the deployment of HighSpeed TCP would serve as
a block to the possible deployment of alternate experimental
protocols for high-speed congestion control, such as Scalable TCP,
XCP [KHR02], or FAST TCP [JWL03]. In particular, we don’t expect
HighSpeed TCP to interact any more poorly with alternative
experimental proposals than would the N parallel TCP connections
commonly used today in the absence of HighSpeed TCP.
11.2. Deployment issues of Scalable TCP
We believe that Scalable TCP and HighSpeed TCP have sufficiently
similar response functions that they could easily coexist in the
Internet. However, we have not investigated Scalable TCP
sufficiently to be able to claim, in this document, that Scalable TCP
is safe for a widespread deployment in the current Internet.
Bandwidth Avg Cwnd w (pkts) Increase a(w) Decrease b(w)
--------- ----------------- ------------- -------------
1.5 Mbps 12.5 1 0.50
10 Mbps 83 0.4 0.125
100 Mbps 833 4.1 0.125
1 Gbps 8333 41.6 0.125
10 Gbps 83333 416.5 0.125
Table 10: Performance of a Scalable TCP connection.
Table 10 shows the performance of a Scalable TCP connection with
1500-byte packets, an RTT of 100 ms (including average queueing
delay), and no competing traffic. The TCP connection is assumed to
use delayed acknowledgements. The first column of Table 10 gives the
bandwidth, the second column gives the average congestion window
needed to utilize that bandwidth, and the third and fourth columns
give the increase and decrease parameters.
Note that even in an environment with a 10 Mbps link, Scalable TCP’s
behavior is considerably different from that of Standard TCP. The
increase parameter is smaller than that of Standard TCP, and the
decrease is smaller also, 1/8-th instead of 1/2. That is, for 10
Mbps links, Scalable TCP increases less aggressively than Standard
TCP or HighSpeed TCP, but decreases less aggressively as well.
In an environment with a 100 Mbps link, Scalable TCP has an increase
parameter of roughly four segments per round-trip time, with the same
decrease parameter of 1/8-th. A comparison of Tables 9 and 10 shows
that for this scenario of 100 Mbps links, HighSpeed TCP increases
more aggressively than Scalable TCP.
Next we consider the relative fairness between Standard TCP,
HighSpeed TCP and Scalable TCP. The relative fairness between
HighSpeed TCP and Standard TCP was shown in Table 5 earlier in this
document, and the relative fairness between Scalable TCP and Standard
TCP was shown in Table 8. Following the approach in Section 6, for a
given packet drop rate p, for p < 10^-3, we can estimate the relative
fairness between Scalable and HighSpeed TCP as
W_Scalable/W_HighSpeed. This relative fairness is shown in Table 11
below. The bandwidth in the last column of Table 11 is the aggregate
bandwidth of the two competing flows given 100 ms round-trip times
and 1500-byte packets.
Packet Drop Rate P Fairness Aggregate Window Bandwidth
------------------ -------- ---------------- ---------
10^-2 1.0 24 2.8 Mbps
10^-3 1.0 76 9.1 Mbps
10^-4 1.4 643 77.1 Mbps
10^-5 2.1 5595 671.4 Mbps
10^-6 3.1 50279 6.0 Gbps
10^-7 4.5 463981 55.7 Gbps
Table 11: Relative Fairness between the Scalable and HighSpeed
Response Functions.
The second row of Table 11 shows that for a Scalable TCP and a
HighSpeed TCP flow competing in an environment with 100 ms RTTs and a
10 Mbps pipe, the two flows would receive essentially the same
bandwidth. The next row shows that for a Scalable TCP and a
HighSpeed TCP flow competing in an environment with 100 ms RTTs and a
100 Mbps pipe, the Scalable TCP flow would receive roughly 50% more
bandwidth than would HighSpeed TCP. Table 11 shows the relative
fairness in higher-bandwidth environments as well. This relative
fairness seems sufficient that there should be no problems with
Scalable TCP and HighSpeed TCP coexisting in the same environment as
Experimental variants of TCP.
We note that one question that requires more investigation with
Scalable TCP is that of convergence to fairness in environments with
Drop-Tail queue management.
12. Related Work in HighSpeed TCP
HighSpeed TCP has been separately investigated in simulations by
Sylvia Ratnasamy and by Evandro de Souza [SA03]. The simulations in
[SA03] verify the fairness properties of HighSpeed TCP when sharing a
link with Standard TCP.
These simulations explore the relative fairness of HighSpeed TCP
flows when competing with Standard TCP. The simulation environment
includes background forward and reverse-path TCP traffic limited by
the TCP receive window, along with a small amount of forward and
reverse-path traffic from the web traffic generator. Most of the
simulations so far explore performance on a simple dumbbell topology
with a 1 Gbps link with a propagation delay of 50 ms. Simulations
have been run with Adaptive RED and with DropTail queue management.
The simulations in [SA03] explore performance with a varying number
of competing flows, with the competing traffic being all standard
TCP; all HighSpeed TCP; or a mix of standard and HighSpeed TCP. For
the simulations in [SA03] with RED queue management, the relative
fairness between standard and HighSpeed TCP is consistent with the
relative fairness predicted in Table 5. For the simulations with
Drop Tail queues, the relative fairness is more skewed, with the
HighSpeed TCP flows receiving an even larger share of the link
bandwidth. This is not surprising; with Active Queue Management at
the congested link, the fraction of packet drops received by each
flow should be roughly proportional to that flow’s share of the link
bandwidth, while this property no longer holds with Drop Tail queue
management. We also note that relative fairness in simulations with
Drop Tail queue management can sometimes depend on small details of
the simulation scenario, and that Drop Tail simulations need special
care to avoid phase effects [F92].
[SA03] explores the bandwidth `stolen’ by HighSpeed TCP from standard
TCP by exploring the fraction of the link bandwidth N standard TCP
flows receive when competing against N other standard TCP flows, and
comparing this to the fraction of the link bandwidth the N standard
TCP flows receive when competing against N HighSpeed TCP flows. For
the 1 Gbps simulation scenarios dominated by long-lived traffic, a
small number of standard TCP flows are able to achieve high link
utilization, and the HighSpeed TCP flows can be viewed as stealing
bandwidth from the competing standard TCP flows, as predicted in
Section 6 on the Fairness Implications of the HighSpeed Response
Function. However, [SA03] shows that when even a small fraction of
the link bandwidth is used by more bursty, short TCP connections, the
standard TCP flows are unable to achieve high link utilization, and
the HighSpeed TCP flows in this case are not `stealing’ bandwidth
from the standard TCP flows, but instead are using bandwidth that
otherwise would not be utilized.
The conclusions of [SA03] are that "HighSpeed TCP behaved as forseen
by its response function, and appears to be a real and viable option
for use on high-speed wide area TCP connections."
Future work that could be explored in more detail includes
convergence times after new flows start-up; recovery time after a
transient outage; the response to sudden severe congestion, and
investigations of the potential for oscillations. We invite
contributions from others in this work.
13. Relationship to other Work
Our assumption is that HighSpeed TCP will be used with the TCP SACK
option, and also with the increased Initial Window of three or four
segments, as allowed by [RFC3390]. For paths that have substantial
reordering, TCP performance would be greatly improved by some of the
mechanisms still in the research stages for robust performance in the
presence of reordered packets.
Our view is that HighSpeed TCP is largely orthogonal to proposals for
higher PMTU (Path MTU) values [M02]. Unlike changes to the PMTU,
HighSpeed TCP does not require any changes in the network or at the
TCP receiver, and works well in the current Internet. Our assumption
is that HighSpeed TCP would be useful even with larger values for the
PMTU. Unlike the current congestion window, the PMTU gives no
information about the bandwidth-delay product available to that
particular flow.
A related approach is that of a virtual MTU, where the actual MTU of
the path might be limited [VMSS,S02]. The virtual MTU approach has
not been fully investigated, and we do not explore the virtual MTU
approach further in this document.
14. Conclusions
This document has proposed HighSpeed TCP, a modification to TCP’s
congestion control mechanism for use with TCP connections with large
congestion windows. We have explored this proposal in simulations,
and others have explored HighSpeed TCP with experiments, and we
believe HighSpeed TCP to be safe to deploy on the current Internet.
We would welcome additional analysis, simulations, and particularly,
experimentation. More information on simulations and experiments is
available from the HighSpeed TCP Web Page [HSTCP]. There are several
independent implementations of HighSpeed TCP [D02,F03] and of
Scalable TCP [K03] for further investigation.
15. Acknowledgements
The HighSpeed TCP proposal is from joint work with Sylvia Ratnasamy
and Scott Shenker (and was initiated by Scott Shenker). Additional
investigations of HighSpeed TCP were joint work with Evandro de Souza
and Deb Agarwal. We thank Tom Dunigan for the implementation in the
Linux 2.4.16 Web100 kernel, and for resulting experimentation with
HighSpeed TCP. We are grateful to the End-to-End Research Group, the
members of the Transport Area Working Group, and to members of the
IPAM program in Large Scale Communication Networks for feedback. We
thank Glenn Vinnicombe for framing the Linear response function in
the parameters of HighSpeed TCP. We are also grateful for
contributions and feedback from the following individuals: Les
Cottrell, Mitchell Erblich, Jeffrey Hsu, Tom Kelly, Chuck Jackson,
Matt Mathis, Jitendra Padhye, Andrew Reiter, Stanislav Shalunov, Alex
Solan, Paul Sutter, Brian Tierney, Joe Touch.
16. Normative References
[RFC2581] Allman, M., Paxson, V. and W. Stevens, "TCP Congestion
Control", RFC 2581, April 1999.
17. Informative References
[ABLLS03] A. Antony, J. Blom, C. de Laat, J. Lee, and W. Sjouw,
"Microscopic Examination of TCP Flows over Transatlantic
Links", iGrid2002 special issue, Future Generation
Computer Systems, volume 19 issue 6 (2003), URL
"http://www.science.uva.nl/~delaat/techrep-2003-2-
tcp.pdf".
[BBFS01] Deepak Bansal, Hari Balakrishnan, Sally Floyd, and Scott
Shenker, "Dynamic Behavior of Slowly-Responsive Congestion
Control Algorithms", SIGCOMM 2001, August 2001.
[CC03] Fabrizio Coccetti and Les Cottrell, "TCP Stack
Measurements on Lightly Loaded Testbeds", 2003. URL
"http://www-iepm.slac.stanford.edu/monitoring/bulk/fast/".
[CJ89] D. Chiu and R. Jain, "Analysis of the Increase and
Decrease Algorithms for Congestion Avoidance in Computer
Networks", Computer Networks and ISDN Systems, Vol. 17,
pp. 1-14, 1989.
[CO98] J. Crowcroft and P. Oechslin, "Differentiated End-to-end
Services using a Weighted Proportional Fair Share TCP",
Computer Communication Review, 28(3):53--69, 1998.
[D02] Tom Dunigan, "Floyd’s TCP slow-start and AIMD mods", URL
"http://www.csm.ornl.gov/~dunigan/net100/floyd.html".
[F03] Gareth Fairey, "High-Speed TCP", 2003. URL
"http://www.hep.man.ac.uk/u/garethf/hstcp/".
[F92] S. Floyd and V. Jacobson, "On Traffic Phase Effects in
Packet-Switched Gateways, Internetworking: Research and
Experience", V.3 N.3, September 1992, p.115-156. URL
"http://www.icir.org/floyd/papers.html".
[Fl03] Sally Floyd, "Re: [Tsvwg] taking NewReno (RFC 2582) to
Proposed Standard", Email to the tsvwg mailing list, May
14, 2003.
URLs "http://www1.ietf.org/mail-archive/working-
groups/tsvwg/current/msg04086.html" and
"http://www1.ietf.org/mail-archive/working-
groups/tsvwg/current/msg04087.html".
[FF98] Floyd, S., and Fall, K., "Promoting the Use of End-to-End
Congestion Control in the Internet", IEEE/ACM Transactions
on Networking, August 1999.
[FRS02] Sally Floyd, Sylvia Ratnasamy, and Scott Shenker,
"Modifying TCP’s Congestion Control for High Speeds", May
2002. URL "http://www.icir.org/floyd/notes.html".
[GRK99] Panos Gevros, Fulvio Risso and Peter Kirstein, "Analysis
of a Method for Differential TCP Service". In Proceedings
of the IEEE GLOBECOM’99, Symposium on Global Internet ,
December 1999, Rio de Janeiro, Brazil.
[GV02] S. Gorinsky and H. Vin, "Extended Analysis of Binary
Adjustment Algorithms", Technical Report TR2002-39,
Department of Computer Sciences, The University of Texas
at Austin, August 2002. URL
"http://www.cs.utexas.edu/users/gorinsky/pubs.html".
[HSTCP] HighSpeed TCP Web Page, URL
"http://www.icir.org/floyd/hstcp.html".
[J02] Amit Jain and Sally Floyd, "Quick-Start for TCP and IP",
Work in Progress, 2002.
[JWL03] Cheng Jin, David X. Wei and Steven H. Low, "FAST TCP for
High-speed Long-distance Networks", Work in Progress, June
2003.
[K03] Tom Kelly, "Scalable TCP: Improving Performance in
HighSpeed Wide Area Networks", February 2003. URL
"http://www-lce.eng.cam.ac.uk/~ctk21/scalable/".
[KHR02] Dina Katabi, Mark Handley, and Charlie Rohrs, "Congestion
Control for High Bandwidth-Delay Product Networks",
SIGCOMM 2002.
[M02] Matt Mathis, "Raising the Internet MTU", Web Page, URL
"http://www.psc.edu/~mathis/MTU/".
[Net100] The DOE/MICS Net100 project. URL
"http://www.csm.ornl.gov/~dunigan/net100/".
[NS] The NS Simulator, "http://www.isi.edu/nsnam/ns/".
[RFC 1323] Jacobson, V., Braden, R. and D. Borman, "TCP Extensions
for High Performance", RFC 1323, May 1992.
[RFC3390] Allman, M., Floyd, S. and C., Partridge, "Increasing TCP’s
Initial Window", RFC 3390, October 2002.
[RFC3448] Handley, M., Padhye, J., Floyd, S. and J. Widmer, "TCP
Friendly Rate Control (TFRC): Protocol Specification", RFC
3448, January 2003.
[SA03] Souza, E. and D.A., Agarwal, "A HighSpeed TCP Study:
Characteristics and Deployment Issues", LBNL Technical
Report LBNL-53215. URL
"http://www.icir.org/floyd/hstcp.html".
[S02] Stanislav Shalunov, "TCP Armonk", Work in Progress, 2002,
URL "http://www.internet2.edu/~shalunov/tcpar/".
[S03] Alex Solan, private communication, 2003.
[VMSS] "Web100 at ORNL", Web Page,