for the commercial RDMA-based adapters [KSZ95, DWB+93]. In software,
many implementations have successfully achieved zero-copy transmit,
but few have accomplished zero-copy receive. And those that have
done so make strict alignment and no-touch requirements on the
application, greatly reducing the portability and usefulness of the
implementation.
In contrast, experience has proven satisfactory with memory-to-memory
systems that permit RDMA; performance has been good and there have
not been system or networking difficulties. RDMA is a single
solution. Once implemented, it can be used with any OS and machine
architecture, and it does not need to be revised when either of these
are changed.
In early work, one goal of the software approaches was to show that
TCP could go faster with appropriate OS support [CJRS89, CFF+94].
While this goal was achieved, further investigation and experience
showed that, though possible to craft software solutions, specific
system optimizations have been complex, fragile, extremely
interdependent with other system parameters in complex ways, and
often of only marginal improvement [CFF+94, CGY01, Ch96, DAPP93,
KSZ95, PDZ99]. The network I/O system interacts with other aspects
of the Operating System such as machine architecture and file I/O,
and disk I/O [Br99, Ch96, DP93].
For example, the Solaris Zero-Copy TCP work [Ch96], which relies on
page remapping, shows that the results are highly interdependent with
other systems, such as the file system, and that the particular
optimizations are specific for particular architectures, meaning that
for each variation in architecture, optimizations must be re-crafted
[Ch96].
With RDMA, application I/O buffers are mapped directly, and the
authorized peer may access it without incurring additional processing
overhead. When RDMA is implemented in hardware, arbitrary data
movement can be performed without involving the host CPU at all.
A number of research projects and industry products have been based
on the memory-to-memory approach to copy avoidance. These include
U-Net [EBBV95], SHRIMP [BLA+94], Hamlyn [BJM+96], Infiniband [IB],
Winsock Direct [Pi01]. Several memory-to-memory systems have been
widely used and have generally been found to be robust, to have good
performance, and to be relatively simple to implement. These include
VI [VI], Myrinet [BCF+95], Quadrics [QUAD], Compaq/Tandem Servernet
[SRVNET]. Networks based on these memory-to-memory architectures
have been used widely in scientific applications and in data centers
for block storage, file system access, and transaction processing.
By exporting direct memory access "across the wire", applications may
direct the network stack to manage all data directly from application
buffers. A large and growing class that takes advantage of such
capabilities of applications has already emerged. It includes all
the major databases, as well as network protocols such as Sockets
Direct [SDP].
5.1. A Conceptual Framework: DDP and RDMA
An RDMA solution can be usefully viewed as being comprised of two
distinct components: "direct data placement (DDP)" and "remote direct
memory access (RDMA) semantics". They are distinct in purpose and
also in practice -- they may be implemented as separate protocols.
The more fundamental of the two is the direct data placement
facility. This is the means by which memory is exposed to the remote
peer in an appropriate fashion, and the means by which the peer may
access it, for instance, reading and writing.
The RDMA control functions are semantically layered atop direct data
placement. Included are operations that provide "control" features,
such as connection and termination, and the ordering of operations
and signaling their completions. A "send" facility is provided.
While the functions (and potentially protocols) are distinct,
historically both aspects taken together have been referred to as
"RDMA". The facilities of direct data placement are useful in and of
themselves, and may be employed by other upper layer protocols to
facilitate data transfer. Therefore, it is often useful to refer to
DDP as the data placement functionality and RDMA as the control
aspect.
[BT05] develops an architecture for DDP and RDMA atop the Internet
Protocol Suite, and is a companion document to this problem
statement.
6. Conclusions
This Problem Statement concludes that an IP-based, general solution
for reducing processing overhead in end-hosts is desirable.
It has shown that high overhead of the processing of network data
leads to end-host bottlenecks. These bottlenecks are in large part
attributable to the copying of data. The bus bandwidth of machines
has historically been limited, and the bandwidth of high-speed
interconnects taxes it heavily.
An architectural solution to alleviate these bottlenecks best
satisfies the issue. Further, the high speed of today’s
interconnects and the deployment of these hosts on Internet
Protocol-based networks leads to the desirability of layering such a
solution on the Internet Protocol Suite. The architecture described
in [BT05] is such a proposal.
7. Security Considerations
Solutions to the problem of reducing copying overhead in high
bandwidth transfers may introduce new security concerns. Any
proposed solution must be analyzed for security vulnerabilities and
any such vulnerabilities addressed. Potential security weaknesses --
due to resource issues that might lead to denial-of-service attacks,
overwrites and other concurrent operations, the ordering of
completions as required by the RDMA protocol, the granularity of
transfer, and any other identified vulnerabilities -- need to be
examined, described, and an adequate resolution to them found.
Layered atop Internet transport protocols, the RDMA protocols will
gain leverage from and must permit integration with Internet security
standards, such as IPsec and TLS [IPSEC, TLS]. However, there may be
implementation ramifications for certain security approaches with
respect to RDMA, due to its copy avoidance.
IPsec, operating to secure the connection on a packet-by-packet
basis, seems to be a natural fit to securing RDMA placement, which
operates in conjunction with transport. Because RDMA enables an
implementation to avoid buffering, it is preferable to perform all
applicable security protection prior to processing of each segment by
the transport and RDMA layers. Such a layering enables the most
efficient secure RDMA implementation.
The TLS record protocol, on the other hand, is layered on top of
reliable transports and cannot provide such security assurance until
an entire record is available, which may require the buffering and/or
assembly of several distinct messages prior to TLS processing. This
defers RDMA processing and introduces overheads that RDMA is designed
to avoid. Therefore, TLS is viewed as potentially a less natural fit
for protecting the RDMA protocols.
It is necessary to guarantee properties such as confidentiality,
integrity, and authentication on an RDMA communications channel.
However, these properties cannot defend against all attacks from
properly authenticated peers, which might be malicious, compromised,
or buggy. Therefore, the RDMA design must address protection against
such attacks. For example, an RDMA peer should not be able to read
or write memory regions without prior consent.
Further, it must not be possible to evade memory consistency checks
at the recipient. The RDMA design must allow the recipient to rely
on its consistent memory contents by explicitly controlling peer
access to memory regions at appropriate times.
Peer connections that do not pass authentication and authorization
checks by upper layers must not be permitted to begin processing in
RDMA mode with an inappropriate endpoint. Once associated, peer
accesses to memory regions must be authenticated and made subject to
authorization checks in the context of the association and connection
on which they are to be performed, prior to any transfer operation or
data being accessed.
The RDMA protocols must ensure that these region protections be under
strict application control. Remote access to local memory by a
network peer is particularly important in the Internet context, where
such access can be exported globally.
8. Terminology
This section contains general terminology definitions for this
document and for Remote Direct Memory Access in general.
Remote Direct Memory Access (RDMA)
A method of accessing memory on a remote system in which the
local system specifies the location of the data to be
transferred.
RDMA Protocol
A protocol that supports RDMA Operations to transfer data
between systems.
Fabric
The collection of links, switches, and routers that connect a
set of systems.
Storage Area Network (SAN)
A network where disks, tapes, and other storage devices are made
available to one or more end-systems via a fabric.
System Area Network
A network where clustered systems share services, such as
storage and interprocess communication, via a fabric.
Fibre Channel (FC)
An ANSI standard link layer with associated protocols, typically
used to implement Storage Area Networks. [FIBRE]
Virtual Interface Architecture (VI, VIA)
An RDMA interface definition developed by an industry group and
implemented with a variety of differing wire protocols. [VI]
Infiniband (IB)
An RDMA interface, protocol suite and link layer specification
defined by an industry trade association. [IB]
9. Acknowledgements
Jeff Chase generously provided many useful insights and information.
Thanks to Jim Pinkerton for many helpful discussions.
10. Informative References
[ATM] The ATM Forum, "Asynchronous Transfer Mode Physical Layer
Specification" af-phy-0015.000, etc. available from
http://www.atmforum.com/standards/approved.html.
[BCF+95] N. J. Boden, D. Cohen, R. E. Felderman, A. E. Kulawik, C.
L. Seitz, J. N. Seizovic, and W. Su. "Myrinet - A
gigabit-per-second local-area network", IEEE Micro,
February 1995.
[BJM+96] G. Buzzard, D. Jacobson, M. Mackey, S. Marovich, J.
Wilkes, "An implementation of the Hamlyn send-managed
interface architecture", in Proceedings of the Second
Symposium on Operating Systems Design and Implementation,
USENIX Assoc., October 1996.
[BLA+94] M. A. Blumrich, K. Li, R. Alpert, C. Dubnicki, E. W.
Felten, "A virtual memory mapped network interface for the
SHRIMP multicomputer", in Proceedings of the 21st Annual
Symposium on Computer Architecture, April 1994, pp. 142-
153.
[Br99] J. C. Brustoloni, "Interoperation of copy avoidance in
network and file I/O", Proceedings of IEEE Infocom, 1999,
pp. 534-542.
[BS96] J. C. Brustoloni, P. Steenkiste, "Effects of buffering
semantics on I/O performance", Proceedings OSDI’96,
USENIX, Seattle, WA October 1996, pp. 277-291.
[BT05] Bailey, S. and T. Talpey, "The Architecture of Direct Data
Placement (DDP) And Remote Direct Memory Access (RDMA) On
Internet Protocols", RFC 4296, December 2005.
[CFF+94] C-H Chang, D. Flower, J. Forecast, H. Gray, B. Hawe, A.
Nadkarni, K. K. Ramakrishnan, U. Shikarpur, K. Wilde,
"High-performance TCP/IP and UDP/IP networking in DEC
OSF/1 for Alpha AXP", Proceedings of the 3rd IEEE
Symposium on High Performance Distributed Computing,
August 1994, pp. 36-42.
[CGY01] J. S. Chase, A. J. Gallatin, and K. G. Yocum, "End system
optimizations for high-speed TCP", IEEE Communications
Magazine, Volume: 39, Issue: 4 , April 2001, pp 68-74.
http://www.cs.duke.edu/ari/publications/end-
system.{ps,pdf}.
[Ch96] H.K. Chu, "Zero-copy TCP in Solaris", Proc. of the USENIX
1996 Annual Technical Conference, San Diego, CA, January
1996.
[Ch02] Jeffrey Chase, Personal communication.
[CJRS89] D. D. Clark, V. Jacobson, J. Romkey, H. Salwen, "An
analysis of TCP processing overhead", IEEE Communications
Magazine, volume: 27, Issue: 6, June 1989, pp 23-29.
[CT90] D. D. Clark, D. Tennenhouse, "Architectural considerations
for a new generation of protocols", Proceedings of the ACM
SIGCOMM Conference, 1990.
[DAPP93] P. Druschel, M. B. Abbott, M. A. Pagels, L. L. Peterson,
"Network subsystem design", IEEE Network, July 1993, pp.
8-17.
[DP93] P. Druschel, L. L. Peterson, "Fbufs: a high-bandwidth
cross-domain transfer facility", Proceedings of the 14th
ACM Symposium of Operating Systems Principles, December
1993.
[DWB+93] C. Dalton, G. Watson, D. Banks, C. Calamvokis, A. Edwards,
J. Lumley, "Afterburner: architectural support for high-
performance protocols", Technical Report, HP Laboratories
Bristol, HPL-93-46, July 1993.
[EBBV95] T. von Eicken, A. Basu, V. Buch, and W. Vogels, "U-Net: A
user-level network interface for parallel and distributed
computing", Proc. of the 15th ACM Symposium on Operating
Systems Principles, Copper Mountain, Colorado, December
3-6, 1995.
[FDDI] International Standards Organization, "Fibre Distributed
Data Interface", ISO/IEC 9314, committee drafts available
from http://www.iso.org.
[FGM+99] Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext
Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.
[FIBRE] ANSI Technical Committee T10, "Fibre Channel Protocol
(FCP)" (and as revised and updated), ANSI X3.269:1996
[R2001], committee draft available from
http://www.t10.org/drafts.htm#FibreChannel
[HP97] J. L. Hennessy, D. A. Patterson, Computer Organization and
Design, 2nd Edition, San Francisco: Morgan Kaufmann
Publishers, 1997.
[IB] InfiniBand Trade Association, "InfiniBand Architecture
Specification, Volumes 1 and 2", Release 1.1, November
2002, available from http://www.infinibandta.org/specs.
[IPSEC] Kent, S. and R. Atkinson, "Security Architecture for the
Internet Protocol", RFC 2401, November 1998.
[KP96] J. Kay, J. Pasquale, "Profiling and reducing processing
overheads in TCP/IP", IEEE/ACM Transactions on Networking,
Vol 4, No. 6, pp.817-828, December 1996.
[KSZ95] K. Kleinpaste, P. Steenkiste, B. Zill, "Software support
for outboard buffering and checksumming", SIGCOMM’95.
[Ma02] K. Magoutis, "Design and Implementation of a Direct Access
File System (DAFS) Kernel Server for FreeBSD", in
Proceedings of USENIX BSDCon 2002 Conference, San
Francisco, CA, February 11-14, 2002.
[MAF+02] K. Magoutis, S. Addetia, A. Fedorova, M. I. Seltzer, J.
S. Chase, D. Gallatin, R. Kisley, R. Wickremesinghe, E.
Gabber, "Structure and Performance of the Direct Access
File System (DAFS)", in Proceedings of the 2002 USENIX
Annual Technical Conference, Monterey, CA, June 9-14,
2002.
[Mc95] J. D. McCalpin, "A Survey of memory bandwidth and machine
balance in current high performance computers", IEEE TCCA
Newsletter, December 1995.
[PAC+97] D. Patterson, T. Anderson, N. Cardwell, R. Fromm, K.
Keeton, C. Kozyrakis, R. Thomas, K. Yelick , "A case for
intelligient RAM: IRAM", IEEE Micro, April 1997.
[PDZ99] V. S. Pai, P. Druschel, W. Zwaenepoel, "IO-Lite: a unified
I/O buffering and caching system", Proc. of the 3rd
Symposium on Operating Systems Design and Implementation,
New Orleans, LA, February 1999.
[Pi01] J. Pinkerton, "Winsock Direct: The Value of System Area
Networks", May 2001, available from
http://www.microsoft.com/windows2000/techinfo/
howitworks/communications/winsock.asp.
[Po81] Postel, J., "Transmission Control Protocol", STD 7, RFC
793, September 1981.
[QUAD] Quadrics Ltd., Quadrics QSNet product information,
available from
http://www.quadrics.com/website/pages/02qsn.html.
[SDP] InfiniBand Trade Association, "Sockets Direct Protocol
v1.0", Annex A of InfiniBand Architecture Specification
Volume 1, Release 1.1, November 2002, available from
http://www.infinibandta.org/specs.
[SRVNET] R. Horst, "TNet: A reliable system area network", IEEE
Micro, pp. 37-45, February 1995.
[STREAM] J. D. McAlpin, The STREAM Benchmark Reference Information,
http://www.cs.virginia.edu/stream/.
[TK95] M. N. Thadani, Y. A. Khalidi, "An efficient zero-copy I/O
framework for UNIX", Technical Report, SMLI TR-95-39, May
1995.
[TLS] Dierks, T. and C. Allen, "The TLS Protocol Version 1.0",
RFC 2246, January 1999.
[VI] D. Cameron and G. Regnier, "The Virtual Interface
Architecture", ISBN 0971288704, Intel Press, April 2002,
more info at http://www.intel.com/intelpress/via/.
[Wa97] J. R. Walsh, "DART: Fast application-level networking via
data-copy avoidance", IEEE Network, July/August 1997, pp.
28-38.
Authors’ Addresses
Stephen Bailey
Sandburst Corporation
600 Federal Street
Andover, MA 01810 USA
Phone: +1 978 689 1614
EMail: steph@sandburst.com
Jeffrey C. Mogul
HP Labs
Hewlett-Packard Company
1501 Page Mill Road, MS 1117
Palo Alto, CA 94304 USA
Phone: +1 650 857 2206 (EMail preferred)
EMail: JeffMogul@acm.org
Allyn Romanow
Cisco Systems, Inc.
170 W. Tasman Drive
San Jose, CA 95134 USA
Phone: +1 408 525 8836
EMail: allyn@cisco.com
Tom Talpey
Network Appliance
1601 Trapelo Road
Waltham, MA 02451 USA
Phone: +1 781 768 5329
EMail: thomas.talpey@netapp.com
Full Copyright Statement
Copyright (C) The Internet Society (2005).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at ietf-
ipr@ietf.org.
Acknowledgement
Funding for the RFC Editor function is currently provided by the
Internet Society.