grants a capability to the DDP peer, and the socket (using the
underlying properties of its chosen transport and possible security)
identifies the peer and authenticates the stag_t.
The same buffer may be enabled by ddp_post_recv() on multiple
sockets. In this case any ddp_recv() untagged message reception
indication may be provided on a different socket from that on which
the buffer was posted. Such indications are not ordered among
multiple DDP sockets.
When multiple sockets reference an untagged message reception buffer,
local interfaces are responsible for managing the mechanisms of
allocating posted buffers to received untagged messages, the handling
of received untagged messages when no buffer is available, and of
resource management among multiple sockets. Where underprovisioning
of buffers on multiple sockets is allowed, mechanisms should be
provided to manage buffer consumption on a per-socket or group of
related sockets basis.
Architecturally, therefore, DDP is a flexible and general paradigm
that may be applied to any variety of transports. Implementations of
DDP may, however, adapt themselves to these differences in ways
appropriate to each transport. In all cases, the layering of DDP
must continue to express the transport’s underlying characteristics.
2.2. Remote Direct Memory Access (RDMA) Protocol Architecture
Remote Direct Memory Access (RDMA) extends the capabilities of DDP
with two primary functions.
First, it adds the ability to read from buffers registered to a
socket (RDMA Read). This allows a client protocol to perform
arbitrary, bidirectional data movement without involving the remote
client. When RDMA is implemented in hardware, arbitrary data
movement can be performed without involving the remote host CPU at
all.
In addition, RDMA specifies a transport-independent untagged message
service (Send) with characteristics that are both very efficient to
implement in hardware, and convenient for client protocols.
The RDMA architecture is patterned after the traditional model for
device programming, where the client requests an operation using
Send-like actions (programmed I/O), the server performs the necessary
data transfers for the operation (DMA reads and writes), and notifies
the client of completion. The programmed I/O+DMA model efficiently
supports a high degree of concurrency and flexibility for both the
client and server, even when operations have a wide range of
intrinsic latencies.
RDMA is layered as a client protocol on top of DDP:
Client Protocol
| ^
Sends | | Send reception indications
RDMA Read Requests | | RDMA Read Completion indications
RDMA Writes | | RDMA Write Completion indications
v |
RDMA
| ^
untagged messages | | untagged message delivery
tagged messages | | tagged message delivery
v |
DDP+---> data placement
^
| transport messages
v
. . .
In addition to in-line data flow, read (get()) and update (set())
operations are performed on buffers registered with RDMA as a result
of RDMA Read Requests and RDMA Writes, respectively.
An RDMA `buffer’ extends a DDP buffer with a get() operation that
retrieves the value of the octet at address `a’:
typedef struct {
const address_t start;
const address_t end;
void set(address_t a, data_t v);
data_t get(address_t a);
} rdma_buffer_t;
2.2.1. RDMA Operations
The RDMA layer provides:
void rdma_send(socket_t s, message_t m);
void rdma_write(socket_t s, message_t m, ddp_addr_t d,
rdma_notify_t n);
void rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d);
void rdma_post_recv(socket_t s, bdesc_t b);
rdma_ind_t rdma_recv(socket_t s);
bdesc_t rdma_register(socket_t s, rdma_buffer_t b,
bmode_t mode);
void rdma_deregister(bhand_t bh);
msizes_t rdma_max_msizes(socket_t s);
Although, for clarity, these data transfer interfaces are
synchronous, rdma_read() and possibly rdma_send() (in the presence of
Send flow control) can require an arbitrary amount of time to
complete. To express the full concurrency and interleaving of RDMA
data transfer, these interfaces should also be reentrant. For
example, a client protocol may perform an rdma_send(), while an
rdma_read() operation is in progress.
rdma_notify_t
RDMA Write notification information, used to signal that the
message represents the final fragment of a multi-segmented RDMA
message:
typedef struct {
boolean_t notify;
rdma_write_id_t i;
} rdma_notify_t;
identical in function to ddp_notify_t, except that the type
rdma_write_id_t may not be equivalent to ddp_msg_id_t.
rdma_write_id_t (scalar)
an RDMA Write identifier.
rdma_ind_t
a Send message, or an RDMA error:
typedef union {
recv_message_t m;
rdma_err_t e;
} rdma_ind_t;
rdma_err_t
an RDMA protocol error indication. RDMA errors include buffer
addressing errors corresponding to ddp_err_ts, and buffer
protection violations (e.g., RDMA Writing a buffer only
registered for reading).
bmode_t
buffer registration mode (permissions). Any combination of
permitting RDMA Read (BMODE_READ) and RDMA Write (BMODE_WRITE)
operations.
rdma_send(socket_t s, message_t m)
send a message, delivering it to the next untagged RDMA buffer
at the remote peer.
rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n)
RDMA Write to remote buffer address d.
rdma_read(socket_t s, ddp_addr_t s, length_t l, ddp_addr_t d)
RDMA Read l octets from remote buffer address s to local buffer
address d.
rdma_post_recv(socket_t s, bdesc_t b)
post a registered buffer to accept a single Send message, to be
filled and returned in-order to a subsequent caller of
rdma_recv(). As with DDP, buffers may be enabled on multiple
sockets, in which case ordering guarantees are relaxed. Also as
with DDP, local interfaces must manage the mechanisms of
allocation and management of buffers posted to multiple sockets.
rdma_recv(socket_t s);
get the next received Send message, RDMA Write completion
identifier, or RDMA error.
rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode)
register a buffer for RDMA on a socket (for read access, write
access or both). As with DDP, the same buffer may be registered
multiple times on the same or different sockets, and different
buffers may refer to portions of the same underlying addressable
object.
rdma_deregister(bhand_t bh)
remove a registration from a buffer.
rdma_max_msizes(socket_t s)
get the current maximum Send (max_untagged) and RDMA Read or
Write (max_tagged) operations that will fit in a single
transport message. The values returned by rdma_max_msizes() are
closely related to the values returned by ddp_max_msizes(), but
may not be equal.
2.2.2. Transport Characteristics in RDMA
As with DDP, RDMA can be used on transports with a variety of
different characteristics that manifest themselves directly in the
service provided by RDMA. Also, as with DDP, the fundamental
characteristics of the transport will not be changed by the presence
of RDMA.
Like DDP, an RDMA protocol must specify how:
o set()s,
o get()s,
o Send messages, and
o RDMA Read completions
are ordered among themselves and how they relate to corresponding
operations on the remote peer(s). These relationships are likely to
be a function of the underlying transport characteristics.
There are some additional characteristics of RDMA that may translate
poorly to unreliable or multipoint transports due to attendant
complexities in managing endpoint state:
o Send flow control
o RDMA Read
These difficulties can be overcome by placing restrictions on the
service provided by RDMA. However, many RDMA clients, especially
those that separate data transfer and application logic concerns, are
likely to depend upon capabilities only provided by RDMA on a point-
to-point, reliable transport. In other words, many potential Upper
Layers, which might avail themselves of RDMA services, are naturally
already biased toward these transport classes.
3. Security Considerations
Fundamentally, the DDP and RDMA protocols themselves should not
introduce additional vulnerabilities. They are intermediate
protocols and so should not perform or require functions such as
authorization, which are the domain of Upper Layers. However, the
DDP and RDMA protocols should allow mapping by strict Upper Layers
that are not permissive of new vulnerabilities; DDP and RDMAP
implementations should be prohibited from `cutting corners’ that
create new vulnerabilities. Implementations must ensure that only
`supplied’ resources (i.e., buffers) can be manipulated by DDP or
RDMAP messages.
System integrity must be maintained in any RDMA solution. Mechanisms
must be specified to prevent RDMA or DDP operations from impairing
system integrity. For example, threats can include potential buffer
reuse or buffer overflow, and are not merely a security issue. Even
trusted peers must not be allowed to damage local integrity. Any DDP
and RDMA protocol must address the issue of giving end-systems and
applications the capabilities to offer protection from such
compromises.
Because a Steering Tag exports access to a buffer, one critical
aspect of security is the scope of this access. It must be possible
to individually control specific attributes of the access provided by
a Steering Tag on the endpoint (socket) on which it was registered,
including remote read access, remote write access, and others that
might be identified. DDP and RDMA specifications must provide both
implementation requirements relevant to this issue, and guidelines to
assist implementors in making the appropriate design decisions.
For example, it must not be possible for DDP to enable evasion of
buffer consistency checks at the recipient. The DDP and RDMA
specifications must allow the recipient to rely on its consistent
buffer contents by explicitly controlling peer access to buffer
regions at appropriate times.
The use of DDP and RDMA on a transport connection may interact with
any security mechanism, and vice-versa. For example, if the security
mechanism is implemented above the transport layer, the DDP and RDMA
headers may not be protected. Therefore, such a layering may be
inappropriate, depending on requirements.
3.1. Security Services
The following end-to-end security services protect DDP and RDMAP
operation streams:
o Authentication of the data source, to protect against peer
impersonation, stream hijacking, and man-in-the-middle attacks
exploiting capabilities offered by the RDMA implementation.
Peer connections that do not pass authentication and
authorization checks must not be permitted to begin processing
in RDMA mode with an inappropriate endpoint. Once associated,
peer accesses to buffer regions must be authenticated and made
subject to authorization checks in the context of the
association and endpoint (socket) on which they are to be
performed, prior to any transfer operation or data being
accessed. The RDMA protocols must ensure that these region
protections be under strict application control.
o Integrity, to protect against modification of the control
content and buffer content.
While integrity is of concern to any transport, it is
important for the DDP and RDMAP protocols that the RDMA
control information carried in each operation be protected, in
order to direct the payloads appropriately.
o Sequencing, to protect against replay attacks (a special case
of the above modifications).
o Confidentiality, to protect the stream from eavesdropping.
IPsec, operating to secure the connection on a packet-by-packet
basis, is a natural fit to securing RDMA placement, which operates in
conjunction with transport. Because RDMA enables an implementation
to avoid buffering, it is preferable to perform all applicable
security protection prior to processing of each segment by the
transport and RDMA layers. Such a layering enables the most
efficient secure RDMA implementation.
The TLS record protocol, on the other hand, is layered on top of
reliable transports and cannot provide such security assurance until
an entire record is available, which may require the buffering and/or
assembly of several distinct messages prior to TLS processing. This
defers RDMA processing and introduces overheads that RDMA is designed
to avoid. In addition, TLS length restrictions on records themselves
impose additional buffering and processing for long operations that
must span multiple records. TLS therefore is viewed as potentially a
less natural fit for protecting the RDMA protocols.
Any DDP and RDMAP specification must provide the means to satisfy the
above security service requirements.
IPsec is sufficient to provide the required security services to the
DDP and RDMAP protocols, while enabling efficient implementations.
3.2. Error Considerations
Resource issues leading to denial-of-service attacks, overwrites and
other concurrent operations, the ordering of completions as required
by the RDMA protocol, and the granularity of transfer are all within
the required scope of any security analysis of RDMA and DDP.
The RDMA operations require checking of what is essentially user
information, explicitly including addressing information and
operation type (read or write), and implicitly including protection
and attributes. The semantics associated with each class of error
resulting from possible failure of such checks must be clearly
defined, and the expected action to be taken by the protocols in each
case must be specified.
In some cases, this will result in a catastrophic error on the RDMA
association; however, in others, a local or remote error may be
signalled. Certain of these errors may require consideration of
abstract local semantics. The result of the error on the RDMA
association must be carefully specified so as to provide useful
behavior, while not constraining the implementation.
4. Acknowledgements
The authors wish to acknowledge the valuable contributions of Caitlin
Bestler, David Black, Jeff Mogul, and Allyn Romanow.
5. Informative References
[FCVI] ANSI Technical Committee T11, "Fibre Channel Standard
Virtual Interface Architecture Mapping", ANSI/NCITS 357-
2001, March 2001, available from
http://www.t11.org/t11/stat.nsf/fcproj.
[IB] InfiniBand Trade Association, "InfiniBand Architecture
Specification Volumes 1 and 2", Release 1.1, November 2002,
available from http://www.infinibandta.org/specs.
[MYR] VMEbus International Trade Association, "Myrinet on VME
Protocol Specification", ANSI/VITA 26-1998, August 1998,
available from http://www.myri.com/open-specs.
[ROM] Romanow, A., Mogul, J., Talpey, T., and S. Bailey, "Remote
Direct Memory Access (RDMA) over IP Problem Statement", RFC
4297, December 2005.
[SCTP] Stewart, R., Xie, Q., Morneault, K., Sharp, C.,
Schwarzbauer, H., Taylor, T., Rytina, I., Kalla, M., Zhang,
L., and V. Paxson, "Stream Control Transmission Protocol",
RFC 2960, October 2000.
[SDP] InfiniBand Trade Association, "Sockets Direct Protocol
v1.0", Annex A of InfiniBand Architecture Specification
Volume 1, Release 1.1, November 2002, available from
http://www.infinibandta.org/specs.
[SRVNET] R. Horst, "TNet: A reliable system area network", IEEE
Micro, pp. 37-45, February 1995.
[VI] D. Cameron and G. Regnier, "The Virtual Interface
Architecture", ISBN 0971288704, Intel Press, April 2002,
more info at http://www.intel.com/intelpress/via/.
Authors’ Addresses
Stephen Bailey
Sandburst Corporation
600 Federal Street
Andover, MA 01810 USA
USA
Phone: +1 978 689 1614
EMail: steph@sandburst.com
Tom Talpey
Network Appliance
1601 Trapelo Road
Waltham, MA 02451 USA
Phone: +1 781 768 5329
EMail: thomas.talpey@netapp.com
Full Copyright Statement
Copyright (C) The Internet Society (2005).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at ietf-
ipr@ietf.org.
Acknowledgement
Funding for the RFC Editor function is currently provided by the
Internet Society.