Request for Comments: 4296 Sandburst
Category: Informational T. Talpey
NetApp
December 2005
The Architecture of Direct Data Placement (DDP)
and Remote Direct Memory Access (RDMA) on Internet Protocols
Status of This Memo
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2005).
Abstract
This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports. This architecture does
not necessarily reflect the proper way to implement such protocols,
but is, rather, a descriptive tool for defining and understanding the
protocols. DDP allows the efficient placement of data into buffers
designated by Upper Layer Protocols (e.g., RDMA). RDMA provides the
semantics to enable Remote Direct Memory Access between peers in a
way consistent with application requirements.
Table of Contents
1. Introduction ....................................................2
1.1. Terminology ................................................2
1.2. DDP and RDMA Protocols .....................................3
2. Architecture ....................................................4
2.1. Direct Data Placement (DDP) Protocol Architecture ..........4
2.1.1. Transport Operations ................................6
2.1.2. DDP Operations ......................................7
2.1.3. Transport Characteristics in DDP ...................10
2.2. Remote Direct Memory Access (RDMA) Protocol Architecture ..12
2.2.1. RDMA Operations ....................................14
2.2.2. Transport Characteristics in RDMA ..................16
3. Security Considerations ........................................17
3.1. Security Services .........................................18
3.2. Error Considerations ......................................19
4. Acknowledgements ...............................................19
5. Informative References .........................................20
1. Introduction
This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports. This architecture does
not necessarily reflect the proper way to implement such protocols,
but is, rather, a descriptive tool for defining and understanding the
protocols. This document uses C language notation as a shorthand to
describe the architectural elements of DDP and RDMA protocols. The
choice of C notation is not intended to describe concrete protocols
or programming interfaces.
The first part of the document describes the architecture of DDP
protocols, including what assumptions are made about the transports
on which DDP is built. The second part describes the architecture of
RDMA protocols layered on top of DDP.
1.1. Terminology
Before introducing the protocols, certain definitions will be useful
to guide discussion:
o Placement - writing to a data buffer.
o Operation - a protocol message, or sequence of messages, which
provide an architectural semantic, such as reading or writing of
a data buffer.
o Delivery - informing any Upper Layer or application that a
particular message is available for use. Therefore, delivery
may be viewed as the "control" signal associated with a unit of
data. Note that the order of delivery is defined more strictly
than it is for placement.
o Completion - informing any Upper Layer or application that a
particular operation has finished. A completion, for instance,
may require the delivery of several messages, or it may also
reflect that some local processing has finished.
o Data Sink - the peer on which any placement occurs.
o Data Source - the peer from which the placed data originates.
o Steering Tag - a "handle" used to identify the buffer that is
the target of placement. A "tagged" message is one that
references such a handle.
o RDMA Write - an Operation that places data from a local data
buffer to a remote data buffer specified by a Steering Tag.
o RDMA Read - an Operation that places data to a local data buffer
specified by a Steering Tag from a remote data buffer specified
by another Steering Tag.
o Send - an Operation that places data from a local data buffer to
a remote data buffer of the data sink’s choice. Therefore,
sends are "untagged".
1.2. DDP and RDMA Protocols
The goal of the DDP protocol is to allow the efficient placement of
data into buffers designated by protocols layered above DDP (e.g.,
RDMA). This is described in detail in [ROM]. Efficiency may be
characterized by the minimization of the number of transfers of the
data over the receiver’s system buses.
The goal of the RDMA protocol is to provide the semantics to enable
Remote Direct Memory Access between peers in a way consistent with
application requirements. The RDMA protocol provides facilities
immediately useful to existing and future networking, storage, and
other application protocols. [FCVI, IB, MYR, SDP, SRVNET, VI]
The DDP and RDMA protocols work together to achieve their respective
goals. DDP provides facilities to safely steer payloads to specific
buffers at the Data Sink. RDMA provides facilities to Upper Layers
for identifying these buffers, controlling the transfer of data
between peers’ buffers, supporting authorized bidirectional transfer
between buffers, and signalling completion. Upper Layer Protocols
that do not require the features of RDMA may be layered directly on
top of DDP.
The DDP and RDMA protocols are transport independent. The following
figure shows the relationship between RDMA, DDP, Upper Layer
Protocols, and Transport.
+--------------------------------------------------+
| Upper Layer Protocol |
+---------+------------+---------------------------+
| | | RDMA |
| | +---------------------------+
| | DDP |
| +----------------------------------------+
| Transport |
+--------------------------------------------------+
2. Architecture
The Architecture section is presented in two parts: Direct Data
Placement Protocol architecture and Remote Direct Memory Access
Protocol architecture.
2.1. Direct Data Placement (DDP) Protocol Architecture
The central idea of general-purpose DDP is that a data sender will
supplement the data it sends with placement information that allows
the receiver’s network interface to place the data directly at its
final destination without any copying. DDP can be used to steer
received data to its final destination, without requiring layer-
specific behavior for each different layer. Data sent with such DDP
information is said to be `tagged’.
The central components of the DDP architecture are the `buffer’,
which is an object with beginning and ending addresses, and a method
(set()), which sets the value of an octet at an address. In many
cases, a buffer corresponds directly to a portion of host user
memory. However, DDP does not depend on this; a buffer could be a
disk file, or anything else that can be viewed as an addressable
collection of octets. Abstractly, a buffer provides the interface:
typedef struct {
const address_t start;
const address_t end;
void set(address_t a, data_t v);
} ddp_buffer_t;
address_t
a reference to local memory
data_t
an octet data value.
The protocol layering and in-line data flow of DDP is:
DDP Client Protocol
(e.g., RDMA or Upper Layer Protocol)
| ^
untagged messages | | untagged message delivery
tagged messages | | tagged message delivery
v |
DDP+---> data placement
^
| transport messages
v
Transport
(e.g., SCTP, DCCP, framed TCP)
^
| IP datagrams
v
. . .
In addition to in-line data flow, the client protocol registers
buffers with DDP, and DDP performs buffer update (set()) operations
as a result of receiving tagged messages.
DDP messages may be split into multiple, smaller DDP messages, each
in a separate transport message. However, if the transport is
unreliable or unordered, messages split across transport messages may
or may not provide useful behavior, in the same way as splitting
arbitrary Upper Layer messages across unreliable or unordered
transport messages may or may not provide useful behavior. In other
words, the same considerations apply to building client protocols on
different types of transports with or without the use of DDP.
A DDP message split across transport messages looks like:
DDP message: Transport messages:
stag=s, offset=o, message 1:
notify=y, id=i |type=ddp |
message= |stag=s |
|aabbccddee|-------. |offset=o |
~ ... ~----. \ |notify=n |
|vvwwxxyyzz|-. \ \ |id=? |
| \ `--->|aabbccddee|
| \ ~ ... ~
| +----->|iijjkkllmm|
| |
+ | message 2:
\ | |type=ddp |
\ | |stag=s |
\ + |offset=o+n|
\ \ |notify=y |
\ \ |id=i |
\ `-->|nnooppqqrr|
\ ~ ... ~
`---->|vvwwxxyyzz|
Although this picture suggests that DDP information is carried in-
line with the message payload, components of the DDP information may
also be in transport-specific fields, or derived from transport-
specific control information if the transport permits.
2.1.1. Transport Operations
For the purposes of this architecture, the transport provides:
void xpt_send(socket_t s, message_t m);
message_t xpt_recv(socket_t s);
msize_t xpt_max_msize(socket_t s);
socket_t
a transport address, including IP addresses, ports and other
transport-specific identifiers.
message_t
a string of octets.
msize_t (scalar)
a message size.
xpt_send(socket_t s, message_t m)
send a transport message.
xpt_recv(socket_t s)
receive a transport message.
xpt_max_msize(socket_t s)
get the current maximum transport message size. Corresponds,
roughly, to the current path Maximum Transfer Unit (PMTU),
adjusted by underlying protocol overheads.
Real implementations of xpt_send() and xpt_recv() typically return
error indications, but that is not relevant to this architecture.
2.1.2. DDP Operations
The DDP layer provides:
void ddp_send(socket_t s, message_t m);
void ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d,
ddp_notify_t n);
void ddp_post_recv(socket_t s, bdesc_t b);
ddp_ind_t ddp_recv(socket_t s);
bdesc_t ddp_register(socket_t s, ddp_buffer_t b);
void ddp_deregister(bhand_t bh);
msizes_t ddp_max_msizes(socket_t s);
ddp_addr_t
the buffer address portion of a tagged message:
typedef struct {
stag_t stag;
address_t offset;
} ddp_addr_t;
stag_t (scalar)
a Steering Tag. A stag_t identifies the destination buffer for
tagged messages. stag_ts are generated when the buffer is
registered, communicated to the sender by some client protocol
convention and inserted in DDP messages. stag_t values in this
DDP architecture are assumed to be completely opaque to the
client protocol, and implementation-dependent. However,
particular implementations, such as DDP on a multicast transport
(see below), may provide the buffer holder some control in
selecting stag_ts.
ddp_notify_t
the notification portion of a DDP message, used to signal
that the message represents the final fragment of a
multi-segmented DDP message:
typedef struct {
boolean_t notify;
ddp_msg_id_t i;
} ddp_notify_t;
ddp_msg_id_t (scalar)
a DDP message identifier. msg_id_ts are chosen by the DDP
message receiver (buffer holder), communicated to the sender by
some client protocol convention and inserted in DDP messages.
Whether a message reception indication is requested for a DDP
message is a matter of client protocol convention. Unlike
stag_ts, the structure of msg_id_ts is opaque to DDP, and
therefore, it is completely in the hands of the client protocol.
bdesc_t
a description of a registered buffer:
typedef struct {
bhand_t bh;
ddp_addr_t a;
} bdesc_t;
`a.offset’ is the starting offset of the registered buffer,
which may have no relationship to the `start’ or `end’ addresses
of that buffer. However, particular implementations, such as
DDP on a multicast transport (see below), may allow some client
protocol control over the starting offset.
bhand_t
an opaque buffer handle used to deregister a buffer.
recv_message_t
a description of a completed untagged receive buffer:
typedef struct {
bdesc_t b;
length_t l;
} recv_message_t;
ddp_ind_t
an untagged message, a tagged message reception indication, or a
tagged message reception error:
typedef union {
recv_message_t m;
ddp_msg_id_t i;
ddp_err_t e;
} ddp_ind_t;
ddp_err_t
indicates an error while receiving a tagged message, typically
`offset’ out of bounds, or `stag’ is not registered to the
socket.
msizes_t
The maximum untagged and tagged messages that fit in a single
transport message:
typedef struct {
msize_t max_untagged;
msize_t max_tagged;
} msizes_t;
ddp_send(socket_t s, message_t m)
send an untagged message.
ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n)
send a tagged message to remote buffer address d.
ddp_post_recv(socket_t s, bdesc_t b)
post a registered buffer to accept a single received untagged
message. Each buffer is returned to the caller in a ddp_recv()
untagged message reception indication, in the order in which it
was posted. The same buffer may be enabled on multiple sockets;
receipt of an untagged message into the buffer from any of these
sockets unposts the buffer from all sockets.
ddp_recv(socket_t s)
get the next received untagged message, tagged message reception
indication, or tagged message error.
ddp_register(socket_t s, ddp_buffer_t b)
register a buffer for DDP on a socket. The same buffer may be
registered multiple times on the same or different sockets. The
same buffer registered on different sockets may result in a
common registration. Different buffers may also refer to
portions of the same underlying addressable object (buffer
aliasing).
ddp_deregister(bhand_t bh)
remove a registration from a buffer.
ddp_max_msizes(socket_t s)
get the current maximum untagged and tagged message sizes that
will fit in a single transport message.
2.1.3. Transport Characteristics in DDP
Certain characteristics of the transport on which DDP is mapped
determine the nature of the service provided to client protocols.
Fundamentally, the characteristics of the transport will not be
changed by the presence of DDP. The choice of transport is therefore
driven not by DDP, but by the requirements of the Upper Layer, and
employing the DDP service.
Specifically, transports are:
o reliable or unreliable,
o ordered or unordered,
o single source or multisource,
o single destination or multidestination (multicast or anycast).
Some transports support several combinations of these
characteristics. For example, SCTP [SCTP] is reliable, single
source, single destination (point-to-point) and supports both ordered
and unordered modes.
DDP messages carried by transport are framed for processing by the
receiver, and may be further protected for integrity or privacy in
accordance with the transport capabilities. DDP does not provide
such functions.
In general, transport characteristics equally affect transport and
DDP message delivery. However, there are several issues specific to
DDP messages.
A key component of DDP is how the following operations on the
receiving side are ordered among themselves, and how they relate to
corresponding operations on the sending side:
o set()s,
o untagged message reception indications, and
o tagged message reception indications.
These relationships depend upon the characteristics of the underlying
transport in a way that is defined by the DDP protocol. For example,
if the transport is unreliable and unordered, the DDP protocol might
specify that the client protocol is subject to the consequences of
transport messages being lost or duplicated, rather than requiring
that different characteristics be presented to the client protocol.
Buffer access must be implemented consistently across endpoint IP
addresses on transports allowing multiple IP addresses per endpoint,
for example, SCTP. In particular, the Steering Tag must be
consistently scoped and must address the same buffer across all IP
address associations belonging to the endpoint. Additionally,
operation ordering relationships across IP addresses within an
association (set(), get(), etc.) depend on the underlying transport.
If the above consistency relationships cannot be maintained by a
transport endpoint, then the endpoint is unsuitable for a DDP
connection.
Multidestination data delivery is a transport characteristic that may
require specific consideration in a DDP protocol. As mentioned
above, the basic DDP model assumes that buffer address values
returned by ddp_register() are opaque to the client protocol, and can
be implementation dependent. The most natural way to map DDP to a
multidestination transport is to require that all receivers produce
the same buffer address when registering a multidestination
destination buffer. Restriction of the DDP model to accommodate
multiple destinations involves engineering tradeoffs comparable to
those of providing non-DDP multidestination transport capability.
A registered buffer is identified within DDP by its stag_t, which in
turn is associated with a socket. Therefore, this registration