link-layer address option in IPv6 and the "hardware address" in
IPv4/ARP has the same format.
The format is as described below:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved | Queue Pair Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ +
| |
+ GID +
| |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5
a) Reserved Flags
These 8 bits are reserved for future use. These bits MUST be
set to zero on send and ignored on receive unless specified
differently in a future document.
b) QPN
Every unicast communication in IB architecture is directed to a
specific QP [IBTA]. This QP number is included in the link
description. All IP communication to the relevant IPoIB
interface MUST be directed to this QPN. In the case of IPv4
subnets, the Address Resolution Protocol (ARP) reply packets
are also directed to the same QPN.
The choice of the QPN for IP/ARP communication is up to the
implementation.
c) GID
This is one of the GIDs of the port associated with the IPoIB
interface [IBTA]. IB associates multiple GIDs with a port. It
is RECOMMENDED that the GID formed by the combination of the IB
subnet prefix and the port’s "Port GUID" [IBTA] be included in
the link-layer/hardware address.
9.1.2. Auxiliary Link Information
The rest of the parameters are determined as follows:
a) LID
The method of determining the peer’s LID is not defined in this
document. It is up to the implementation to use any of the
IBA-approved methods to determine the destination LID. One
such method is to use the GID determined during the address
resolution, to retrieve the associated LID from the IB routing
infrastructure or the Subnet Administrator (SA).
It is the responsibility of the administrator to ensure that
the IB subnet(s) have unicast connectivity between the IPoIB
nodes. The GID exchanged between two endpoints in a multicast
message (ARP/ND) does not guarantee the existence of a unicast
path between the two.
There may be multiple LIDs, and hence paths, between the
endpoints. The criteria for selection of the LIDs are beyond
the scope of this document.
b) Q_Key
The Q_Key received on joining the broadcast group MUST be used
for all IPoIB communication over the particular IPoIB link.
c) P_Key
The P_Key to be used in the IP subnet is not discovered but is
a configuration parameter.
d) SL
The method of determining the SL is not defined in this
document. The SL is determined by any of the IBA-approved
methods.
e) Path rate
The implementation must leverage IB methods to determine the
path rate as required.
9.2. Address Resolution in IPv4 Subnets
The ARP packet header is as defined in [ARP]. The hardware type is
set to 32 (decimal) as specified by IANA [IANA]. The rest of the
fields are used as per [ARP].
16 bits: hardware type
16 bits: protocol
8 bits: length of hardware address
8 bits: length of protocol address
16 bits: ARP operation
The remaining fields in the packet hold the sender/target hardware
and protocol addresses.
[ sender hardware address ]
[ sender protocol address ]
[ target hardware address ]
[ target protocol address ]
The hardware address included in the ARP packet will be as specified
in section 9.1.1 and depicted in figure 5.
The length of the hardware address used in ARP packet header
therefore is 20.
9.3. Address Resolution in IPv6 Subnets
The Source/Target Link-layer address option is used in Router
Solicit, Router advertisements, Redirect, Neighbor Solicitation, and
Neighbor Advertisement messages when such messages are transmitted on
InfiniBand networks.
The source/target address option is specified as follows:
Type:
Source Link-layer address 1
Target Link-layer address 2
Length: 3
Link-layer address:
The link-layer address is as specified in section 9.1.1 and
depicted in figure 5.
[DISC] specifies the length of source/target option in
number of 8-octets as indicated by a length of ’3’ above.
Since the IPoIB link-layer address is only 20 octets long,
two octets of zero MUST be prepended to fill the total
option length of 24 octets.
9.4. Cautionary Note on QPN Caching
The link-layer address for IPoIB includes the QPN, which might not be
constant across reboots or even across network interface resets.
Cached QPN entries, such as in static ARP entries or in Reverse
Address Resolution Protocol (RARP) servers, will only work if the
implementation(s) using these options ensure that the QPN associated
with an interface is invariant across reboots/network resets.
It is RECOMMENDED that implementations revalidate ARP caches
periodically due to the aforementioned QPN-induced volatility of
IPoIB link-layer addresses.
10. Sending and Receiving IP Multicast Packets
Multicast in InfiniBand differs in a number of ways from multicast in
ethernet. This adds some complexity to an IPoIB implementation when
supporting IP multicast over IB.
A) An IB multicast group must be explicitly created through the SA
before it can be used.
This implies that in order to send a packet destined for an IP
multicast address, the IPoIB implementation must check with the
SA on the outbound link first for a "MCMemberRecord" that
matches the MGID. If one does exist, the Multicast Local
Identifier (MLID) associated with the multicast group is used
as the Destination Local Identifier (DLID) for the packet.
Otherwise, it implies no member exists on the local link. If
the scope of the IP multicast group is beyond link-local, the
packet must be sent to the on-link routers through the use of
the all-router multicast group or the broadcast group. This is
to allow local routers to forward the packet to multicast
listeners on remote networks. The all-router multicast group
is preferred over the broadcast group for better efficiency.
If the all-router multicast group does not exist, the sender
can assume that there are no routers on the local link; hence
the packet can be safely dropped.
B) A multicast sender must join the target multicast group before
outgoing multicast messages from it can be successfully routed.
The "SendOnlyNonMember" join is different from the regular
"FullMember" join in two aspects. First, both types of joins
enable multicast packets to be routed FROM the local port, but
only the "FullMember" join causes multicast packets to be
routed TO the port. Second, the sender port of a
"SendOnlyNonMember" join will not be counted as a member of the
multicast group for purposes of group creation and deletion.
The following code snippet demonstrates the steps in a typical
implementation when processing an egress multicast packet.
if the egress port is already a "SendOnlyNonMember", or a
"FullMember"
=> send the packet
else if the target multicast group exists
=> do "SendOnlyNonMember" join
=> send the packet
else if scope > link-local AND the all-router multicast group exists
=> send the packet to all routers
else
=> drop the packet
Implementations should cache the information about the existence of
an IB multicast group, its MLID and other attributes. This is to
avoid expensive SA calls on every outgoing multicast packet. Senders
MUST subscribe to the multicast group create and delete traps in
order to monitor the status of specific IB multicast groups. For
example, multicast packets directed to the all-router multicast group
due to a lack of listener on the local subnet must be forwarded to
the right multicast group if the group is created later. This
happens when a listener shows up on the local subnet.
A node joining an IP multicast group must first construct an MGID
according to the rule described in section 4 above. Once the correct
MGID is calculated, the node must call the SA of the outbound link to
attempt a "FullMember" join of the IB multicast group corresponding
to the MGID. If the IB multicast group does not already exist, one
must be created first with the IPoIB link MTU. The MGID MUST use the
same P_Key, Q_Key, SL, MTU, and HopLimit as those used in the
broadcast-GID. The rest of attributes SHOULD follow the values used
in the broadcast-GID as well.
The join request will cause the local port to be added to the
multicast group. It also enables the SM to program IB switches and
routers with the new multicast information to ensure the correct
forwarding of multicast packets for the group.
When a node leaves an IP multicast group, it SHOULD make a
"FullMember" leave request to the SA. This gives the SM an
opportunity to update relevant forwarding information, to delete an
IB multicast group if the local port is the last FullMember to leave,
and to free up the MLID allocated for it. The specific algorithm is
implementation-dependent and is out of the scope of this document.
Note that for an IPoIB link that spans more than one IB subnet
connected by IB routers, an adequate multicast forwarding support at
the IB level is required for multicast packets to reach listeners on
a remote IB subnet. The specific mechanism for this is beyond the
scope of IPoIB.
11. IP Multicast Routing
IP multicast routing requires each interface over which the router is
operating to be configured to listen to all link-layer multicast
addresses generated by IP [IPMULT, IP6MLD]. For an Ethernet
interface, this is often achieved by turning on the promiscuous
multicast mode on the interface.
IBA does not provide any hardware support for promiscuous multicast
mode. Fortunately, a promiscuous multicast mode can be emulated in
the software running on a router through the following steps:
A) Obtain a list of all active IB multicast groups from the local
SA.
B) Make a "NonMember" join request to the SA for every group that
has a signature in its MGID matching the one for either IPv4 or
IPv6.
C) Subscribe to the IB multicast group creation events using a
wildcarded MGID so that the router can "NonMember" join all IB
multicast groups created subsequently for IPv4 or IPv6.
The "NonMember" join has the same effect as a "FullMember" join
except that the former will not be counted as a member of the
multicast group for purposes of group creation or deletion. That is,
when the last "FullMember" leaves a multicast group, the group can be
safely deleted by the SA without concerning any "NonMember" routers.
12. New Types of Vulnerability in IB Multicast
Many IB multicast functions are subject to failures due to a number
of possible resource constraints. These include the creation of IB
multicast groups, the join calls ("SendOnlyNonMember", "FullMember",
and "NonMember"), and the attaching of a QP to a multicast group.
In general, the occurrence of these failure conditions is highly
implementation-dependent, and is believed to be rare. Usually, a
failed multicast operation at the IB level can be propagated back to
the IP level, causing the original operation to fail and the
initiator of the operation to be notified. But some IB multicast
functions are not tied to any foreground operation, making their
failures hard to detect. For example, if an IP multicast router
attempts to "NonMember" join a newly created multicast group in the
local subnet, but the join call fails, packet forwarding for that
particular multicast group will likely fail silently, that is,
without the attention of local multicast senders. This type of
problem can add more vulnerability to the already unreliable IP
multicast operations.
Implementations SHOULD log error messages upon any failure from an IB
multicast operation. Network administrators should be aware of this
vulnerability, and preserve enough multicast resources at the points
where IP multicast will be used heavily. For example, HCAs with
ample multicast resources should be used at any IP multicast router.
13. Security Considerations
This document specifies IP transmission over a multicast network.
Any network of this kind is vulnerable to a sender claiming another’s
identity and forging traffic or eavesdropping. It is the
responsibility of the higher layers or applications to implement
suitable countermeasures if this is a problem.
Successful transmission of IP packets depends on the correct setup of
the IPoIB link, creation of the broadcast-GID, creation of the QP and
its attachment to the broadcast-GID, and the correct determination of
various link parameters such as the LID, service level, and path
rate. These operations, many of which involve interactions with the
SM/SA, MUST be protected by the underlying operating system. This is
to prevent malicious, non-privileged software from hijacking
important resources and configurations.
Controlled Q_Keys SHOULD be used in all transmissions. This is to
prevent non-privileged software from fabricating IP datagrams.
14. IANA Considerations
To support ARP over InfiniBand, a value for the Address Resolution
Parameter "Number Hardware Type (hrd)" is required. IANA has
assigned the number "32" to indicate InfiniBand [IANA_ARP].
Future uses of the reserved bits in the frame format (Figure 3) and
link-layer address (Figure 5) MUST be published as RFCs. This
document requires that the reserved bits be set to zero on send and
ignored on receive.
15. Acknowledgements
The authors would like to thank Bruce Beukema, David Brean, Dan
Cassiday, Aditya Dube, Yaron Haviv, Michael Krause, Thomas Narten,
Erik Nordmark, Greg Pfister, Jim Pinkerton, Renato Recio, Kevin
Reilly, Kanoj Sarcar, Satya Sharma, Madhu Talluri, and David L.
Stevens for their suggestions and many clarifications on the IBA
specification.
16. References
16.1. Normative References
[AARCH] Hinden, R. and S. Deering, "Internet Protocol Version 6
(IPv6) Addressing Architecture", RFC 3513, April 2003.
[ARP] Plummer, David C., "Ethernet Address Resolution
Protocol: Or converting network protocol addresses to
48.bit Ethernet address for transmission on Ethernet
hardware ", STD 37, RFC 826, November 1982.
[DISC] Narten, T., Nordmark, E., and W. Simpson, "Neighbor
Discovery for IP Version 6 (IPv6)", RFC 2461, December
1998.
[IANA] Internet Assigned Numbers Authority, URL
http://www.iana.org
[IANA_ARP] URL http://www.iana.org/assignments/arp-parameters
[IBTA] InfiniBand Architecture Specification, URL
http://www.infinibandta.org/specs
[RFC4392] Kashyap, V., "IP over InfiniBand (IPoIB) Architecture",
RFC 4392, April 2006.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
16.2. Informative References
[HOSTS] Braden, R., "Requirements for Internet Hosts -
Communication Layers", STD 3, RFC 1122, October 1989.
[IGMP3] Cain, B., Deering, S., Kouvelas, I., Fenner, B., and A.
Thyagarajan, "Internet Group Management Protocol,
Version 3", RFC 3376, October 2002.
[IP6MLD] Deering, S., Fenner, W., and B. Haberman, "Multicast
Listener Discovery (MLD) for IPv6", RFC 2710, October
1999.
[IPMULT] Deering, S., "Host extensions for IP multicasting", STD
5, RFC 1112, August 1989.
[IPV6] Deering, S. and R. Hinden, "Internet Protocol, Version 6
(IPv6) Specification", RFC 2460, December 1998.
Authors’ Addresses
H.K. Jerry Chu
17 Network Circle, UMPK17-201
Menlo Park, CA 94025
USA
Phone: +1 650 786 5146
EMail: jerry.chu@sun.com
Vivek Kashyap
15350, SW Koll Parkway
Beaverton, OR 97006
USA
Phone: +1 503 578 3422
EMail: vivk@us.ibm.com
Full Copyright Statement
Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Acknowledgement
Funding for the RFC Editor function is provided by the IETF
Administrative Support Activity (IASA).