Request for Comments: 3821 E. Rodriguez
Category: Standards Track R. Weber
July 2004
Fibre Channel Over TCP/IP (FCIP)
Status of this Memo
This document specifies an Internet standards track protocol for the
Internet community, and requests discussion and suggestions for
improvements. Please refer to the current edition of the "Internet
Official Protocol Standards" (STD 1) for the standardization state
and status of this protocol. Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2004).
Abstract
Fibre Channel Over TCP/IP (FCIP) describes mechanisms that allow the
interconnection of islands of Fibre Channel storage area networks
over IP-based networks to form a unified storage area network in a
single Fibre Channel fabric. FCIP relies on IP-based network
services to provide the connectivity between the storage area network
islands over local area networks, metropolitan area networks, or wide
area networks.
Table Of Contents
1. Purpose, Motivation, and Objectives. . . . . . . . . . . . . . 3
2. Relationship to Fibre Channel Standards. . . . . . . . . . . . 4
2.1. Relevant Fibre Channel Standards . . . . . . . . . . . . 4
2.2. This Specification and Fibre Channel Standards . . . . . 5
3. Terminology. . . . . . . . . . . . . . . . . . . . . . . . . . 5
4. Protocol Summary . . . . . . . . . . . . . . . . . . . . . . . 7
5. The FCIP Model . . . . . . . . . . . . . . . . . . . . . . . . 9
5.1. FCIP Protocol Model. . . . . . . . . . . . . . . . . . . 9
5.2. FCIP Link. . . . . . . . . . . . . . . . . . . . . . . . 10
5.3. FC Entity. . . . . . . . . . . . . . . . . . . . . . . . 11
5.4. FCIP Entity. . . . . . . . . . . . . . . . . . . . . . . 12
5.5. FCIP Link Endpoint (FCIP_LEP). . . . . . . . . . . . . . 13
5.6. FCIP Data Engine (FCIP_DE) . . . . . . . . . . . . . . . 14
5.6.1. FCIP Encapsulation of FC Frames. . . . . . . . . 16
5.6.2. FCIP Data Engine Error Detection and Recovery. . 19
6. Checking FC Frame Transit Times in the IP Network. . . . . . . 22
7. The FCIP Special Frame (FSF) . . . . . . . . . . . . . . . . . 23
7.1. FCIP Special Frame Format. . . . . . . . . . . . . . . . 23
7.2. Overview of FSF Usage in Connection Establishment. . . . 26
8. TCP Connection Management. . . . . . . . . . . . . . . . . . . 28
8.1. TCP Connection Establishment . . . . . . . . . . . . . . 28
8.1.1. Connection Establishment Model . . . . . . . . . 28
8.1.2. Creating New TCP Connections . . . . . . . . . . 29
8.1.3. Processing Incoming TCP Connect Requests . . . . 32
8.1.4. Simultaneous Connection Establishment. . . . . . 36
8.2. Closing TCP Connections. . . . . . . . . . . . . . . . . 36
8.3. TCP Connection Parameters. . . . . . . . . . . . . . . . 36
8.3.1. TCP Selective Acknowledgement Option . . . . . . 36
8.3.2. TCP Window Scale Option. . . . . . . . . . . . . 36
8.3.3. Protection Against Sequence Number Wrap. . . . . 37
8.3.4. TCP_NODELAY Option . . . . . . . . . . . . . . . 37
8.4. TCP Connection Considerations. . . . . . . . . . . . . . 37
8.5. Flow Control Mapping between TCP and FC. . . . . . . . . 37
9. Security . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
9.1. Threat Models. . . . . . . . . . . . . . . . . . . . . . 38
9.2. FC Fabric and IP Network Deployment Models . . . . . . . 40
9.3. FCIP Security Components . . . . . . . . . . . . . . . . 40
9.3.1. IPsec ESP Authentication and Confidentiality . . 40
9.3.2. Key Management . . . . . . . . . . . . . . . . . 41
9.3.3. ESP Replay Protection and Rekeying Issues. . . . 43
9.4. Secure FCIP Link Operation . . . . . . . . . . . . . . . 44
9.4.1. FCIP Link Initialization Steps . . . . . . . . . 44
9.4.2. TCP Connection Security Associations (SAs) . . . 44
9.4.3. Handling Data Integrity and Confidentiality
Violations . . . . . . . . . . . . . . . . . . . 45
10. Performance. . . . . . . . . . . . . . . . . . . . . . . . . . 45
10.1. Performance Considerations . . . . . . . . . . . . . . . 45
10.2. IP Quality of Service (QoS) Support. . . . . . . . . . . 46
11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 47
11.1. Normative References . . . . . . . . . . . . . . . . . . 47
11.2. Informative References . . . . . . . . . . . . . . . . . 49
12. Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . 50
Appendix A Fibre Channel Bit and Byte Numbering Guidance. . . . . 51
B IANA Considerations. . . . . . . . . . . . . . . . . . 51
C FCIP Usage of Addresses and Identifiers. . . . . . . . 52
D Example of synchronization Recovery Algorithm. . . . . 53
E Relationship between FCIP and IP over FC (IPFC). . . . 58
F FC Frame Format. . . . . . . . . . . . . . . . . . . . 59
G FC Encapsulation Format. . . . . . . . . . . . . . . . 61
H FCIP Requirements on an FC Entity. . . . . . . . . . . 63
Editors and Contributors Acknowledgements. . . . . . . . . . . . . 69
Editors and Contributors Addresses . . . . . . . . . . . . . . . . 70
Full Copyright Statement . . . . . . . . . . . . . . . . . . . . . 74
1. Purpose, Motivation, and Objectives
Warning to Readers Familiar With Fibre Channel: Both Fibre Channel
and IETF standards use the same byte transmission order. However,
the bit and byte numbering is different. See appendix A for
guidance.
Fibre Channel (FC) is a gigabit or multi-gigabit speed networking
technology primarily used to implement Storage Area Networks (SANs).
See section 2 for information about how Fibre Channel is standardized
and the relationship of this specification to Fibre Channel
standards. An overview of Fibre Channel can be found in [34].
This specification describes mechanisms that allow the
interconnection of islands of Fibre Channel SANs over IP Networks to
form a unified SAN in a single Fibre Channel fabric. The motivation
behind defining these interconnection mechanisms is a desire to
connect physically remote FC sites allowing remote disk access, tape
backup, and live mirroring.
Fibre Channel standards have chosen nominal distances between switch
elements that are less than the distances available in an IP Network.
Since Fibre Channel and IP Networking technologies are compatible, it
is logical to turn to IP Networking for extending the allowable
distances between Fibre Channel switch elements.
The fundamental assumption made in this specification is that the
Fibre Channel traffic is carried over the IP Network in such a manner
that the Fibre Channel Fabric and all Fibre Channel devices on the
Fabric are unaware of the presence of the IP Network. This means
that the FC datagrams must be delivered in such time as to comply
with existing Fibre Channel specifications. The FC traffic may span
LANs, MANs, and WANs, so long as this fundamental assumption is
adhered to.
The objectives of this document are to:
1) specify the encapsulation and mapping of Fibre Channel (FC) frames
employing FC Frame Encapsulation [19].
2) apply the mechanism described in 1) to an FC Fabric using an IP
network as an interconnect between two or more islands in an FC
Fabric.
3) address any FC concerns arising from tunneling FC traffic over an
IP-based network, including security, data integrity (loss),
congestion, and performance. This will be accomplished by
utilizing the existing IETF-specified suite of protocols.
4) be compatible with the referenced FC standards. While new work
may be undertaken in T11 to optimize and enhance FC Fabrics, this
specification REQUIRES conformance only to the referenced FC
standards.
5) be compatible with all applicable IETF standards so that the IP
Network used to extend an FC Fabric can be used concurrently for
other reasonable purposes.
The objectives of this document do not include using an IP Network as
a replacement for the Fibre Channel Arbitrated Loop interconnect. No
definition is provided for encapsulating loop primitive signals for
transmission over an IP Network.
Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119 [1].
2. Relationship to Fibre Channel Standards
2.1. Relevant Fibre Channel Standards
FC is standardized as a family of American National Standards
developed by the T11 technical committee of INCITS (InterNational
Committee for Information Technology Standards). T11 has specified a
number of documents describing FC protocols, operations, and
services. T11 documents of interest to readers of this specification
include (but are not limited to):
- FC-BB - Fibre Channel Backbone [2]
- FC-BB-2 - Fibre Channel Backbone -2 [3]
- FC-SW-2 - Fibre Channel Switch Fabric -2 [4]
- FC-FS - Fibre Channel Framing and Signaling [5]
FC-BB and FC-BB-2 describe the relationship between an FC Fabric and
interconnect technologies not defined by Fibre Channel standards
(e.g., ATM and SONET). FC-BB-2 is the Fibre Channel document
describing the relationships between FC and TCP/IP, including the FC
use of FCIP.
FC-SW-2 describes the switch components of an FC Fabric and FC-FS
describes the FC Frame format and basic control features of Fibre
Channel.
Additional information regarding T11 activities is available on the
committee’s web site www.t11.org.
2.2. This Specification and Fibre Channel Standards
When considering the challenge of transporting FC Frames over an IP
Network, it is logical to divide the standardization effort between
TCP/IP requirements and Fibre Channel requirements. This
specification covers the TCP/IP requirements for transporting FC
Frames; the Fibre Channel documents described in section 2.1 cover
the Fibre Channel requirements.
This specification addresses only the requirements necessary to
properly utilize an IP Network as a conduit for FC Frames. The
result is a specification for an FCIP Entity (see section 5.4).
A product that tunnels an FC Fabric through an IP Network MUST
combine the FCIP Entity with an FC Entity (see section 5.3) using an
implementation specific interface. The requirements placed on an FC
Entity by this specification to achieve proper delivery of FC Frames
are summarized in appendix H. More information about FC Entities can
be found in the Fibre Channel standards and an example of an FC
Entity can be found in FC-BB-2 [3].
No attempt is being made to define a specific API between an FCIP
Entity and an FC Entity. The approach is to specify required
functional interactions between an FCIP Entity and an FC Entity (both
of which are required to forward FC frames across an IP Network), but
allow implementers to choose how these interactions will be realized.
3. Terminology
Terms used to describe FCIP concepts are defined in this section.
FC End Node - An FC device that uses the connection services provided
by the FC Fabric.
FC Entity - The Fibre Channel specific functional component that
combines with an FCIP Entity to form an interface between an FC
Fabric and an IP Network (see section 5.3).
FC Fabric - An entity that interconnects various Nx_Ports (see [5])
attached to it, and is capable of routing FC Frames using only the
destination ID information in an FC Frame header (see appendix F).
FC Fabric Entity - A Fibre Channel specific element containing one
or more Interconnect_Ports (see FC-SW-2 [4]) and one or more
FC/FCIP Entity pairs. See FC-BB-2 [3] for details about FC Fabric
Entities.
FC Frame - The basic unit of Fibre Channel data transfer (see
appendix F).
FC Frame Receiver Portal - The access point through which an FC
Frame and time stamp enter an FCIP Data Engine from the FC Entity.
FC Frame Transmitter Portal - The access point through which a
reconstituted FC Frame and time stamp leave an FCIP Data Engine to
the FC Entity.
FC/FCIP Entity pair - The combination of one FC Entity and one FCIP
entity.
FCIP Data Engine (FCIP_DE) - The component of an FCIP Entity that
handles FC Frame encapsulation, de-encapsulation, and transmission
FCIP Frames through a single TCP Connection (see section 5.6).
FCIP Entity - The entity responsible for the FCIP protocol exchanges
on the IP Network and encompasses FCIP_LEP(s) and FCIP Control and
Services module (see section 5.4).
FCIP Frame - An FC Frame plus the FC Frame Encapsulation [19]
header, encoded SOF and encoded EOF that contains the FC Frame
(see section 5.6.1).
FCIP Link - One or more TCP Connections that connect one FCIP_LEP to
another (see section 5.2).
FCIP Link Endpoint (FCIP_LEP) - The component of an FCIP Entity
that handles a single FCIP Link and contains one or more FCIP_DEs
(see section 5.5).
Encapsulated Frame Receiver Portal - The TCP access point through
which an FCIP Frame is received from the IP Network by an FCIP
Data Engine.
Encapsulated Frame Transmitter Portal - The TCP access point through
which an FCIP Frame is transmitted to the IP Network by an FCIP
Data Engine.
FCIP Special Frame (FSF) - A specially formatted FC Frame containing
information used by the FCIP protocol (see section 7).
4. Protocol Summary
The FCIP protocol is summarized as follows:
1) The primary function of an FCIP Entity is forwarding FC Frames,
employing FC Frame Encapsulation described in [19].
2) Viewed from the IP Network perspective, FCIP Entities are peers
and communicate using TCP/IP. Each FCIP Entity contains one or
more TCP endpoints in the IP-based network.
3) Viewed from the FC Fabric perspective, pairs of FCIP Entities, in
combination with their associated FC Entities, forward FC Frames
between FC Fabric elements. The FC End Nodes are unaware of the
existence of the FCIP Link.
4) FC Primitive Signals, Primitive Sequences, and Class 1 FC Frames
are not transmitted across an FCIP Link because they cannot be
encoded using FC Frame Encapsulation [19].
5) The path (route) taken by an encapsulated FC Frame follows the
normal routing procedures of the IP Network.
6) An FCIP Entity MAY contain multiple FCIP Link Endpoints, but each
FCIP Link Endpoint (FCIP_LEP) communicates with exactly one other
FCIP_LEP.
7) When multiple FCIP_LEPs with multiple FCIP_DEs are in use,
selection of which FCIP_DE to use for encapsulating and
transmitting a given FC Frame is covered in FC-BB-2 [3]. FCIP
Entities do not actively participate in FC Frame routing.
8) The FCIP Control and Services module MAY use TCP/IP quality of
service features (see section 10.2).
9) It is necessary to statically or dynamically configure each FCIP
entity with the IP addresses and TCP port numbers corresponding to
FCIP Entities with which it is expected to initiate communication.
If dynamic discovery of participating FCIP Entities is supported,
the function SHALL be performed using the Service Location
Protocol (SLPv2) [17]. It is outside the scope of this
specification to describe any static configuration method for
participating FCIP Entity discovery. Refer to section 8.1.2.2 for
a detailed description of dynamic discovery of participating FCIP
Entities using SLPv2.
10) Before creating a TCP Connection to a peer FCIP Entity, the FCIP
Entity attempting to create the TCP connection SHALL statically or
dynamically determine the IP address, TCP port, expected FC Fabric
Entity World Wide Name, TCP Connection Parameters, and Quality of
Service Information.
11) FCIP Entities do not actively participate in the discovery of FC
source and destination identifiers. Discovery of FC addresses
(accessible via the FCIP Entity) is provided by techniques and
protocols within the FC architecture as described in FC-FS [5] and
FC-SW-2 [4].
12) To support IP Network security (see section 9), FCIP Entities
MUST:
1) implement cryptographically protected authentication and
cryptographic data integrity keyed to the authentication
process, and
2) implement data confidentiality security features.
13) On an individual TCP Connection, this specification relies on
TCP/IP to deliver a byte stream in the same order that it was
sent.
14) This specification assumes the presence of and requires the use of
TCP and FC data loss and corruption mechanisms. The error
detection and recovery features described in this specification
complement and support these existing mechanisms.
5. The FCIP Model
5.1. FCIP Protocol Model
The relationship between FCIP and other protocols is illustrated in
figure 1.
+------------------------+ FCIP Link +------------------------+
| FCIP |===========| FCIP |
+--------+------+--------+ +--------+------+--------+
| FC-2 | | TCP | | TCP | | FC-2 |
+--------+ +--------+ +--------+ +--------+
| FC-1 | | IP | | IP | | FC-1 |
+--------+ +--------+ +--------+ +--------+
| FC-0 | | LINK | | LINK | | FC-0 |
+--------+ +--------+ +--------+ +--------+
| | PHY | | PHY | |
| +--------+ +--------+ |
| | | |
| | IP Network | |
V +--------------------+ V
to Fibre to Fibre
Channel Channel
Fabric Fabric
Key: FC-0 - Fibre Channel Physical Media Layer
FC-1 - Fibre Channel Encode and Decode Layer
FC-2 - Fibre Channel Framing and Flow Control Layer
TCP - Transmission Control Protocol
IP - Internet Protocol
LINK - IP Link Layer
PHY - IP Physical Layer
Figure 1: FCIP Protocol Stack Model
Note that the objective of the FCIP Protocol is to create and
maintain one or more FCIP Links to transport data.
5.2. FCIP Link
The FCIP Link is the basic unit of service provided by the FCIP
Protocol to an FC Fabric. As shown in figure 2, an FCIP Link
connects two portions of an FC Fabric using an IP Network as a
transport to form a single FC Fabric.
/\/\/\/\/\/\ /\/\/\/\/\/\ /\/\/\/\/\/\
\ FC / \ IP / \ FC /
/ Fabric \=========/ Network \=========/ Fabric \
\/\/\/\/\/\/ \/\/\/\/\/\/ \/\/\/\/\/\/
| |
|<--------- FCIP Link -------->|
Figure: 2 FCIP Link Model
At the points where the ends of the FCIP Link meet portions of the FC
Fabric, an FCIP Entity (see section 5.4) combines with an FC Entity
as described in section 5.3 to serve as the interface between FC and
IP.
An FCIP Link SHALL contain at least one TCP Connection and MAY
contain more than one TCP Connection. The endpoints of a single TCP
Connection are FCIP Data Engines (see section 5.6). The endpoints of
a single FCIP Link are FCIP Link Endpoints (see section 5.5).
5.3. FC Entity
An implementation that tunnels an FC Fabric through an IP Network
MUST combine an FC Entity with an FCIP Entity (see section 5.4) to
form a complete interface between the FC Fabric and IP Network as
shown in figure 3. An FC Fabric Entity may contain multiple