Request for Comments: 4379 Juniper Networks, Inc.
Updates: 1122 G. Swallow
Category: Standards Track Cisco Systems, Inc.
February 2006
Detecting Multi-Protocol Label Switched (MPLS) Data Plane Failures
Status of This Memo
This document specifies an Internet standards track protocol for the
Internet community, and requests discussion and suggestions for
improvements. Please refer to the current edition of the "Internet
Official Protocol Standards" (STD 1) for the standardization state
and status of this protocol. Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2006).
Abstract
This document describes a simple and efficient mechanism that can be
used to detect data plane failures in Multi-Protocol Label Switching
(MPLS) Label Switched Paths (LSPs). There are two parts to this
document: information carried in an MPLS "echo request" and "echo
reply" for the purposes of fault detection and isolation, and
mechanisms for reliably sending the echo reply.
Table of Contents
1. Introduction ....................................................2
1.1. Conventions ................................................3
1.2. Structure of This Document .................................3
1.3. Contributors ...............................................3
2. Motivation ......................................................4
2.1. Use of Address Range 127/8 .................................4
3. Packet Format ...................................................6
3.1. Return Codes ..............................................10
3.2. Target FEC Stack ..........................................11
3.2.1. LDP IPv4 Prefix ....................................12
3.2.2. LDP IPv6 Prefix ....................................13
3.2.3. RSVP IPv4 LSP ......................................13
3.2.4. RSVP IPv6 LSP ......................................14
3.2.5. VPN IPv4 Prefix ....................................14
3.2.6. VPN IPv6 Prefix ....................................15
3.2.7. L2 VPN Endpoint ....................................16
3.2.8. FEC 128 Pseudowire (Deprecated) ....................16
3.2.9. FEC 128 Pseudowire (Current) .......................17
3.2.10. FEC 129 Pseudowire ................................18
3.2.11. BGP Labeled IPv4 Prefix ...........................19
3.2.12. BGP Labeled IPv6 Prefix ...........................20
3.2.13. Generic IPv4 Prefix ...............................20
3.2.14. Generic IPv6 Prefix ...............................21
3.2.15. Nil FEC ...........................................21
3.3. Downstream Mapping ........................................22
3.3.1. Multipath Information Encoding .....................26
3.3.2. Downstream Router and Interface ....................28
3.4. Pad TLV ...................................................29
3.5. Vendor Enterprise Number ..................................29
3.6. Interface and Label Stack .................................29
3.7. Errored TLVs ..............................................31
3.8. Reply TOS Byte TLV ........................................31
4. Theory of Operation ............................................32
4.1. Dealing with Equal-Cost Multi-Path (ECMP) .................32
4.2. Testing LSPs That Are Used to Carry MPLS Payloads .........33
4.3. Sending an MPLS Echo Request ..............................33
4.4. Receiving an MPLS Echo Request ............................34
4.4.1. FEC Validation .....................................40
4.5. Sending an MPLS Echo Reply ................................41
4.6. Receiving an MPLS Echo Reply ..............................42
4.7. Issue with VPN IPv4 and IPv6 Prefixes .....................42
4.8. Non-compliant Routers .....................................43
5. References .....................................................43
5.1. Normative References ......................................43
5.2. Informative References ....................................44
6. Security Considerations ........................................44
7. IANA Considerations ............................................46
7.1. Message Types, Reply Modes, Return Codes ..................46
7.2. TLVs ......................................................47
8. Acknowledgements ...............................................48
1. Introduction
This document describes a simple and efficient mechanism that can be
used to detect data plane failures in MPLS Label Switched Paths
(LSPs). There are two parts to this document: information carried in
an MPLS "echo request" and "echo reply", and mechanisms for
transporting the echo reply. The first part aims at providing enough
information to check correct operation of the data plane, as well as
a mechanism to verify the data plane against the control plane, and
thereby localize faults. The second part suggests two methods of
reliable reply channels for the echo request message for more robust
fault isolation.
An important consideration in this design is that MPLS echo requests
follow the same data path that normal MPLS packets would traverse.
MPLS echo requests are meant primarily to validate the data plane,
and secondarily to verify the data plane against the control plane.
Mechanisms to check the control plane are valuable, but are not
covered in this document.
This document makes special use of the address range 127/8. This is
an exception to the behavior defined in RFC 1122 [RFC1122] and
updates that RFC. The motivation for this change and the details of
this exceptional use are discussed in section 2.1 below.
1.1. Conventions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [KEYWORDS].
The term "Must Be Zero" (MBZ) is used in object descriptions for
reserved fields. These fields MUST be set to zero when sent and
ignored on receipt.
Terminology pertaining to L2 and L3 Virtual Private Networks (VPNs)
is defined in [RFC4026].
Since this document refers to the MPLS Time to Live (TTL) far more
frequently than the IP TTL, the authors have chosen the convention of
using the unqualified "TTL" to mean "MPLS TTL" and using "IP TTL" for
the TTL value in the IP header.
1.2. Structure of This Document
The body of this memo contains four main parts: motivation, MPLS echo
request/reply packet format, LSP ping operation, and a reliable
return path. It is suggested that first-time readers skip the actual
packet formats and read the Theory of Operation first; the document
is structured the way it is to avoid forward references.
1.3. Contributors
The following made vital contributions to all aspects of this
document, and much of the material came out of debate and discussion
among this group.
Ronald P. Bonica, Juniper Networks, Inc.
Dave Cooper, Global Crossing
Ping Pan, Hammerhead Systems
Nischal Sheth, Juniper Networks, Inc.
Sanjay Wadhwa, Juniper Networks, Inc.
2. Motivation
When an LSP fails to deliver user traffic, the failure cannot always
be detected by the MPLS control plane. There is a need to provide a
tool that would enable users to detect such traffic "black holes" or
misrouting within a reasonable period of time, and a mechanism to
isolate faults.
In this document, we describe a mechanism that accomplishes these
goals. This mechanism is modeled after the ping/traceroute paradigm:
ping (ICMP echo request [ICMP]) is used for connectivity checks, and
traceroute is used for hop-by-hop fault localization as well as path
tracing. This document specifies a "ping" mode and a "traceroute"
mode for testing MPLS LSPs.
The basic idea is to verify that packets that belong to a particular
Forwarding Equivalence Class (FEC) actually end their MPLS path on a
Label Switching Router (LSR) that is an egress for that FEC. This
document proposes that this test be carried out by sending a packet
(called an "MPLS echo request") along the same data path as other
packets belonging to this FEC. An MPLS echo request also carries
information about the FEC whose MPLS path is being verified. This
echo request is forwarded just like any other packet belonging to
that FEC. In "ping" mode (basic connectivity check), the packet
should reach the end of the path, at which point it is sent to the
control plane of the egress LSR, which then verifies whether it is
indeed an egress for the FEC. In "traceroute" mode (fault
isolation), the packet is sent to the control plane of each transit
LSR, which performs various checks that it is indeed a transit LSR
for this path; this LSR also returns further information that helps
check the control plane against the data plane, i.e., that forwarding
matches what the routing protocols determined as the path.
One way these tools can be used is to periodically ping an FEC to
ensure connectivity. If the ping fails, one can then initiate a
traceroute to determine where the fault lies. One can also
periodically traceroute FECs to verify that forwarding matches the
control plane; however, this places a greater burden on transit LSRs
and thus should be used with caution.
2.1. Use of Address Range 127/8
As described above, LSP ping is intended as a diagnostic tool. It is
intended to enable providers of an MPLS-based service to isolate
network faults. In particular, LSP ping needs to diagnose situations
where the control and data planes are out of sync. It performs this
by routing an MPLS echo request packet based solely on its label
stack. That is, the IP destination address is never used in a
forwarding decision. In fact, the sender of an MPLS echo request
packet may not know, a priori, the address of the router at the end
of the LSP.
Providers of MPLS-based services also need the ability to trace all
of the possible paths that an LSP may take. Since most MPLS services
are based on IP unicast forwarding, these paths are subject to
equal-cost multi-path (ECMP) load sharing.
This leads to the following requirements:
1. Although the LSP in question may be broken in unknown ways, the
likelihood of a diagnostic packet being delivered to a user of an
MPLS service MUST be held to an absolute minimum.
2. If an LSP is broken in such a way that it prematurely terminates,
the diagnostic packet MUST NOT be IP forwarded.
3. A means of varying the diagnostic packets such that they exercise
all ECMP paths is thus REQUIRED.
Clearly, using general unicast addresses satisfies neither of the
first two requirements. A number of other options for addresses were
considered, including a portion of the private address space (as
determined by the network operator) and the newly designated IPv4
link local addresses. Use of the private address space was deemed
ineffective since the leading MPLS-based service is an IPv4 Virtual
Private Network (VPN). VPNs often use private addresses.
The IPv4 link local addresses are more attractive in that the scope
over which they can be forwarded is limited. However, if one were to
use an address from this range, it would still be possible for the
first recipient of a diagnostic packet that "escaped" from a broken
LSP to have that address assigned to the interface on which it
arrived and thus could mistakenly receive such a packet.
Furthermore, the IPv4 link local address range has only recently been
allocated. Many deployed routers would forward a packet with an
address from that range toward the default route.
The 127/8 range for IPv4 and that same range embedded in as IPv4-
mapped IPv6 addresses for IPv6 was chosen for a number of reasons.
RFC 1122 allocates the 127/8 as "Internal host loopback address" and
states: "Addresses of this form MUST NOT appear outside a host."
Thus, the default behavior of hosts is to discard such packets. This
helps to ensure that if a diagnostic packet is misdirected to a host,
it will be silently discarded.
RFC 1812 [RFC1812] states:
A router SHOULD NOT forward, except over a loopback interface, any
packet that has a destination address on network 127. A router
MAY have a switch that allows the network manager to disable these
checks. If such a switch is provided, it MUST default to
performing the checks.
This helps to ensure that diagnostic packets are never IP forwarded.
The 127/8 address range provides 16M addresses allowing wide
flexibility in varying addresses to exercise ECMP paths. Finally, as
an implementation optimization, the 127/8 provides an easy means of
identifying possible LSP packets.
3. Packet Format
An MPLS echo request is a (possibly labeled) IPv4 or IPv6 UDP packet;
the contents of the UDP packet have the following format:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version Number | Global Flags |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Message Type | Reply mode | Return Code | Return Subcode|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sender’s Handle |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TimeStamp Sent (seconds) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TimeStamp Sent (microseconds) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TimeStamp Received (seconds) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TimeStamp Received (microseconds) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TLVs ... |
. .
. .
. .
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
The Version Number is currently 1. (Note: the version number is to
be incremented whenever a change is made that affects the ability of
an implementation to correctly parse or process an MPLS echo
request/reply. These changes include any syntactic or semantic
changes made to any of the fixed fields, or to any Type-Length-Value
(TLV) or sub-TLV assignment or format that is defined at a certain
version number. The version number may not need to be changed if an
optional TLV or sub-TLV is added.)
The Global Flags field is a bit vector with the following format:
0 1
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MBZ |V|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
One flag is defined for now, the V bit; the rest MUST be set to zero
when sending and ignored on receipt.
The V (Validate FEC Stack) flag is set to 1 if the sender wants the
receiver to perform FEC Stack validation; if V is 0, the choice is
left to the receiver.
The Message Type is one of the following:
Value Meaning
----- -------
1 MPLS echo request
2 MPLS echo reply
The Reply Mode can take one of the following values:
Value Meaning
----- -------
1 Do not reply
2 Reply via an IPv4/IPv6 UDP packet
3 Reply via an IPv4/IPv6 UDP packet with Router Alert
4 Reply via application level control channel
An MPLS echo request with 1 (Do not reply) in the Reply Mode field
may be used for one-way connectivity tests; the receiving router may
log gaps in the Sequence Numbers and/or maintain delay/jitter
statistics. An MPLS echo request would normally have 2 (Reply via an
IPv4/IPv6 UDP packet) in the Reply Mode field. If the normal IP
return path is deemed unreliable, one may use 3 (Reply via an
IPv4/IPv6 UDP packet with Router Alert). Note that this requires
that all intermediate routers understand and know how to forward MPLS
echo replies. The echo reply uses the same IP version number as the
received echo request, i.e., an IPv4 encapsulated echo reply is sent
in response to an IPv4 encapsulated echo request.
Some applications support an IP control channel. One such example is
the associated control channel defined in Virtual Circuit
Connectivity Verification (VCCV) [VCCV]. Any application that
supports an IP control channel between its control entities may set
the Reply Mode to 4 (Reply via application level control channel) to
ensure that replies use that same channel. Further definition of
this codepoint is application specific and thus beyond the scope of
this document.
Return Codes and Subcodes are described in the next section.
The Sender’s Handle is filled in by the sender, and returned
unchanged by the receiver in the echo reply (if any). There are no
semantics associated with this handle, although a sender may find
this useful for matching up requests with replies.
The Sequence Number is assigned by the sender of the MPLS echo
request and can be (for example) used to detect missed replies.
The TimeStamp Sent is the time-of-day (in seconds and microseconds,
according to the sender’s clock) in NTP format [NTP] when the MPLS
echo request is sent. The TimeStamp Received in an echo reply is the
time-of-day (according to the receiver’s clock) in NTP format that
the corresponding echo request was received.
TLVs (Type-Length-Value tuples) have the following format:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type | Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Value |
. .
. .
. .
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Types are defined below; Length is the length of the Value field in
octets. The Value field depends on the Type; it is zero padded to
align to a 4-octet boundary. TLVs may be nested within other TLVs,
in which case the nested TLVs are called sub-TLVs. Sub-TLVs have
independent types and MUST also be 4-octet aligned.
Two examples follow. The Label Distribution Protocol (LDP) IPv4 FEC
sub-TLV has the following format:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 1 (LDP IPv4 FEC) | Length = 5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IPv4 prefix |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Prefix Length | Must Be Zero |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
The Length for this TLV is 5. A Target FEC Stack TLV that contains
an LDP IPv4 FEC sub-TLV and a VPN IPv4 prefix sub-TLV has the
following format:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 1 (FEC TLV) | Length = 12 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sub-Type = 1 (LDP IPv4 FEC) | Length = 5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IPv4 prefix |