Request for Comments: 3479 Movaz Networks, Inc.
Category: Standards Track February 2003
Fault Tolerance for the Label Distribution Protocol (LDP)
Status of this Memo
This document specifies an Internet standards track protocol for the
Internet community, and requests discussion and suggestions for
improvements. Please refer to the current edition of the "Internet
Official Protocol Standards" (STD 1) for the standardization state
and status of this protocol. Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2003). All Rights Reserved.
IESG Note
This specification includes procedures for failure detection and
failover for a TCP connection carrying MPLS LDP control traffic, so
that it can be switched to a new TCP connection. It does not provide
a general approach to using multiple TCP connections to provide this
kind of fault tolerance. The specification lacks adequate guidance
for the timer and retry value choices related to the TCP connection
fault tolerance procedures. The specification should not serve as a
model for TCP connection fault tolerance design for any future
document, and users are advised to test configurations based on this
specification very carefully for problems such as premature
failovers.
Abstract
Multiprotocol Label Switching (MPLS) systems will be used in core
networks where system downtime must be kept to an absolute minimum.
Many MPLS Label Switching Routers (LSRs) may, therefore, exploit
Fault Tolerant (FT) hardware or software to provide high availability
of the core networks.
The details of how FT is achieved for the various components of an FT
LSR, including Label Distribution Protocol (LDP), the switching
hardware and TCP, are implementation specific. This document
identifies issues in the LDP specification in RFC3036, "LDP
Specification", that make it difficult to implement an FT LSR using
the current LDP protocols, and defines enhancements to the LDP
specification to ease such FT LSR implementations.
The issues and extensions described here are equally applicable to
RFC3212, "Constraint-Based LSP Setup Using LDP" (CR-LDP).
Table of Contents
1. Conventions and Terminology used in this document..........3
2. Contributing Authors.......................................4
3. Introduction...............................................4
3.1. Fault Tolerance for MPLS..............................4
3.2. Issues with LDP.......................................5
4. Overview of LDP FT Enhancements............................7
4.1. Establishing an FT LDP Session........................8
4.1.1 Interoperation with Non-FT LSRs.................8
4.2. TCP Connection Failure................................9
4.2.1 Detecting TCP Connection Failures...............9
4.2.2 LDP Processing after Connection Failure.........9
4.3. Data Forwarding During TCP Connection Failure........10
4.4. FT LDP Session Reconnection..........................10
4.5. Operations on FT Labels..............................11
4.6. Check-Pointing.......................................11
4.6.1 Graceful Termination...........................12
4.7. Label Space Depletion and Replenishment..............13
4.8. Tunneled LSPs........................................13
5. FT Operations.............................................14
5.1. FT LDP Messages......................................14
5.1.1 Sequence Numbered FT Label Messages............14
5.1.2 FT Address Messages............................15
5.1.3 Label Resources Available Notifications........15
5.2. FT Operation ACKs....................................17
5.3. Preservation of FT State.............................17
5.4. FT Procedure After TCP Failure.......................19
5.4.1 FT LDP Operations During TCP Failure...........20
5.5. FT Procedure After TCP Re-connection.................21
5.5.1 Re-Issuing FT Messages.........................22
6. Check-Pointing Procedures.................................22
6.1 Check-Pointing with the Keepalive Message.............23
6.2 Quiesce and Keepalive.................................23
7. Changes to Existing Messages..............................24
7.1. LDP Initialization Message...........................24
7.2. LDP Keepalive Messages...............................25
7.3. All Other LDP Session Messages.......................25
8. New Fields and Values.....................................26
8.1. Status Codes.........................................26
8.2. FT Session TLV.......................................27
8.3. FT Protection TLV....................................29
8.4. FT ACK TLV...........................................32
8.5. FT Cork TLV..........................................33
9. Example Use...............................................34
9.1. Session Failure and Recovery - FT Procedures.........34
9.2. Use of Check-Pointing With FT Procedures.............37
9.3. Temporary Shutdown With FT Procedures................38
9.4. Temporary Shutdown With FT Procedures
and Check-Pointing...................................40
9.5. Check-Pointing Without FT Procedures.................42
9.6. Graceful Shutdown With Check-Pointing
But No FT Procedures.................................44
10. Security Considerations..................................45
11. Implementation Notes.....................................47
11.1. FT Recovery Support on Non-FT LSRs..................47
11.2. ACK generation logic................................47
11.2.1 Ack Generation Logic When Using
Check-Pointing...............................47
11.3 Interactions With Other Label Distribution
Mechanisms...........................................48
12. Acknowledgments..........................................48
13. Intellectual Property Consideration......................49
14. References...............................................49
14.1. Normative References................................49
14.2. Informative References..............................50
15. Authors' Addresses.......................................50
16. Full Copyright Statement.................................52
1. Conventions and Terminology used in this document
Definitions of key words and terms applicable to LDP and CR-LDP are
inherited from [RFC3212] and [RFC3036].
The term "FT Label" is introduced in this document to indicate a
label for which some fault tolerant operation is used. A "non-FT
Label" is not fault tolerant and is handled as specified in
[RFC3036].
The term "Sequence Numbered FT Label" is used to indicate an FT label
which is secured using the sequence number in the FT Protection TLV
described in this document.
The term "Check-Pointable FT Label" is used to indicate an FT label
which is secured by using the check-pointing techniques described in
this document.
The extensions to LDP specified in this document are collectively
referred to as the "LDP FT enhancements".
Within the context of this document, "Check-Pointing" refers to a
process of message exchanges that confirm receipt and processing (or
secure storage) of specific protocol messages.
When talking about the individual bits in the 16-bit FT Flag Field,
the words "bit" and "flag" are used interchangeably.
In the examples quoted, the following notation is used: Ln : An LSP.
For example L1. Pn : An LDP peer. For example P1.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC2119
[RFC2119].
2. Contributing Authors
This document was the collective work of several individuals over a
period of several years. The text and content of this document was
contributed by the editor and the co-authors listed in section 15,
"Authors' Addresses".
3. Introduction
High Availability (HA) is typically claimed by equipment vendors when
their hardware achieves availability levels of at least 99.999% (five
9s). To implement this, the equipment must be capable of recovering
from local hardware and software failures through a process known as
fault tolerance (FT).
The usual approach to FT involves provisioning backup copies of
hardware and/or software. When a primary copy fails, processing is
switched to the backup copy. This process, called failover, should
result in minimal disruption to the Data Plane.
In an FT system, backup resources are sometimes provisioned on a
one-to-one basis (1:1), sometimes as one-to-many (1:n), and
occasionally as many-to-many (m:n). Whatever backup provisioning is
made, the system must switch to the backup automatically on failure
of the primary, and the software and hardware state in the backup
must be set to replicate the state in the primary at the point of
failure.
3.1. Fault Tolerance for MPLS
MPLS is a technology that will be used in core networks where system
downtime must be kept to an absolute minimum. Many MPLS LSRs may,
therefore, exploit FT hardware or software to provide high
availability of core networks.
In order to provide HA, an MPLS system needs to be able to survive a
variety of faults with minimal disruption to the Data Plane,
including the following fault types:
- failure/hot-swap of a physical connection between LSRs.
- failure/hot-swap of the switching fabric in an LSR.
- failure of the TCP or LDP stack in an LSR.
- software upgrade to the TCP or LDP stacks in an LSR.
The first two examples of faults listed above are confined to the
Data Plane. Such faults can be handled by providing redundancy in
the Data Plane which is transparent to LDP operating in the Control
Plane. The last two example types of fault require action in the
Control Plane to recover from the fault without disrupting traffic in
the Data Plane. This is possible because many recent router
architectures separate the Control and Data Planes such that
forwarding can continue unaffected by recovery action in the Control
Plane.
3.2. Issues with LDP
LDP uses TCP to provide reliable connections between LSRs over which
they exchange protocol messages to distribute labels and set up LSPs.
A pair of LSRs that have such a connection are referred to as LDP
peers.
TCP enables LDP to assume reliable transfer of protocol messages.
This means that some of the messages do not need to be acknowledged
(for example, Label Release).
LDP is defined such that if the TCP connection fails, the LSR should
immediately tear down the LSPs associated with the session between
the LDP peers, and release any labels and resources assigned to those
LSPs.
It is notoriously hard to provide a Fault Tolerant implementation of
TCP. To do so might involve making copies of all data sent and
received. This is an issue familiar to implementers of other TCP
applications such as BGP.
During failover affecting the TCP or LDP stacks, the TCP connection
may be lost. Recovery from this position is made worse by the fact
that LDP control messages may have been lost during the connection
failure. Since these messages are unconfirmed, it is possible that
LSP or label state information will be lost.
This document describes a solution which involves:
- negotiation between LDP peers of the intent to support extensions
to LDP that facilitate recovery from failover without loss of
LSPs.
- selection of FT survival on a per LSP/label basis.
- acknowledgement of LDP messages to ensure that a full handshake is
performed on those messages either frequently (such as per
message) or less frequently as in check-pointing.
- solicitation of up-to-date acknowledgement (check-pointing) of
previous LDP messages to ensure the current state is flushed to
disk/NVRAM, with an additional option that allows an LDP partner
to request that state is flushed in both directions if graceful
shutdown is required.
- re-issuing lost messages after failover to ensure that LSP/label
state is correctly recovered after reconnection of the LDP
session.
The issues and objectives described above are equally applicable to
CR-LDP.
Other objectives of this document are to:
- offer backward-compatibility with LSRs that do not implement these
extensions to LDP.
- preserve existing protocol rules described in [RFC3036] for
handling unexpected duplicate messages and for processing
unexpected messages referring to unknown LSPs/labels.
- avoid full state refresh solutions (such as those present in RSVP:
see [RFC2205], [RFC2961], [RFC3209] and [RFC3478]) whether they be
continual, or limited to post-failover recovery.
Note that this document concentrates on the preservation of label
state for labels exchanged between a pair of adjacent LSRs when the
TCP connection between those LSRs is lost. This is a requirement for
Fault Tolerant operation of LSPs, but a full implementation of end-
to-end protection for LSPs requires that this be combined with other
techniques that are outside the scope of this document.
In particular, this document does not attempt to describe how to
modify the routing of an LSP or the resources allocated to a label or
LSP, which is covered by [RFC3214]. This document also does not
address how to provide automatic layer 2 or layer 3 protection
switching for a label or LSP, which is a separate area for study.
This specification does not preclude an implementation from
attempting (or require it to attempt) to use the FT behavior
described here to recover from a preemptive failure of a connection
on a non-FT system due to, for example, a partial system crash.
Note, however, that there are potential issues too numerous to list
here - not least the likelihood that the same crash will immediately
occur when processing the restored data.
4. Overview of LDP FT Enhancements
The LDP FT enhancements consist of the following main elements, which
are described in more detail in the sections that follow.
- The presence of an FT Session TLV on the LDP Initialization
message indicates that an LSR supports some form of protection or
recovery from session failure. A flag bit within this TLV (the S
bit) indicates that the LSR supports the LDP FT enhancements on
this session. Another flag (the C bit) indicates that the check-
pointing procedures are to be used.
- An FT Reconnect Flag in the FT Session TLV (the R bit) indicates
whether an LSR has preserved FT Label state across a failure of
the TCP connection.
- An FT Reconnection Timeout, exchanged on the LDP Initialization
message, that indicates the maximum time peer LSRs will preserve
FT Label state after a failure of the TCP connection.
- An FT Protection TLV used to identify operations that affect LDP
labels. All LDP messages carrying the FT Protection TLV need to
be secured (e.g. to NVRAM) and ACKed to the sending LDP peer so
that the state for Sequence Numbered FT Labels can be correctly
recovered after LDP session reconnection.
Note that the implementation within an FT system is left open by
this document. An implementation could choose to secure entire
messages relating to Sequence Numbered FT Labels, or it could
secure only the relevant state information.
- Address advertisement may also be secured by use of the FT
Protection TLV. This enables recovery after LDP session
reconnection without the need to re-advertise what may be a very
large number of addresses.
- The FT Protection TLV may also be used on the Keepalive message to
flush acknowledgement of all previous FT operations. This enables
a check-point for future recovery, either in mid-session or prior
to graceful shutdown of an LDP session. This procedure may also
be used to check-point all (that is both FT and non-FT) operations
for future recovery.
4.1. Establishing an FT LDP Session
In order that the extensions to LDP [RFC3036] described in this
document can be used successfully on an LDP session between a pair of
LDP peers, they MUST negotiate that the LDP FT enhancements are to be
used on the LDP session.
This is done on the LDP Initialization message exchange using a new
FT Session TLV. Presence of this TLV indicates that the peer wants
to support some form of protection or recovery processing. The S bit
within this TLV indicates that the peer wants to support the LDP FT
enhancements on this LDP session. The C bit indicates that the peer
wants to support the check-pointing functions described in this
document. The S and C bits may be set independently.
The relevant LDP FT enhancements MUST be supported on an LDP session
if both LDP peers include an FT Session TLV on the LDP Initialization
message and have the same setting of the S or C bit.
If either LDP Peer does not include the FT Session TLV LDP
Initialization message, or if there is no match of S and C bits
between the peers, the LDP FT enhancements MUST NOT be used during
this LDP session. Use of LDP FT enhancements by a sending LDP peer
in these cases MUST be interpreted by the receiving LDP peer as a
serious protocol error causing the session to be terminated.
An LSR MAY present different FT/non-FT behavior on different TCP
connections, even if those connections are successive instantiations
of the LDP session between the same LDP peers.
4.1.1 Interoperation with Non-FT LSRs
The FT Session TLV on the LDP Initialization message carries the U-
bit. If an LSR does not support any protection or recovery
mechanisms, it will ignore this TLV. Since such partners also do not
include the FT Session TLV, all LDP sessions to such LSRs will not
use the LDP FT enhancements.
The rest of this document assumes that the LDP sessions under
discussion are between LSRs that support the LDP FT enhancements,
except where explicitly stated otherwise.
4.2. TCP Connection Failure
4.2.1 Detecting TCP Connection Failures
TCP connection failures may be detected and reported to the LDP
component in a variety of ways. These should all be treated in the
same way by the LDP component.
- Indication from the management component that a TCP connection or
underlying resource is no longer active.
- Notification from a hardware management component of an interface
failure.
- Sockets keepalive timeout.
- Sockets send failure.
- New (incoming) Socket opened.
- LDP protocol timeout.
4.2.2 LDP Processing after Connection Failure
If the LDP FT enhancements are not in use on an LDP session, the
action of the LDP peers on failure of the TCP connection is as
specified in [RFC3036].
All state information and resources associated with non-FT Labels
MUST be released on the failure of the TCP connection, including
deprogramming the non-FT Label from the switching hardware. This is
equivalent to the behavior specified in [RFC3036].
If the LDP FT enhancements are in use on an LDP session, both LDP
peers SHOULD preserve state information and resources associated with
FT Labels exchanged on the LDP session. Both LDP peers SHOULD use a
timer to release the preserved state information and resources
associated with FT-labels if the TCP connection is not restored
within a reasonable period. The behavior when this timer expires is
equivalent to the LDP session failure behavior described in
[RFC3036].
The FT Reconnection Timeout each LDP peer intends to apply to the LDP
session is carried in the FT Session TLV on the LDP Initialization
messages. Both LDP peers MUST use the value that corresponds to the
lesser timeout interval of the two proposed timeout values from the
LDP Initialization exchange, where a value of zero is treated as
positive infinity.
4.3. Data Forwarding During TCP Connection Failure
An LSR that implements the LDP FT enhancements SHOULD preserve the
programming of the switching hardware across a failover. This
ensures that data forwarding is unaffected by the state of the TCP
connection between LSRs.
It is an integral part of FT failover processing in some hardware
configurations that some data packets might be lost. If data loss is
not acceptable to the applications using the MPLS network, the LDP FT
enhancements described in this document SHOULD NOT be used.
4.4. FT LDP Session Reconnection
When a new TCP connection is established, the LDP peers MUST exchange
LDP Initialization messages. When a new TCP connection is
established after failure, the LDP peers MUST re-exchange LDP
Initialization messages.
If an LDP peer includes the FT Session TLV with the S bit set in the
LDP Initialization message for the new instantiation of the LDP
session, it MUST also set the FT Reconnect Flag according to whether
it has been able to preserve label state. The FT Reconnect Flag is
carried in the FT Session TLV.
If an LDP peer has preserved all state information for previous
instantiations of the LDP session, then it SHOULD set the FT
Reconnect Flag to 1 in the FT Session TLV. Otherwise, it MUST set
the FT Reconnect Flag to 0.
If either LDP peer sets the FT Reconnect Flag to 0, or omits the FT
Session TLV, both LDP peers MUST release any state information and
resources associated with the previous instantiation of the LDP
session between the same LDP peers, including FT Label state and
Addresses. This ensures that network resources are not permanently
lost by one LSR if its LDP peer is forced to undergo a cold start.
If an LDP peer changes any session parameters (for example, the label
space bounds) from the previous instantiation, the nature of any
preserved labels may have changed. In particular, previously
allocated labels may now be out of range. For this reason, session
reconnection MUST use the same parameters as were in use on the
session before the failure. If an LDP peer notices that the
parameters have been changed by the other peer, it SHOULD send a
Notification message with the 'FT Session parameters changed' status
code.
If both LDP peers set the FT Reconnect Flag to 1, both LDP peers MUST
use the procedures indicated in this document to complete any label
operations on Sequence Numbered FT Labels that were interrupted by
the LDP session failure.
If an LDP peer receives an LDP Initialization message with the FT
Reconnect Flag set before it sends its own Initialization message,
but has retained no information about the previous version of the
session, it MUST respond with an Initialization message with the FT
Reconnect Flag clear. If an LDP peer receives an LDP Initialization
message with the FT Reconnect Flag set in response to an
Initialization message that it has sent with the FT Reconnect Flag
clear, it MUST act as if no state was retained by either peer on the
session.
4.5. Operations on FT Labels
Label operations on Sequence Numbered FT Labels are made Fault
Tolerant by providing acknowledgement of all LDP messages that affect
Sequence Numbered FT Labels. Acknowledgements are achieved by means
of sequence numbers on these LDP messages.
The message exchanges used to achieve acknowledgement of label
operations and the procedures used to complete interrupted label
operations are detailed in section 5, "FT Operations".
Using these acknowledgements and procedures, it is not necessary for
LDP peers to perform a complete re-synchronization of state for all
Sequence Numbered FT Labels, either on re-connection of the LDP
session between the LDP peers or on a timed basis.
4.6. Check-Pointing
Check-pointing is a useful feature that allows nodes to reduce the
amount of processing that they need to do to acknowledge LDP
messages. The C bit in the FT Session TLV is used to indicate that
check-pointing is supported.
Under the normal operation on Sequence Numbered FT Labels,
acknowledgments may be deferred during normal processing and only
sent periodically. Check-pointing may be used to flush
acknowledgement from a peer by including a sequence number on a
Keepalive message requesting acknowledgement of that message and all
previous messages. In this case, all Sequence Numbered FT Labels are
Check-Pointable FT Labels.
If the S bit is not agreed upon, check-pointing may still be used.
In this case it is used to acknowledge all messages exchanged between
the peers, and all labels are Check-Pointable FT Labels.
This offers an approach where acknowledgements need not be sent to
every message or even frequently, but are only sent as check-points
in response to requests carried on Keepalive messages. Such an
approach may be considered optimal in systems that do not show a high
degree of change over time (such as targeted LDP sessions) and that
are prepared to risk loss of state for the most recent LDP exchanges.
More dynamic systems (such as LDP discovery sessions) are more likely
to want to acknowledge state changes more frequently so that the
maximum amount of state can be preserved over a failure.
Note that an important consideration of this document is that nodes
acknowledging messages on a one-for-one basis, nodes deferring
acknowledgements, and nodes relying on check-pointing, should all
interoperate seamlessly and without protocol negotiation beyond
session initialization.
Further discussion of this feature is provided in section 5, "FT
Operations".
4.6.1 Graceful Termination
A feature that builds on check-pointing is graceful termination.
In some cases, such as controlled failover or software upgrade, it is
possible for a node to know in advance that it is going to terminate
its session with a peer.
In these cases the node that intends terminating the session can
flush acknowledgement using a check-point request as described above.
The sender SHOULD not send further label or address-related messages
after requesting shutdown check-pointing in order to preserve the
integrity of its saved state.
This, however, only provides for acknowledgement in one direction,
and the node that is being terminated also requires verification that
it has secured all state sent by its peer. This is achieved by a
three-way hand shake of the check-point which is requested by an
additional TLV (the Cork TLV) in the Keepalive message.
Further discussion of this feature is provided in section 5, "FT
Operations".
4.7. Label Space Depletion and Replenishment
When an LDP peer is unable to satisfy a Label Request message because
it has no more available labels, it sends a Notification message
carrying the status code 'No label resources'. This warns the
requesting LDP peer that subsequent Label Request messages are also
likely to fail for the same reason. This message does not need to be
acknowledged for FT purposes since Label Request messages sent after
session recovery will receive the same response. However, the LDP
peer that receives a 'No label resources' Notification stops sending
Label Request messages until it receives a 'Label resources
available' Notification message. Since this unsolicited Notification
might get lost during session failure, it may be protected using the
procedures described in this document.
An alternative approach allows that an implementation may always
assume that labels are available when a session is re-established.
In this case, it is possible that it may throw away the 'No label
resources' information from the previous incarnation of the session
and may send a batch of LDP messages on session re-establishment that
will fail and that it could have known would fail.
Note that the sender of a 'Label resources available' Notification
message may choose whether to add a sequence number requesting
acknowledgement. Conversely, the receiver of 'Label resources
available' Notification message may choose to acknowledge the message
without actually saving any state.
This is an implementation choice made possible by making the FT
parameters on the Notification message optional. Implementations
will interoperate fully if they take opposite approaches, but
additional LDP messages may be sent unnecessarily on session
recovery.
4.8. Tunneled LSPs
The procedures described in this document can be applied to LSPs that
are tunnels and to LSPs that are carried by tunnels. Recall that
tunneled LSPs are managed by a single LDP session that runs end to
end, while the tunnel is managed by a different LDP session for each
hop along the path. Nevertheless, a break in one of the sessions
that manages the tunnel is likely to correspond with a break in the
session that manages the tunneled LSP. This is certainly the case
when the LDP exchanges share a failed link, but need not be the case
if the LDP messages have been routed along a path that is different
from that of the tunnel, or if the failure in the tunnel is caused by
an LDP software failure at a transit LSR.
In order that the forwarding path of a tunneled LSP be preserved, the
forwarding path of the tunnel itself must be preserved. This means
that the tunnel must not be torn down if there is any session failure
along its path. To achieve this, the label exchanges between each
pair of LDP peers along the path of the tunnel must use one of the
procedures in this document or in [RFC3478].
It is perfectly acceptable to mix the restart procedures used for the
tunnel and the tunneled LSP. For example, the tunnel could be set up
using just check-pointing because it is a stable LSP, but the
tunneled LSPs might use full FT procedures so that they can recover
active state.
Lastly, it is permissible to carry tunneled LSPs that do not have FT
protection in an LSP that has FT protection.
5. FT Operations
Once an FT LDP session has been established, using the S bit in the
FT Session TLV on the Session Initialization message as described in
section 4.1, "Establishing an FT LDP Session", both LDP peers MUST
apply the procedures described in this section for FT LDP message
exchanges.
If the LDP session has been negotiated to not use the LDP FT
enhancements, these procedures MUST NOT be used.
5.1. FT LDP Messages
5.1.1 Sequence Numbered FT Label Messages
A label is identified as being a Sequence Numbered FT Label if the
initial Label Request or Label Mapping message relating to that label
carries the FT Protection TLV.
It is a valid implementation option to flag all labels as Sequence
Numbered FT Labels. Indeed this may be a preferred option for
implementations wishing to use Keepalive messages carrying the FT
Protection TLV to achieve periodic saves of the complete label
forwarding state.
If a label is a Sequence Numbered FT Label, all LDP messages
affecting that label MUST carry the FT Protection TLV so that the
state of the label can be recovered after a failure of the LDP
session.
A further valid option is for no labels to be Sequence Numbered FT
Labels. In this case, check-pointing using the Keepalive message
applies to all messages exchanged on the session.
5.1.1.1 Scope of FT Labels
The scope of the FT/non-FT status of a label is limited to the LDP
message exchanges between a pair of LDP peers.
In Ordered Control, when the message is forwarded downstream or
upstream, the TLV may be present or absent according to the
requirements of the LSR sending the message.
If a platform-wide label space is used for FT Labels, an FT Label
value MUST NOT be reused until all LDP FT peers to which the label
was passed have acknowledged the withdrawal of the FT Label, either
by an explicit LABEL WITHDRAW/LABEL RELEASE, exchange or implicitly
if the LDP session is reconnected after failure but without the FT
Reconnect Flag set. In the event that a session is not re-
established within the Reconnection Timeout, a label MAY become
available for re-use if it is not still in use on some other session.
5.1.2 FT Address Messages
If an LDP session uses the LDP FT enhancements, both LDP peers MUST
secure Address and Address Withdraw messages using FT Operation ACKs,
as described below. This avoids any ambiguity over whether an
Address is still valid after the LDP session is reconnected.
If an LSR determines that an Address message it sent on a previous
instantiation of a recovered LDP session is no longer valid, it MUST
explicitly issue an Address Withdraw for that address when the
session is reconnected.
If the FT Reconnect Flag is not set by both LDP peers upon
reconnection of an LDP session (i.e. state has not been preserved),
both LDP peers MUST consider all Addresses to have been withdrawn.
The LDP peers SHOULD issue new Address messages for all their valid
addresses, as specified in [RFC3036].
5.1.3 Label Resources Available Notifications
In LDP, it is possible that a downstream LSR may not have labels
available to respond to a Label Request. In this case, as specified
in RFC3036, the downstream LSR must respond with a Notification - No
Label Resources message. The upstream LSR then suspends asking for
new labels until it receives a Notification - Label Resources
Available message from the downstream LSR.
When the FT extensions are used on a session, implementations may
choose whether or not to secure the label resource state of their
peer. This choice impacts the number of LDP messages that will be
incorrectly routed to a peer with depleted resources on session re-
establishment, but does not otherwise impact interoperability.
For full preservation of state:
- The downstream LSR must preserve the label availability state
across a failover so that it remembers to send Notification -
Label Resources Available when the resources become available.
- The upstream LSR must recall the label availability state across
failover so that it can optimize not sending Label Requests when
it recovers.
- The downstream LSR must use sequence numbers on Notification -
Label Resources Available so that it can check that LSR A has
received the message and clear its secured state, or resend the
message if LSR A recovers without having received it.
However, the following options also exist:
- The downstream LSR may choose to not include a sequence number on
Notification - Label Resources Available. This means that on
session re-establishment it does not know what its peer thinks the
LSR's resource state is, because the Notification may or may not
have been delivered. Such an implementation MUST begin recovered
sessions by sending an additional Notification - Label Resources
Available to reset its peer.
- The upstream node may choose not to secure information about its
peer's resource state. It would acknowledge a Notification -
Label Resources Available, but would not save the information.
Such an implementation MUST assume that its peer's resource state
has been reset to Label Resources Available when the session is
re-established.
If the FT Reconnect Flag is not set by both LDP peers upon
reconnection of an LDP session (i.e. state has not been preserved),
both LDP peers MUST consider the label availability state to have
been reset as if the session had been set up for the first time.
5.2. FT Operation ACKs
Handshaking of FT LDP messages is achieved by use of ACKs.
Correlation between the original message and the ACK is by means of
the FT Sequence Number contained in the FT Protection TLV, and passed
back in the FT ACK TLV. The FT ACK TLV may be carried on any LDP
message that is sent on the TCP connection between LDP peers.
An LDP peer maintains a separate FT sequence number for each LDP
session in which it participates. The FT Sequence number is
incremented by one for each FT LDP message (i.e. containing the FT
Protection TLV) issued by this LSR on the FT LDP session with which
the FT sequence number is associated.
When an LDP peer receives a message containing the FT Protection TLV,
it MUST take steps to secure this message (or the state information
derived from processing the message). Once the message is secured,
it MUST be ACKed. However, there is no requirement on the LSR to
send this ACK immediately.
ACKs may be accumulated to reduce the message flow between LDP peers.
For example, if an LSR received FT LDP messages with sequence numbers
1, 2, 3, 4, it could send a single ACK with sequence number 4 to ACK
receipt, securing of all these messages. There is no protocol reason
why the number of ACKs accumulated, or the time for which an ACK is
deferred, should not be allowed to become relatively large.
ACKs MUST NOT be sent out of sequence, as this is incompatible with
the use of accumulated ACKs. Duplicate ACKs (that is two successive
messages that acknowledge the same sequence number) are acceptable.
If an LDP peer discovers that its sequence number space for a
specific session is full of un-acknowledged sequence numbers (because
its partner on the session has not acknowledged them in a timely
way), it cannot allocate a new sequence number for any further FT LPD
message. It SHOULD send a Notification message with the status code
'FT Seq Numbers Exhausted'.
5.3. Preservation of FT State
If the LDP FT enhancements are in use on an LDP session, each LDP
peer SHOULD NOT release the state information and resources
associated with FT Labels exchanged on that LDP session when the TCP
connection fails. This is contrary to [RFC3036], but allows label
operations on FT Labels to be completed after re-connection of the
TCP connection.
Both LDP peers on an LDP session that is using the LDP FT
enhancements SHOULD preserve the state information and resources they
hold for that LDP session as described below.
- An upstream LDP peer SHOULD release the resources (in particular
bandwidth) associated with a Sequence Numbered FT Label when it
initiates a Label Release or Label Abort message for the label.
The upstream LDP peer MUST preserve state information for the
Sequence Numbered FT Label, even if it releases the resources
associated with the label, as it may need to reissue the label
operation if the TCP connection is interrupted.
- An upstream LDP peer MUST release the state information and
resources associated with a Sequence Numbered FT Label when it
receives an acknowledgement to a Label Release or Label Abort
message that it sent for the label, or when it sends a Label
Release message in response to a Label Withdraw message received
from the downstream LDP peer.
- A downstream LDP peer SHOULD NOT release the resources associated
with a Sequence Numbered FT Label when it sends a Label Withdraw
message for the label as it has not yet received confirmation that
the upstream LDP peer has ceased to send data using the label.
The downstream LDP peer MUST NOT release the state information it
holds for the label as it may yet have to reissue the label
operation if the TCP connection is interrupted.
- A downstream LDP peer MUST release the resources and state
information associated with a Sequence Numbered FT Label when it
receives an acknowledgement to a Label Withdraw message for the
label.
- When the FT Reconnection Timeout expires, an LSR SHOULD release
all state information and resources from previous instantiations
of the (permanently) failed LDP session.
- Either LDP peer MAY elect to release state information based on
its internal knowledge of the loss of integrity of the state
information or an inability to pend (or queue) LDP operations (as
described in section 5.4.1, "LDP Operations During TCP Failure")
during a TCP failure. That is, the peer is not required to wait
for the duration of the FT Reconnection Timeout before releasing
state; the timeout provides an upper limit on the persistence of
state. However, in the event that a peer releases state before
the expiration of the Reconnection Timeout, it MUST NOT re-use any
label that was in use on the session until the Reconnection
Timeout has expired.
- When an LSR receives a Status TLV with the E-bit set in the status
code, which causes it to close the TCP connection, the LSR MUST
release all state information and resources associated with the
session. This behavior is mandated because it is impossible for
the LSR to predict the precise state and future behavior of the
partner LSR that set the E-bit without knowledge of the
implementation of that partner LSR.
Note that the 'Temporary Shutdown' status code does not have the
E-bit set, and MAY be used during maintenance or upgrade
operations to indicate that the LSR intends to preserve state
across a closure and re-establishment of the TCP session.
- If an LSR determines that it must release state for any single FT
Label during a failure of the TCP connection on which that label
was exchanged, it MUST release all state for all labels on the LDP
session.
The release of state information and resources associated with non-FT
labels is as described in [RFC3036].
Note that a Label Release and the acknowledgement to a Label Withdraw
may be received by a downstream LSR in any order. The downstream LSR
MAY release its resources upon receipt of the first message and MUST
release its resources upon receipt of the second message.
5.4. FT Procedure After TCP Failure
When an LSR discovers or is notified of a TCP connection failure it
SHOULD start an FT Reconnection Timer to allow a period for re-
connection of the TCP connection between the LDP peers.
The RECOMMENDED default value for this timer is 5 seconds. During
this time, failure must be detected and reported, new hardware may
need to be activated, software state must be audited, and a new TCP
session must be set up.
Once the TCP connection between LDP peers has failed, the active LSR
SHOULD attempt to re-establish the TCP connection. The mechanisms,
timers and retry counts to re-establish the TCP connection are an
implementation choice. It is RECOMMENDED that any attempt to re-
establish the connection should take into account the failover
processing necessary on the peer LSR, the nature of the network
between the LDP peers, and the FT Reconnection Timeout chosen on the
previous instantiation of the TCP connection (if any).
If the TCP connection cannot be re-established within the FT
Reconnection Timeout period, the LSR detecting this timeout SHOULD
release all state preserved for the failed LDP session. If the TCP
connection is subsequently re-established (for example, after a
further Hello exchange to set up a new LDP session), the LSR MUST set
the FT Reconnect Flag to 0 if it released the preserved state
information on this timeout event.
If the TCP connection is successfully re-established within the FT
Reconnection Timeout, both peers MUST re-issue LDP operations that
were interrupted by (that is, un-acknowledged as a result of) the TCP
connection failure. This procedure is described in section 5.5, "FT
Procedure After TCP Re-connection".
The Hold Timer for an FT LDP Session (see [RFC3036] section 2.5.5)
SHOULD be ignored while the FT Reconnection Timer is running. The
hold timer SHOULD be restarted when the TCP connection is re-
established.
5.4.1 FT LDP Operations During TCP Failure
When the LDP FT enhancements are in use for an LDP session, it is
possible for an LSR to determine that it needs to send an LDP message
to an LDP peer, but that the TCP connection to that peer is currently
down. These label operations affect the state of FT Labels preserved
for the failed TCP connection, so it is important that the state
changes are passed to the LDP peer when the TCP connection is
restored.
If an LSR determines that it needs to issue a new FT LDP operation to
an LDP peer to which the TCP connection is currently failed, it MUST
pend the operation (e.g. on a queue) and complete that operation with
the LDP peer when the TCP connection is restored, unless the label
operation is overridden by a subsequent additional operation during
the TCP connection failure (see section 5.5, "FT Procedure After TCP
Re-connection").
If, during TCP Failure, an LSR determines that it cannot pend an
operation which it cannot simply fail (for example, a Label Withdraw,
Release or Abort operation), it MUST NOT attempt to re-establish the
previous LDP session. The LSR MUST behave as if the Reconnection
Timer expired and release all state information with respect to the
LDP peer. An LSR may be unable (or unwilling) to pend operations;
for instance, if a major routing transition occurred while TCP was
inoperable between LDP peers, it might result in excessively large
numbers of FT LDP Operations. An LSR that releases state before the
expiration of the Reconnection Timeout MUST NOT re-use any label that
was in use on the session until the Reconnection Timeout has expired.
In ordered operation, received FT LDP operations that cannot be
correctly forwarded because of a TCP connection failure MAY be
processed immediately (provided sufficient state is kept to forward
the label operation) or pended for processing when the onward TCP
connection is restored and the operation can be correctly forwarded
upstream or downstream. Operations on existing FT Labels SHOULD NOT
be failed during TCP session failure.
It is RECOMMENDED that Label Request operations for new FT Labels not
be pended awaiting the re-establishment of TCP connection that is
awaiting recovery at the time the LSR determines that it needs to
issue the Label Request message. Instead, such Label Request
operations SHOULD be failed and, if necessary, a notification message
containing the 'No LDP Session' status code sent upstream.
Label Requests for new non-FT Labels MUST be rejected during TCP
connection failure, as specified in [RFC3036].
5.5. FT Procedure After TCP Re-connection
The FT operation handshaking described above means that all state
changes for Sequence Numbered FT Labels and Address messages are
confirmed or reproducible at each LSR.
If the TCP connection between LDP peers fails but is re-connected
within the FT Reconnection Timeout, and both LSRs have indicated they
will be re-establishing the previous LDP session, both LDP peers on
the connection MUST complete any label operations for Sequence
Numbered FT Labels that were interrupted by the failure and re-