Request for Comments: 4274 K. Patel
Category: Informational Cisco Systems
January 2006
BGP-4 Protocol Analysis
Status of This Memo
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2006).
Abstract
The purpose of this report is to document how the requirements for
publication of a routing protocol as an Internet Draft Standard have
been satisfied by Border Gateway Protocol version 4 (BGP-4).
This report satisfies the requirement for "the second report", as
described in Section 6.0 of RFC 1264. In order to fulfill the
requirement, this report augments RFC 1774 and summarizes the key
features of BGP-4, as well as analyzes the protocol with respect to
scaling and performance.
Table of Contents
1. Introduction ....................................................2
2. Key Features and Algorithms of BGP ..............................3
2.1. Key Features ...............................................3
2.2. BGP Algorithms .............................................4
2.3. BGP Finite State Machine (FSM) .............................4
3. BGP Capabilities ................................................5
4. BGP Persistent Peer Oscillations ................................6
5. Implementation Guidelines .......................................6
6. BGP Performance Characteristics and Scalability .................6
6.1. Link Bandwidth and CPU Utilization .........................7
7. BGP Policy Expressiveness and its Implications ..................9
7.1. Existence of Unique Stable Routings .......................10
7.2. Existence of Stable Routings ..............................11
8. Applicability ..................................................12
9. Acknowledgements ...............................................12
10. Security Considerations .......................................12
11. References ....................................................13
11.1. Normative References ....................................13
11.2. Informative References ..................................14
1. Introduction
BGP-4 is an inter-autonomous system routing protocol designed for
TCP/IP internets. Version 1 of BGP-4 was published in [RFC1105].
Since then, BGP versions 2, 3, and 4 have been developed. Version 2
was documented in [RFC1163]. Version 3 is documented in [RFC1267].
Version 4 is documented in [BGP4] (version 4 of BGP will hereafter be
referred to as BGP). The changes between versions are explained in
Appendix A of [BGP4]. Possible applications of BGP in the Internet
are documented in [RFC1772].
BGP introduced support for Classless Inter-Domain Routing (CIDR)
[RFC1519]. Because earlier versions of BGP lacked the support for
CIDR, they are considered obsolete and unusable in today’s Internet.
The purpose of this report is to document how the requirements for
publication of a routing protocol as an Internet Draft Standard have
been satisfied by Border Gateway Protocol version 4 (BGP-4).
This report satisfies the requirement for "the second report", as
described in Section 6.0 of [RFC1264]. In order to fulfill the
requirement, this report augments [RFC1774] and summarizes the key
features of BGP-4, as well as analyzes the protocol with respect to
scaling and performance.
2. Key Features and Algorithms of BGP
This section summarizes the key features and algorithms of BGP. BGP
is an inter-autonomous system routing protocol; it is designed to be
used between multiple autonomous systems. BGP assumes that routing
within an autonomous system is done by an intra-autonomous system
routing protocol. BGP also assumes that data packets are routed from
source towards destination independent of the source. BGP does not
make any assumptions about intra-autonomous system routing protocols
deployed within the various autonomous systems. Specifically, BGP
does not require all autonomous systems to run the same intra-
autonomous system routing protocol (i.e., interior gateway protocol
or IGP).
Finally, note that BGP is a real inter-autonomous system routing
protocol; and, as such, it imposes no constraints on the underlying
interconnect topology of the autonomous systems. The information
exchanged via BGP is sufficient to construct a graph of autonomous
systems connectivity from which routing loops may be pruned, and many
routing policy decisions at the autonomous system level may be
enforced.
2.1. Key Features
The key features of the protocol are the notion of path attributes
and aggregation of Network Layer Reachability Information (NLRI).
Path attributes provide BGP with flexibility and extensibility. Path
attributes are either well-known or optional. The provision for
optional attributes allows experimentation that may involve a group
of BGP routers without affecting the rest of the Internet. New
optional attributes can be added to the protocol in much the same way
that new options are added to, for example, the Telnet protocol
[RFC854].
One of the most important path attributes is the Autonomous System
Path, or AS_PATH. As the reachability information traverses the
Internet, this (AS_PATH) information is augmented by the list of
autonomous systems that have been traversed thus far, forming the
AS_PATH. The AS_PATH allows straightforward suppression of the
looping of routing information. In addition, the AS_PATH serves as a
powerful and versatile mechanism for policy-based routing.
BGP enhances the AS_PATH attribute to include sets of autonomous
systems as well as lists via the AS_SET attribute. This extended
format allows generated aggregate routes to carry path information
from the more specific routes used to generate the aggregate. It
should be noted, however, that as of this writing, AS_SETs are rarely
used in the Internet [ROUTEVIEWS].
2.2. BGP Algorithms
BGP uses an algorithm that is neither a pure distance vector
algorithm or a pure link state algorithm. Instead, it uses a
modified distance vector algorithm, referred to as a "Path Vector"
algorithm. This algorithm uses path information to avoid traditional
distance vector problems. Each route within BGP pairs information
about the destination with path information to that destination.
Path information (also known as AS_PATH information) is stored within
the AS_PATH attribute in BGP. The path information assists BGP in
detecting AS loops, thereby allowing BGP speakers to select loop-free
routes.
BGP uses an incremental update strategy to conserve bandwidth and
processing power. That is, after initial exchange of complete
routing information, a pair of BGP routers exchanges only the changes
to that information. Such an incremental update design requires
reliable transport between a pair of BGP routers in order to function
correctly. BGP solves this problem by using TCP for reliable
transport.
In addition to incremental updates, BGP has added the concept of
route aggregation so that information about groups of destinations
that use hierarchical address assignment (e.g., CIDR) may be
aggregated and sent as a single Network Layer Reachability (NLRI).
Finally, note that BGP is a self-contained protocol. That is, BGP
specifies how routing information is exchanged, both between BGP
speakers in different autonomous systems, and between BGP speakers
within a single autonomous system.
2.3. BGP Finite State Machine (FSM)
The BGP FSM is a set of rules that is applied to a BGP speaker’s set
of configured peers for the BGP operation. A BGP implementation
requires that a BGP speaker must connect to and listen on TCP port
179 for accepting any new BGP connections from its peers. The BGP
Finite State Machine, or FSM, must be initiated and maintained for
each new incoming and outgoing peer connection. However, in steady
state operation, there will be only one BGP FSM per connection per
peer.
There may be a short period during which a BGP peer may have separate
incoming and outgoing connections resulting in the creation of two
different BGP FSMs relating to a peer (instead of one). This can be
resolved by following the BGP connection collision rules defined in
the [BGP4] specification.
The BGP FSM has the following states associated with each of its
peers:
IDLE: State when BGP peer refuses any incoming connections.
CONNECT: State in which BGP peer is waiting for its TCP
connection to be completed.
ACTIVE: State in which BGP peer is trying to acquire a peer
by listening and accepting TCP connection.
OPENSENT: BGP peer is waiting for OPEN message from its peer.
OPENCONFIRM: BGP peer is waiting for KEEPALIVE or NOTIFICATION
message from its peer.
ESTABLISHED: BGP peer connection is established and exchanges
UPDATE, NOTIFICATION, and KEEPALIVE messages with its
peer.
There are a number of BGP events that operate on the above mentioned
states of the BGP FSM for BGP peers. Support of these BGP events is
either mandatory or optional. These events are triggered by the
protocol logic as part of the BGP or by using an operator
intervention via a configuration interface to the BGP protocol.
These BGP events are of following types: Optional events linked to
Optional Session attributes, Administrative Events, Timer Events, TCP
Connection-based Events, and BGP Message-based Events. Both the FSM
and the BGP events are explained in detail in [BGP4].
3. BGP Capabilities
The BGP capability mechanism [RFC3392] provides an easy and flexible
way to introduce new features within the protocol. In particular,
the BGP capability mechanism allows a BGP speaker to advertise to its
peers during startup various optional features supported by the
speaker (and receive similar information from the peers). This
allows the base BGP to contain only essential functionality, while
providing a flexible mechanism for signaling protocol extensions.
4. BGP Persistent Peer Oscillations
Whenever a BGP speaker detects an error in a peer connection, it
shuts down the peer and changes its FSM state to IDLE. BGP speaker
requires a Start event to re-initiate an idle peer connection. If
the error remains persistent and BGP speaker generates a Start event
automatically, then it may result in persistent peer flapping.
Although peer oscillation is found to be wide-spread in BGP
implementations, methods for preventing persistent peer oscillations
are outside the scope of base BGP specification.
5. Implementation Guidelines
A robust BGP implementation is "work conserving". This means that if
the number of prefixes is bounded, arbitrarily high levels of route
change can be tolerated. High levels can be tolerated with bounded
impact on route convergence for occasional changes in generally
stable routes.
A robust implementation of BGP should have the following
characteristics:
1. It is able to operate in almost arbitrarily high levels of
route flap without losing peerings (failing to send
keepalives) or losing other protocol adjacencies as a result
of BGP load.
2. Instability of a subset of routes should not affect the route
advertisements or forwarding associated with the set of stable
routes.
3. Instability should not be caused by peers with high levels of
instability or with different CPU speed or load that result in
faster or slower processing of routes. These instable peers
should have a bounded impact on the convergence time for
generally stable routes.
Numerous robust BGP implementations exist. Producing a robust
implementation is not a trivial matter, but is clearly achievable.
6. BGP Performance Characteristics and Scalability
In this section, we provide "order of magnitude" answers to the
questions of how much link bandwidth, router memory and router CPU
cycles BGP will consume under normal conditions. In particular, we
will address the scalability of BGP and its limitations.
6.1. Link Bandwidth and CPU Utilization
Immediately after the initial BGP connection setup, BGP peers
exchange complete sets of routing information. If we denote the
total number of routes in the Internet as N, the total path
attributes (for all N routes) received from a peer as A, and assume
that the networks are uniformly distributed among the autonomous
systems, then the worst-case amount of bandwidth consumed during the
initial exchange between a pair of BGP speakers (P) is
BW = O((N + A) * P)
BGP-4 was created specifically to reduce the size of the set of NLRI
entries, which has to be carried and exchanged by border routers.
The aggregation scheme, defined in [RFC1519], describes the
provider-based aggregation scheme in use in today’s Internet.
Due to the advantages of advertising a few large aggregate blocks
(instead of many smaller class-based individual networks), it is
difficult to estimate the actual reduction in bandwidth and
processing that BGP-4 has provided over BGP-3. If we simply
enumerate all aggregate blocks into their individual, class-based
networks, we would not take into account "dead" space that has been
reserved for future expansion. The best metric for determining the
success of BGP’s aggregation is to sample the number NLRI entries in
the globally-connected Internet today, and compare it to growth rates
that were projected before BGP was deployed.
At the time of this writing, the full set of exterior routes carried
by BGP is approximately 134,000 network entries [ROUTEVIEWS].
6.1.1. CPU Utilization
An important and fundamental feature of BGP is that BGP’s CPU
utilization depends only on the stability of its network which
relates to BGP in terms of BGP UPDATE message announcements. If the
BGP network is stable, all the BGP routers within its network are in
the steady state. Thus, the only link bandwidth and router CPU
cycles consumed by BGP are due to the exchange of the BGP KEEPALIVE
messages. The KEEPALIVE messages are exchanged only between peers.
The suggested frequency of the exchange is 30 seconds. The KEEPALIVE
messages are quite short (19 octets) and require virtually no
processing. As a result, the bandwidth consumed by the KEEPALIVE
messages is about 5 bits/sec. Operational experience confirms that
the overhead (in terms of bandwidth and CPU) associated with the
KEEPALIVE messages should be viewed as negligible.
During periods of network instability, BGP routers within the network
are generating routing updates that are exchanged using the BGP
UPDATE messages. The greatest overhead per UPDATE message occurs
when each UPDATE message contains only a single network. It should
be pointed out that, in practice, routing changes exhibit strong
locality with respect to the route attributes. That is, routes that
change are likely to have common route attributes. In this case,
multiple networks can be grouped into a single UPDATE message, thus
significantly reducing the amount of bandwidth required (see also
Appendix F.1 of [BGP4]).
6.1.2. Memory Requirements
To quantify the worst-case memory requirements for BGP, we denote the
total number of networks in the Internet as N, the mean AS distance
of the Internet as M (distance at the level of an autonomous system,
expressed in terms of the number of autonomous systems), the total
number of unique AS paths as A. Then the worst-case memory
requirements (MR) can be expressed as
MR = O(N + (M * A))
Because a mean AS distance M is a slow moving function of the
interconnectivity ("meshiness") of the Internet, for all practical
purposes the worst-case router memory requirements are on the order
of the total number of networks in the Internet multiplied by the
number of peers that the local system is peering with. We expect
that the total number of networks in the Internet will grow much
faster than the average number of peers per router. As a result,
BGP’s memory-scaling properties are linearly related to the total
number of networks in the Internet.
The following table illustrates typical memory requirements of a
router running BGP. We denote the average number of routes
advertised by each peer as N, the total number of unique AS paths as
A, the mean AS distance of the Internet as M (distance at the level
of an autonomous system, expressed in terms of the number of
autonomous systems), the number of octets required to store a network
as R, and the number of bytes required to store one AS in an AS path
as P. It is assumed that each network is encoded as four bytes, each
AS is encoded as two bytes, and each networks is reachable via some
fraction of all the peers (# BGP peers/per net). For purposes of the
estimates here, we will calculate MR = (((N * R) + (M * A) * P) * S).
# Networks Mean AS Distance # ASes # BGP peers/per net Memory Req
(N) (M) (A) (P) (MR)
---------- ---------------- ------ ------------------- -------------
100,000 20 3,000 20 10,400,000
100,000 20 15,000 20 20,000,000
120,000 10 15,000 100 78,000,000
140,000 15 20,000 100 116,000,000
In analyzing BGP’s memory requirements, we focus on the size of the
BGP RIB table (ignoring implementation details). In particular, we
derive upper bounds for the size of the BGP RIB table. For example,
at the time of this writing, the BGP RIB tables of a typical backbone
router carry on the order of 120,000 entries. Given this number, one
might ask whether it would be possible to have a functional router
with a table containing 1,000,000 entries. Clearly, the answer to
this question is more related to how BGP is implemented. A robust
BGP implementation with a reasonable CPU and memory should not have
issues scaling to such limits.
7. BGP Policy Expressiveness and its Implications
BGP is unique among deployed IP routing protocols in that routing is
determined using semantically rich routing policies. Although
routing policies are usually the first BGP issue that comes to a
network operator’s mind, it is important to note that the languages
and techniques for specifying BGP routing policies are not part of
the protocol specification (see [RFC2622] for an example of such a
policy language). In addition, the BGP specification contains few
restrictions, explicit or implicit, on routing policy languages.
These languages have typically been developed by vendors and have
evolved through interactions with network engineers in an environment
lacking vendor-independent standards.
The complexity of typical BGP configurations, at least in provider
networks, has been steadily increasing. Router vendors typically
provide hundreds of special commands for use in the configuration of
BGP, and this command set is continually expanding. For example, BGP
communities [RFC1997] allow policy writers to selectively attach tags
to routes and to use these to signal policy information to other
BGP-speaking routers. Many providers allow customers, and sometimes
peers, to send communities that determine the scope and preference of
their routes. Due to these developments, the task of writing BGP
configurations has increasingly more aspects associated with open-
ended programming. This has allowed network operators to encode
complex policies in order to address many unforeseen situations, and
has opened the door for a great deal of creativity and
experimentation in routing policies. This policy flexibility is one
of the main reasons why BGP is so well suited to the commercial
environment of the current Internet.
However, this rich policy expressiveness has come with a cost that is
often not recognized. In particular, it is possible to construct
locally defined routing policies that can lead to protocol divergence
and unexpected global routing anomalies such as (unintended) non-
determinism. If the interacting policies causing such anomalies are
defined in different autonomous systems, then these problems can be
very difficult to debug and correct. In the following sections, we
describe two such cases relating to the existence (or lack thereof)
of stable routings.
7.1. Existence of Unique Stable Routings
One can easily construct sets of policies for which BGP cannot
guarantee that stable routings are unique. This is illustrated by
the following simple example. Consider four Autonomous Systems, AS1,
AS2, AS3, and AS4. AS1 and AS2 are peers, and they provide transit
for AS3 and AS4, respectively. Suppose AS3 provides transit for AS4
(in this case AS3 is a customer of AS1, and AS4 is a multihomed
customer of both AS3 and AS2). AS4 may want to use the link to AS3
as a "backup" link, and sends AS3 a community value that AS3 has
configured to lower the preference of AS4’s routes to a level below
that of its upstream provider, AS1. The intended "backup routing" to
AS4 is illustrated here:
AS1 ------> AS2
/|\ |
| |
| |
| \|/
AS3 ------- AS4
That is, the AS3-AS4 link is intended to be used only when the AS2-
AS4 link is down (for outbound traffic, AS4 simply gives routes from
AS2 a higher local preference). This is a common scenario in today’s
Internet. But note that this configuration has another stable
solution:
AS1 ------- AS2
| |
| |
| |
\|/ \|/
AS3 ------> AS4
In this case, AS3 does not translate the "depref my route" community
received from AS4 into a "depref my route" community for AS1.
Therefore, if AS1 hears the route to AS4 that transits AS3, it will
prefer that route (because AS3 is a customer). This state could be
reached, for example, by starting in the "correct" backup routing
shown first, bringing down the AS2-AS4 BGP session, and then bringing
it back up. In general, BGP has no way to prefer the "intended"
solution over the anomalous one. The solution picked will depend on
the unpredictable order of BGP messages.
While this example is relatively simple, many operators may fail to
recognize that the true source of the problem is that the BGP
policies of ASes can interact in unexpected ways, and that these
interactions can result in multiple stable routings. One can imagine
that the interactions could be much more complex in the real
Internet. We suspect that such anomalies will only become more
common as BGP continues to evolve with richer policy expressiveness.