filtering, beyond that which is done by the PEs.
This additional filtering would not require
coordination with the SP.
On the other hand, using BGP may be something new for the CE
administrators.
If a site is not in a transit VPN, note that it need not have a
unique Autonomous System Number (ASN). Every CE whose site is
not in a transit VPN can use the same ASN. This can be chosen
from the private ASN space, and it will be stripped out by the
PE. Routing loops are prevented by use of the Site of Origin
attribute (see below).
What if a set of sites constitutes a transit VPN? This will
generally be the case only if the VPN is itself an Internet
Service Provider’s (ISP’s) network, where the ISP is itself
buying backbone services from another SP. The latter SP may be
called a "carrier’s carrier". In this case, the best way to
provide the VPN is to have the CE routers support MPLS, and to
use the technique described in Section 9.
When we do not need to distinguish among the different ways in which
a PE can be informed of the address prefixes that exist at a given
site, we will simply say that the PE has "learned" the routes from
that site. This includes the case where the PE has been manually
configured with the routes.
Before a PE can redistribute a VPN-IPv4 route learned from a site, it
must assign a Route Target attribute (see Section 4.3.1) to the
route, and it may assign a Site of Origin attribute to the route.
The Site of Origin attribute, if used, is encoded as a Route Origin
Extended Community [BGP-EXTCOMM]. The purpose of this attribute is
to uniquely identify the set of routes learned from a particular
site. This attribute is needed in some cases to ensure that a route
learned from a particular site via a particular PE/CE connection is
not distributed back to the site through a different PE/CE
connection. It is particularly useful if BGP is being used as the
PE/CE protocol, but different sites have not been assigned distinct
ASNs.
8. How CEs Learn Routes from PEs
In this section, we assume that the CE device is a router.
If the PE places a particular route in the VRF it uses to route
packets received from a particular CE, then in general, the PE may
distribute that route to the CE. Of course, the PE may distribute
that route to the CE only if this is permitted by the rules of the
PE/CE protocol. (For example, if a particular PE/CE protocol has
"split horizon", certain routes in the VRF cannot be redistributed
back to the CE.) We add one more restriction on the distribution of
routes from PE to CE: if a route’s Site of Origin attribute
identifies a particular site, that route must never be redistributed
to any CE at that site.
In most cases, however, it will be sufficient for the PE to simply
distribute the default route to the CE. (In some cases, it may even
be sufficient for the CE to be configured with a default route
pointing to the PE.) This will generally work at any site that does
not itself need to distribute the default route to other sites.
(E.g., if one site in a corporate VPN has the corporation’s access to
the Internet, that site might need to have default distributed to the
other site, but one could not distribute default to that site
itself.)
Whatever procedure is used to distribute routes from CE to PE will
also be used to distribute routes from PE to CE.
9. Carriers’ Carriers
Sometimes a VPN may actually be the network of an ISP, with its own
peering and routing policies. Sometimes a VPN may be the network of
an SP that is offering VPN services in turn to its own customers.
VPNs like these can also obtain backbone service from another SP, the
"carrier’s carrier", using essentially the same methods described in
this document. However, it is necessary in these cases that the CE
routers support MPLS. In particular:
- The CE routers should distribute to the PE routers ONLY those
routes that are internal to the VPN. This allows the VPN to be
handled as a stub VPN.
- The CE routers should support MPLS, in that they should be able
to receive labels from the PE routers, and send labeled packets
to the PE routers. They do not need to distribute labels of
their own, though.
- The PE routers should distribute, to the CE routers, labels for
the routes they distribute to the CE routers.
The PE must not distribute the same label to two different CEs
unless one of the following conditions holds:
* The two CEs are associated with exactly the same set of VRFs;
* The PE maintains a different Incoming Label Map ([MPLS-ARCH])
for each CE.
Further, when the PE receives a labeled packet from a CE, it must
verify that the top label is one that was distributed to that CE.
- Routers at the different sites should establish BGP connections
among themselves for the purpose of exchanging external routes
(i.e., routes that lead outside of the VPN).
- All the external routes must be known to the CE routers.
Then when a CE router looks up a packet’s destination address, the
routing lookup will resolve to an internal address, usually the
address of the packet’s BGP next hop. The CE labels the packet
appropriately and sends the packet to the PE. The PE, rather than
looking up the packet’s IP destination address in a VRF, uses the
packet’s top MPLS label to select the BGP next hop. As a result, if
the BGP next hop is more than one hop away, the top label will be
replaced by two labels, a tunnel label and a VPN route label. If the
BGP next hop is one hop away, the top label may be replaced by just
the VPN route label. If the ingress PE is also the egress PE, the
top label will just be popped. When the packet is sent from its
egress PE to a CE, the packet will have one fewer MPLS labels than it
had when it was first received by its ingress PE.
In the above procedure, the CE routers are the only routers in the
VPN that need to support MPLS. If, on the other hand, all the
routers at a particular VPN site support MPLS, then it is no longer
required that the CE routers know all the external routes. All that
is required is that the external routes be known to whatever routers
are responsible for putting the label stack on a hitherto unlabeled
packet and that there be label switched path that leads from those
routers to their BGP peers at other sites. In this case, for each
internal route that a CE router distributes to a PE router, it must
also distribute a label.
10. Multi-AS Backbones
What if two sites of a VPN are connected to different Autonomous
Systems (e.g., because the sites are connected to different SPs)?
The PE routers attached to that VPN will then not be able to maintain
IBGP connections with each other, or with a common route reflector.
Rather, there needs to be some way to use EBGP to distribute VPN-IPv4
addresses.
There are a number of different ways of handling this case, which we
present in order of increasing scalability.
a) VRF-to-VRF connections at the AS (Autonomous System) border
routers.
In this procedure, a PE router in one AS attaches directly to a
PE router in another. The two PE routers will be attached by
multiple sub-interfaces, at least one for each of the VPNs
whose routes need to be passed from AS to AS. Each PE will
treat the other as if it were a CE router. That is, the PEs
associate each such sub-interface with a VRF, and use EBGP to
distribute unlabeled IPv4 addresses to each other.
This is a procedure that "just works", and that does not
require MPLS at the border between ASes. However, it does not
scale as well as the other procedures discussed below.
b) EBGP redistribution of labeled VPN-IPv4 routes from AS to
neighboring AS.
In this procedure, the PE routers use IBGP to redistribute
labeled VPN-IPv4 routes either to an Autonomous System Border
Router (ASBR), or to a route reflector of which an ASBR is a
client. The ASBR then uses EBGP to redistribute those labeled
VPN-IPv4 routes to an ASBR in another AS, which in turn
distributes them to the PE routers in that AS, or perhaps to
another ASBR which in turn distributes them, and so on.
When using this procedure, VPN-IPv4 routes should only be
accepted on EBGP connections at private peering points, as part
of a trusted arrangement between SPs. VPN-IPv4 routes should
neither be distributed to nor accepted from the public
Internet, or from any BGP peers that are not trusted. An ASBR
should never accept a labeled packet from an EBGP peer unless
it has actually distributed the top label to that peer.
If there are many VPNs having sites attached to different
Autonomous Systems, there does not need to be a single ASBR
between those two ASes that holds all the routes for all the
VPNs; there can be multiple ASBRs, each of which holds only the
routes for a particular subset of the VPNs.
This procedure requires that there be a label switched path
leading from a packet’s ingress PE to its egress PE. Hence the
appropriate trust relationships must exist between and among
the set of ASes along the path. Also, there must be agreement
among the set of SPs as to which border routers need to receive
routes with which Route Targets.
c) Multi-hop EBGP redistribution of labeled VPN-IPv4 routes
between source and destination ASes, with EBGP redistribution
of labeled IPv4 routes from AS to neighboring AS.
In this procedure, VPN-IPv4 routes are neither maintained nor
distributed by the ASBRs. An ASBR must maintain labeled IPv4
/32 routes to the PE routers within its AS. It uses EBGP to
distribute these routes to other ASes. ASBRs in any transit
ASes will also have to use EBGP to pass along the labeled /32
routes. This results in the creation of a label switched path
from the ingress PE router to the egress PE router. Now PE
routers in different ASes can establish multi-hop EBGP
connections to each other, and can exchange VPN-IPv4 routes
over those connections.
If the /32 routes for the PE routers are made known to the P
routers of each AS, everything works normally. If the /32
routes for the PE routers are NOT made known to the P routers
(other than the ASBRs), then this procedure requires a packet’s
ingress PE to put a three-label stack on it. The bottom label
is assigned by the egress PE, corresponding to the packet’s
destination address in a particular VRF. The middle label is
assigned by the ASBR, corresponding to the /32 route to the
egress PE. The top label is assigned by the ingress PE’s IGP
Next Hop, corresponding to the /32 route to the ASBR.
To improve scalability, one can have the multi-hop EBGP
connections exist only between a route reflector in one AS and
a route reflector in another. (However, when the route
reflectors distribute routes over this connection, they do not
modify the BGP next hop attribute of the routes.) The actual
PE routers would then only have IBGP connections to the route
reflectors in their own AS.
This procedure is very similar to the "carrier’s carrier"
procedures described in Section 9. Like the previous
procedure, it requires that there be a label switched path
leading from a packet’s ingress PE to its egress PE.
11. Accessing the Internet from a VPN
Many VPN sites will need to be able to access the public Internet, as
well as to access other VPN sites. The following describes some of
the alternative ways of doing this.
1. In some VPNs, one or more of the sites will obtain Internet
access by means of an "Internet gateway" (perhaps a firewall)
attached to a non-VRF interface to an ISP. The ISP may or may
not be the same organization as the SP that is providing the
VPN service. Traffic to/from the Internet gateway would then
be routed according to the PE router’s default forwarding
table.
In this case, the sites that have Internet access may be
distributing a default route to their PEs, which in turn
redistribute it to other PEs and hence into other sites of the
VPN. This provides Internet access for all of the VPN’s sites.
In order to properly handle traffic from the Internet, the ISP
must distribute, to the Internet, routes leading to addresses
that are within the VPN. This is completely independent of any
of the route distribution procedures described in this
document. The internal structure of the VPN will in general
not be visible from the Internet; such routes would simply lead
to the non-VRF interface that attaches to the VPN’s Internet
gateway.
In this model, there is no exchange of routes between a PE
router’s default forwarding table and any of its VRFs. VPN
route distribution procedures and Internet route distribution
procedures are completely independent.
Note that although some sites of the VPN use a VRF interface to
communicate with the Internet, ultimately all packets to/from
the Internet traverse a non-VRF interface before
leaving/entering the VPN, so we refer to this as "non-VRF
Internet access".
Note that the PE router to which the non-VRF interface attaches
does not necessarily need to maintain all the Internet routes
in its default forwarding table. The default forwarding table
could have as few as one route, "default", which leads to
another router (probably an adjacent one) that has the Internet
routes. A variation of this scheme is to tunnel packets
received over the non-VRF interface from the PE router to
another router, where this other router maintains the full set
of Internet routes.
2. Some VPNs may obtain Internet access via a VRF interface ("VRF
Internet access"). If a packet is received by a PE over a VRF
interface, and if the packet’s destination address does not
match any route in the VRF, then it may be matched against the
PE’s default forwarding table. If a match is made there, the
packet can be forwarded natively through the backbone to the
Internet, instead of being forwarded by MPLS.
In order for traffic to flow natively in the opposite direction
(from Internet to VRF interface), some of the routes from the
VRF must be exported to the Internet forwarding table.
Needless to say, any such routes must correspond to globally
unique addresses.
In this scheme, the default forwarding table might have the
full set of Internet routes, or it might have as little as a
single default route leading to another router that does have
the full set of Internet routes in its default forwarding
table.
3. Suppose the PE has the capability to store "non-VPN routes" in
a VRF. If a packet’s destination address matches a "non-VPN
route", then the packet is transmitted natively, rather than
being transmitted via MPLS. If the VRF contains a non-VPN
default route, all packets for the public Internet will match
it, and be forwarded natively to the default route’s next hop.
At that next hop, the packets’ destination addresses will be
looked up in the default forwarding table, and may match more
specific routes.
This technique would only be available if none of the CE
routers is distributing a default route.
4. It is also possible to obtain Internet access via a VRF
interface by having the VRF contain the Internet routes.
Compared with model 2, this eliminates the second lookup, but
it has the disadvantage of requiring the Internet routes to be
replicated in each such VRF.
If this technique is used, the SP may want to make its
interface to the Internet be a VRF interface, and to use the
techniques of Section 4 to distribute Internet routes, as VPN-
IPv4 routes, to other VRFs.
It should be clearly understood that by default, there is no exchange
of routes between a VRF and the default forwarding table. This is
done ONLY upon agreement between a customer and an SP, and only if it
suits the customer’s policies.
12. Management VPNs
This specification does not require that the sub-interface connecting
a PE router and a CE router be a "numbered" interface. If it is a
numbered interface, this specification allows the addresses assigned
to the interface to come from either the address space of the VPN or
the address space of the SP.
If a CE router is being managed by the Service Provider, then the
Service Provider will likely have a network management system that
needs to be able to communicate with the CE router. In this case,
the addresses assigned to the sub-interface connecting the CE and PE
routers should come from the SP’s address space, and should be unique
within that space. The network management system should itself
connect to a PE router (more precisely, be at a site that connects to
a PE router) via a VRF interface. The address of the network
management system will be exported to all VRFs that are associated
with interfaces to CE routers that are managed by the SP. The
addresses of the CE routers will be exported to the VRF associated
with the network management system, but not to any other VRFs.
This allows communication between the CE and network management
system, but does not allow any undesired communication to or among
the CE routers.
One way to ensure that the proper route import/exports are done is to
use two Route Targets; call them T1 and T2. If a particular VRF
interface attaches to a CE router that is managed by the SP, then
that VRF is configured to:
- import routes that have T1 attached to them, and
- attach T2 to addresses assigned to each end of its VRF
interfaces.
If a particular VRF interface attaches to the SP’s network management
system, then that VRF is configured to attach T1 to the address of
that system, and to import routes that have T2 attached to them.
13. Security Considerations
13.1. Data Plane
By security in the "data plane", we mean protection against the
following possibilities:
- Packets from within a VPN travel to a site outside the VPN, other
than in a manner consistent with the policies of the VPN.
- Packets from outside a VPN enter one of the VPN’s sites, other
than in a manner consistent with the policies of the VPN.
Under the following conditions:
1. a backbone router does not accept labeled packets over a
particular data link, unless it is known that that data link
attaches only to trusted systems, or unless it is known that
such packets will leave the backbone before the IP header or
any labels lower in the stack will be inspected, and
2. labeled VPN-IPv4 routes are not accepted from untrusted or
unreliable routing peers,
3. no successful attacks have been mounted on the control plane,
the data plane security provided by this architecture is virtually
identical to that provided to VPNs by Frame Relay or ATM backbones.
If the devices under the control of the SP are properly configured,
data will not enter or leave a VPN unless authorized to do so.
Condition 1 above can be stated more precisely. One should discard a
labeled packet received from a particular neighbor unless one of the
following two conditions holds:
- the packet’s top label has a label value that the receiving
system has distributed to that neighbor, or
- the packet’s top label has a label value that the receiving
system has distributed to a system beyond that neighbor (i.e.,
when it is known that the path from the system to which the label
was distributed to the receiving system may be via that
neighbor).
Condition 2 above is of most interest in the case of inter-provider
VPNs (see Section 10). For inter-provider VPNs constructed according
to scheme b) of Section 10, condition 2 is easily checked. (The
issue of security when scheme (c) of Section 10 is used is for
further study.)
It is worth noting that the use of MPLS makes it much simpler to
provide data plane security than might be possible if one attempted
to use some form of IP tunneling in place of the MPLS outer label.
It is a simple matter to have one’s border routers refuse to accept a
labeled packet unless the first of the above conditions applies to
it. It is rather more difficult to configure a router to refuse to
accept an IP packet if that packet is an IP tunneled packet whose
destination address is that of a PE router; certainly, this is not
impossible to do, but it has both management and performance
implications.
MPLS-in-IP and MPLS-in-GRE tunneling are specified in