traffic sent on one PE/CE link as opposed to the other. One can also
make use of the Link Bandwidth extended community [BGP-EXT-COMM] to
control how traffic is distributed among multiple egress PE/CE links.
The VPN scheme is of course compatible with the use of traffic
engineering techniques, Resource Reservation Protocol - Traffic
Engineering (RSVP-TE) based or otherwise, in the backbone network.
10. Service Access
10.1. Internet Access
Internet access and VPN access are possible from the same site. This
is even possible over the same interface, as long as the VPN’s
internal addresses are distinct from the addresses of the systems
that must be reached via the Internet. This requires only that
Internet routes as well as VPN routes be imported into the VRF
associated with that interface. This may be as simple as putting a
default route to the Internet into that VRF.
The "route to the Internet" that is in a particular VRF need not lead
directly to the Internet; it may lead to a firewall or other security
device at another site of the VPN. The VPN customer can cause this
to happen simply by exporting a default route from the site with the
firewall. Generally, a site with a firewall will use a different
virtual interface for Internet access than for VPN access, since the
firewall needs to distinguish the "clean interface" from the "dirty
interface".
In such a configuration, the customer would export his routes to the
Internet via the firewall’s dirty interface, but would export the
same routes to the VPN via the clean interface. Thus, all traffic
from the Internet would come through the dirty interface, then
through the firewall, and possibly go to another VPN site though the
clean interface. This also allows any necessary Network Address
Translation (NAT) functionality to be done in the firewall.
10.2. Other Services
Any externally provided service can be accessed from the VPN,
provided that it can be addressed with an address that is not
otherwise in use within the VPN. Access can be firewalled or non-
firewalled. If the client accessing the service does not have a
globally unique IP address, and a single server provides a service to
multiple VPNs, NAT will have to be applied to the client’s packets
before they reach the server. This can be done at a customer site,
or by a VRF-specific NAT function in a PE router.
11. SP Routing
Routing through the backbone is independent of the VPN scheme and is
unaffected by the presence or absence of VPNs. The only impact is
that the backbone routing must carry routes to the PE routers.
The VPN routes themselves are carried in BGP as a distinct address
family, different than the address family that is used to carry
"ordinary" IP routes. These routes are passed from PE router to
Route Reflector to PE router, and are never seen by the P routers.
The Route Reflectors that carry the VPN routes can be entirely
separate from the Route Reflectors that carry the "ordinary" IP
routes.
The fact that two PE routers support a common VPN does not require
those PE routers to form an IGP routing adjacency between themselves.
The number of adjacencies in the backbone IGP is independent of and
unrelated to the number of VPNs supported by any set of PE routers.
No VPN-specific protection and restoration mechanisms are needed;
these are general routing considerations, and the VPN scheme is
compatible with any protection and restoration mechanisms that may be
available.
The SP does not manage the customer’s IGP in any way, and routes are
never leaked between the SP’s IGP and any customer’s IGP.
If the PE/CE protocol is EBGP, the SP and the customer do not ever
participate in a common IGP.
12. Migration Impact
Generally, this means replacement of an existing legacy backbone with
VPN backbone. The general migration mechanism would be to hook up
the sites one at a time to the VPN backbone, and to start giving the
routes via the VPN backbone preference to routes via the legacy
backbone. Details depend on the legacy backbone’s IGP. In general,
one would have to manipulate the IGP metrics to provide the proper
route preference.
If the legacy backbone routing protocol is OSPF, then migration is
best done with OSPF as the PE/CE protocol and the PE supporting the
[VPN-OSPF] procedures, OR with BGP as the PE/CE protocol, and the CE
supporting the BGP/OSPF interaction specified in [VPN-OSPF].
With other legacy backbone routing protocols, the proper metrics must
be set at the point (PE or CE) where the BGP routes from the SP
network are being redistributed into the legacy IGP.
13. Scalability
There is no upper limit on the number of VPNs per SP network, as
there is no one box in the SP network that needs to know of all VPNs.
Knowledge of a particular VPN is confined to the PE routers that
attach to sites in that VPN, and to the BGP Route Reflectors that
receive routing data from those PEs; other systems maintain no state
at all for the VPN. Note though that there is no need for any one
Route Reflector to know of all VPNs.
If the SP is providing the VPN service over an MPLS backbone, then
the backbone IGP must carry a host route for every Label Switched
Path (LSP) egress node within the routing domain. Every PE router in
the routing domain is an LSP egress node. If there are VPNs attached
to PE routers that are within the routing domain, as well as PE
routers that are in some second routing domain, then the border
routers leading towards the second routing domain will also be LSP
egress nodes. Thus, the sum of the number of PE routers plus number
of border routers within a routing domain is limited by the number of
routes that can be carried within the domain’s IGP. This does not
seem to create any practical scalability issue.
There is no upper limit on the number of site interfaces per VPN, as
state for a particular interface is maintained only at the PE router
to which that interface attaches. The number of site interfaces per
VPN at a given PE router is limited only by the number of interfaces
that that PE router can support.
The number of routes per VPN is constrained only by the number of
routes that can be supported in BGP, the number of routes that can be
maintained in the PEs that attach to that VPN, and the number of
routes that can be maintained in the BGP Route Reflectors that hold
the routes of that VPN.
The major constraint in considering scalability is the number of
routes that a given PE can support. In general, a given PE can
support as many VPNs as it has interfaces (including virtual
interfaces or "sub-interfaces", not just physical interfaces), but it
is constrained in the total number of routes it can handle. The
number of routes a given PE must handle depends on the particular set
of VPNs it attaches to, and the number of routes in each such VPN,
and the number of "non-VPN" Internet routes (if any) that it must
also handle.
The SP may need to engage in significant planning to ensure that
these limits are not often reached. If these limits are reached, it
may be necessary either to replace the PE with one of larger capacity
or to reorganize the way in which access links lead from CEs to PEs,
in order to better concentrate the set of access links from sites
that are in the same VPN. Rehoming a site to a different PE may not
involve actual rewiring; if the access technology is switched, this
is a matter of provisioning, but may still be a significant
undertaking. If it is necessary to have downtime while performing
the rehoming, the customer is impacted as well. Rehoming can also be
done "virtually", by creating a layer 2 tunnel from a CE’s "old" PE
to its "new" PE.
An important consideration to remember is that one may have any
number of INDEPENDENT BGP systems carrying VPN routes. This is
unlike the case of the Internet, where the Internet BGP system must
carry all the Internet routes. The difference stems from the fact
that all Internet addresses must be reachable from each other, but a
given VPN address is only supposed to be reachable from other
addresses in the same VPN.
Scalability is also affected by the rate of changes in the
reachability advertisements from CE to PE, as changes reported by a
CE to its attached PE may be propagated to the other PEs. BGP
mechanisms to control the rate of reported changes should be used by
the SP.
Another constraint on the number of VPNs that can be supported by a
particular PE router is based on the number of routing instances that
the PE router can support. If the PE/CE routing is static, or is
done by BGP, the number of routing protocol instances in a PE device
does not depend on the number of CEs supported by the PE device. In
the case of BGP, a single BGP protocol instance can support all CEs
that exchange routing information using BGP. If the PE/CE router is
done via RIP or OSPF, then the PE must maintain one RIP or OSPF
instance per VRF. Note that the number of routing instances that can
be supported may be different for different routing protocols.
Inter-AS scenarios constructed according to option (b) of section 10
of [BGP-MPLS-IP-VPN] require BGP "border routers" to hold the routes
for a set of VPNs. If two SPs share in a small number of VPNs, a
single border router between them provides adequate capacity. As the
number of shared VPNs increases, additional border routers may be
needed to handle the increased number of routes. Again, no single
border router would handle all the routes from all the VPNs, so an
increase in the number of VPNs can always be supported by adding more
border routers.
Inter-AS scenarios constructed according to option (c) of section 10
of [BGP-MPLS-IP-VPN] eliminate the need for border routers to contain
VPN routes (thus improving scalability in that dimension), but at the
cost of requiring that each AS have a route to the PEs in the others.
(Inter-AS scenarios constructed according to option (a) of section 10
of [BGP-MPLS-IP-VPN] do not scale well.)
The solution of [BGP-MPLS-IP-VPN] is intended to simplify CE and site
operations, by hiding the structure of the rest of the VPN from a
site, and by hiding the structure of the backbone. Thus, CEs need
have only a single sub-interface to the backbone, CEs at one site
need not even be aware of the existence of CEs at another, and CEs at
one site need not be routing peers of CEs at another. CEs are never
routing peers of P routers. These factors help to scale the
customer’s network, but limiting the number of adjacencies each CE
must see, and by limiting the total number of links that the
customer’s IGP must handle.
The solution of [BGP-MPLS-IP-VPN] is also intended to simplify the
SP’s VPN provisioning, so that potentially the SP will have to do
little more than say which sites belong to which VPNs. However, as
the system scales up, planning is needed to determine which PEs
should home which VPNs, and which BGP RRs should take which VPNs’
routing information.
P routers maintain NO per-VPN state at all; the only requirement on
them is to maintain routes to the PE routers. When MPLS is used, a P
router must also maintain one multipoint-to-point LSP for each such
route.
However, certain VPN multicast schemes require per-multicast-group
state in the P routers, summed over all VPNs. Others require only no
state in the P routers at all, but will result in sending more
unnecessary traffic. The complete set of tradeoffs for multicast is
not that well understood yet.
Note that as the scaling of a particular PE is primarily a matter of
the total number of routes that it must maintain, scalability is
facilitated if the addresses are assigned in a way that permits them
to be aggregated (i.e., if the customers have a sensible addressing
plan).
When a dynamic routing protocol is run on the link between a CE
router and a PE router, routing instability in the private network
may have an effect on the PE router. For example, an unusually large
number of routing updates could be sent from the CE router to the PE
router, placing an unusually large processing load on the PE router.
This issue can be mitigated via resource partitioning in the PE, in
order to limit the amount of resources (e.g., CPU and memory) that
any one VPN is permitted to use in PE routers. Also, rate limits may
be applied to the routing traffic sent from the CE to the PE.
Alternately, when this problem is detected, the CE-to-PE interface
may be shut down.
14. QoS, SLA
The provision of appropriate QoS capabilities may require any
combination of the following:
- QoS in the access network.
- Admission control (policing) by the PE router on the ingress
access links.
- Traffic conditioning (shaping) by the PE router on the ingress
access links.
- Traffic engineering in the backbone.
- Intserv/diffserv classification by the PE, for traffic arriving
from the CE. Once the PE classifies the user packets, this
classification needs to be preserved in the encapsulation (MPLS
or IP) used to send the packet across the backbone.
- Differentiated Services Codepoint (DSCP) mapping.
- DSCP transparency.
- Random Early Discard in the backbone.
None of these features are VPN-specific. The ability to support them
depends on whether the features are available on the edge and core
platforms, rather than on any particular VPN scheme.
MPLS support for differentiated services is detailed in RFC 3270
[MPLS-DIFFSERV]. DSCP mapping and transparency are covered in
section 2.6 of that document.
It is possible to use traffic engineering to provide, e.g.,
guaranteed bandwidth between two PEs for the traffic of a given VPN.
The VRF entries for that VPN in each PE need to be modified so that
the traffic to the other PE is directed onto the traffic-engineered
path. How this is done is a local matter.
BGP/MPLS IP VPNs can support both the "hose model" and the "pipe
model" of QoS. In the "pipe model", a particular quality of service
(e.g., a guaranteed amount of bandwidth) would be applied to all or
some of the packets traveling between a given pair of CEs. In the
"hose model", a particular quality of service (e.g., a guaranteed
amount of bandwidth) would be applied to all traffic to or from a
particular CE, irrespective of which other CE the traffic is going to
or coming from. Since BGP/MPLS IP VPNs do not usually make use of
CE-CE tunnels, the hose model is the more natural fit. Providing the
pipe model would require the use of traffic engineering to explicitly
create the necessary tunnels.
Many of the requirements specified in [L3VPN-REQS] stipulate that the
Network Monitoring System (NMS) should support SLA monitoring and
verification between the SP and the various customers by measurement
of the indicators defined within the context of the SLA. The
measurement of these indicators (i.e., counters) can be achieved when
BGP/MPLS IP VPNs are used by employing a combination of the
Management Information Base (MIB) module designed for BGP/MPLS IP
VPNs [L3VPN-MIB] as well as other standard MIB modules such as the
IF-MIB [IF-MIB]. Devices supporting these MIB modules can calculate
SLAs based on real-time performance measurements using indicators and
threshold crossing alerts. Devices can make these thresholds
configurable either via a management interface such as SNMP.
15. Management
The L3VPN Requirements document [L3VPN-REQS] stipulates that the term
"Provider Provisioned VPN" refers to VPNs for which the service
provider participates in management and provisioning of the VPN. RFC
BGP/MPLS IP VPNs can be provisioned and managed to meet these
requirements. The following subsections will outline how devices
supporting BGP/MPLS IP VPNs can satisfy these requirements.
15.1. Management by the Provider
The SP manages all the VPN-specific information in the PE device.
This can be done using the MIB designed for BGP/MPLS IP VPNs
[L3VPN-MIB], in combination with other standard MIB modules such as
IF-MIB [IF-MIB], and other MPLS MIB modules [LSRMIB], [LDPMIB],
[TEMIB], [FTNMIB].
Devices supporting BGP/MPLS IP VPNs that employ the management
interface characteristics described above will also support the ITU-T
Telecommunications Management Network Model "FCAPS" functionalities
as required in the L3VPN Requirements document. These include Fault,
Configuration, Accounting, Provisioning, and Security.
In BGP/MPLS IP VPNs, the SP is not required to manage the CE devices.
However, if it is desired for the SP to do so, the SP may manage CE
devices from a central site, provided that a route to the central
site is exported into the CE’s VPN, and the central site is in a VPN
into which the routes to the managed CE devices have been imported.
This is a form of extranet.
If the central site is managing CE devices from several VPNs, those
CE devices must have mutually unique addresses. Note that this does
not enable the CE devices from different VPNs to reach each other.
The CE devices have no VPN-specific information in them. Hence the
fact that they are connected together into a VPN does not require
them to have any VPN-specific management MIB modules or capabilities.
15.2. Management by the Customer
CE devices may be managed from within the VPN, transparently to the
SP. The CE devices have no VPN-specific information in them, and the
fact that they are tied together into a VPN does not impact the
customer’s management of them.
Customer access to a PE device is totally at the discretion of the
SP, but is not required by the solution. The PE device is a routing
peer of a CE device, and can be pinged, etc.
If a customer is permitted to access the PE router for management
purposes, the functions available to any particular customer need to
be strictly controlled, and the use of resource partitioning may be
appropriate.
Network management traffic from the CE to the PE may be rate limited
(for example, to prevent network management traffic from CE to PE to
be used in a DoS attack).
16. Acknowledgements
Many thanks to Jeremy De Clercq, Luyuan Fang, Dave McDysan, Ananth
Nagarajan, Yakov Rekhter, and Muneyoshi Suzuki, for their comments,
criticisms, and help in preparing this document. Thanks also to
Thomas Nadeau for his help with the section on management, to
Francois LeFaucheur for his help with the section on QoS, and to Ross
Callon for his review of the document.
17. Normative References
[BGP-EXT-COMM] Sangli, S., Tappan, D., and Y. Rekhter, "BGP
Extended Communities Attribute", RFC 4360,
February 2006.
[BGP-MPLS-IP-VPN] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual
Private Networks (VPNs)", RFC 4364, February
2006.
[L3VPN-FRMWRK] Callon, R. and M. Suzuki, "A Framework for Layer
3 Provider-Provisioned Virtual Private Networks
(PPVPNs)", RFC 4110, July 2005.
[L3VPN-REQS] Carugi, M. and D. McDysan, "Service Requirements
for Layer 3 Provider Provisioned Virtual Private
Networks (PPVPNs)", RFC 4031, April 2005.
[L2VPN-SEC-FRMWRK] Fang, L., "Security Framework for Provider-
Provisioned Virtual Private Networks (PPVPNs)",
RFC 4111, July 2005.
18. Informative References
[VPN-OSPF] Rosen, E., Psenak, P., and P. Pillay-Esnault,
"OSPF as the PE/CE Protocol in BGP/MPLS VPNs",
Work in Progress, February 2004.
[OSPF-2547-DNBIT] Rosen, E., Psenak, P., and P. Pillay-Esnault,
"Using an LSA Options Bit to Prevent Looping in
BGP/MPLS IP VPNs", Work in Progress, March 2004.
[MPLS/BGP-IPsec] Rosen, E., De Clercq, J., Paridaens, O.,
T’Joens, Y., and C. Sargor, "Architecture for
the Use of PE-PE IPsec Tunnels in BGP/MPLS IP
VPNs", Work in Progress, March 2004.
[BGP-MPLS-MCAST-VPN] Rosen, E., Cai, Y., and IJ. Wijsnands,
"Multicast in MPLS/BGP VPNs", Work in Progress,
May 2004.
[CE-VERIF] Bonica, R., Rekhter, Y., Raszuk, R., Rosen, E.,
and D. Tappan, "CE-to-CE Member Verification for
Layer 3 VPNs", Work in Progress, September 2003.
[FTNMIB] Nadeau, T., Srinivasan, C., and A. Viswanathan,
"Multiprotocol Label Switching (MPLS) Forwarding
Equivalence Class To Next Hop Label Forwarding
Entry (FEC-To-NHLFE) Management Information Base
(MIB)", RFC 3814, June 2004.
[IPSEC-VPN] De Clercq, J., Paridaens, O., Krywaniuk, A., and
C. Wang, "An Architecture for Provider
Provisioned CE-based Virtual Private Networks
using IPsec", Work in Progress, February 2004.
[LDPMIB] Cucchiara, J., Sjostrand, H., and J. Luciani,
"Definitions of Managed Objects for the
Multiprotocol Label Switching (MPLS), Label
Distribution Protocol (LDP)", RFC 3815, June
2004.
[LSRMIB] Srinivasan, C., Viswanathan, A., and T. Nadeau,
"Multiprotocol Label Switching (MPLS) Label
Switching Router (LSR) Management Information
Base (MIB)", RFC 3813, June 2004.
[MPLS-DIFFSERV] Le Faucheur, F., Wu, L., Davie, B., Davari, S.,
Vaananen, P., Krishnan, R., Cheval, P., and J.
Heinanen, "Multi-Protocol Label Switching (MPLS)
Support of Differentiated Services", RFC 3270,
May 2002.
[L3VPN-MIB] Nadeau, T. and H. Van Der Linde, "MPLS/BGP
Virtual Private Network Management Information
Base Using SMIv2", Work in Progress, August