The SPs will therefore need to determine and agree upon the expected
amount of membership information to be exchanged, and the dynamic
nature of this information. Mechanisms may also be needed to
authenticate the VPN membership information.
VPN information should be distributed only to places where it needs
to go, whether that is intra-provider or inter-provider. In this
way, the distribution of VPN information is unlike the distribution
of inter-provider routing information, as the latter needs to be
distributed throughout the Internet. In addition, the joint support
of a VPN by two SPs should not require any third SP to maintain state
for that VPN. Again, notice the difference with respect to
inter-provider routing; in inter-provider routing: sending traffic
from one SP to another may indeed require routing state in a third
SP.
As one possible example: Suppose that there are two SPs A and C,
which want to support a common VPN. Suppose that A and C are
interconnected via SP B. In this case B will need to know how to
route traffic between A and C, and therefore will need to know
something about A and C (such as enough routing information to
forward IP traffic and/or connect MPLS LSPs between PEs or route
reflectors in A and C). However, for scaling purposes it is
desirable that B not need to know VPN-specific information about the
VPNs which are supported by A and C.
4.2.2. Constraining Distribution of VPN Routing Information
In layer 3 provider-provisioned CE-based VPNs, the VPN tunnels
connect CE devices. In this case, distribution of IP routing
information occurs between CE devices on the customer sites. No
additional constraints on the distribution of VPN routing information
are necessary.
In layer 3 PE-based VPNs, however, the PE devices must be aware of
VPN routing information (for the VPNs to which they are attached).
For scalability reasons, one does not want a scheme in which all PEs
contain all routes for all VPNs. Rather, only the PEs that are
attached to sites in a given VPN should contain the routing
information for that VPN. This means that the distribution of VPN
routing information between PE devices must be constrained.
As VPN membership may change dynamically, it is necessary to have a
mechanism that allows VPN route information to be distributed to any
PE where there is an attached user for that VPN, and allows for the
removal of this information when it is no longer needed.
In the Virtual Router scheme, per-VPN tunnels must be established
before any routes for a VPN are distributed, and the routes are then
distributed through those tunnels. Thus by establishing the proper
set of tunnels, one implicitly constrains and controls the
distribution of per-VPN routing information. In this scheme, the
distribution of membership information consists of the set of VPNs
that exists on each PE, as well as information about the desired
topology. This enables a PE to determine the set of remote PEs to
which it must establish tunnels for a particular VPN.
In the aggregated routing scheme (see section 4.4.4), the
distribution of VPN routing information is constrained by means of
route filtering. As VPN membership changes on a PE, the route
filters in use between the PE and its peers can be adjusted. Each
peer may then adjust the filters in use with each of its peers in
turn, and thus the changes propagate across the network. When BGP is
used, this filtering may take place at route reflectors as discussed
in section 4.4.4.
4.2.3. Controlling VPN Topology
The topology for a VPN consists of a set of nodes interconnected via
tunnels. The topology may be a full mesh, a hub and spoke topology,
or an arbitrary topology. For a VPN the set of nodes will include
all VPN edge devices that have attached sites for that VPN.
Naturally, whatever the topology, all VPN sites are reachable from
each other; the topology simply constrains the way traffic is routed
among the sites. For example, in one topology traffic between site A
and site B goes from one to the other directly over the VPN backbone;
in another topology, traffic from site A to site B must traverse site
C before reaching site B.
The simplest topology is a full mesh, where a tunnel exists between
every pair of VPN edge devices. If we assume the use of point-to-
point tunnels (rather than multipoint-to-point), then with a full
mesh topology there are N*(N-1)/2 duplex tunnels or N*(N-1) simplex
tunnels for N VPN edge devices. Each tunnel consumes some resources
at a VPN edge device, and depending on the type of tunnel, may or may
not consume resources in intermediate routers or LSRs. One reason
for using a partial mesh topology is to reduce the number of tunnels
a VPN edge device, and/or the network, needs to support. Another
reason is to support the scenario where an administrator requires all
traffic from certain sites to traverse some particular site for
policy or control reasons, such as to force traffic through a
firewall, or for monitoring or accounting purposes. Note that the
topologies used for each VPN are separate, and thus the same VPN edge
device may be part of a full mesh topology for one VPN, and of a
partial mesh topology for another VPN.
An example of where a partial mesh topology could be suitable is for
a VPN that supports a large number of telecommuters and a small
number of corporate sites. Most traffic will be between
telecommuters and the corporate sites, not between pairs of
telecommuters. A hub and spoke topology for the VPN would thus map
onto the underlying traffic flow, with the telecommuters attached to
spoke VPN edge devices and the corporate sites attached to hub VPN
edge devices. Traffic between telecommuters is still supported, but
this traffic traverses a hub VPN edge device.
The selection of a topology for a VPN is an administrative choice,
but it is useful to examine protocol mechanisms that can be used to
automate the construction of the desired topology, and thus reduce
the amount of configuration needed. To this end it is useful for a
VPN edge device to be able to advertise per-VPN topology information
to other VPN edge devices. It may be simplest to advertise this at
the same time as the membership information is advertised, using the
same mechanisms.
A simple scheme is where a VPN edge device advertises itself either
as a hub or as a spoke, for each VPN that it has. When received by
other VPN edge devices this information can be used when determining
whether to establish a tunnel. A more comprehensive scheme allows a
VPN edge device to advertise a set of topology groups, with tunnels
established between a pair of VPN edge devices if they have a group
in common.
4.3. VPN Tunneling
VPN solutions use tunneling in order to transport VPN packets across
the VPN backbone, from one VPN edge device to another. There are
different types of tunneling protocols, different ways of
establishing and maintaining tunnels, and different ways to associate
tunnels with VPNs (e.g., shared versus dedicated per-VPN tunnels).
Sections 4.3.1 through 4.3.5 discusses some common characteristics
shared by all forms of tunneling, and some common problems to which
tunnels provide a solution. Section 4.3.6 provides a survey of
available tunneling techniques. Note that tunneling protocol issues
are generally independent of the mechanisms used for VPN membership
and VPN routing.
One motivation for the use of tunneling is that the packet addressing
used in a VPN may have no relation to the packet addressing used
between the VPN edge devices. For example the customer VPN traffic
could use non-unique or private IP addressing [RFC1918]. Also an
IPv6 VPN could be implemented across an IPv4 provider backbone. As
such the packet forwarding between the VPN edge devices must use
information other than that contained in the VPN packets themselves.
A tunneling protocol adds additional information, such an extra
header or label, to a VPN packet, and this additional information is
then used for forwarding the packet between the VPN edge devices.
Another capability optionally provided by tunneling is that of
isolation between different VPN traffic flows. The QoS and security
requirements for these traffic flows may differ, and can be met by
using different tunnels with the appropriate characteristics. This
allows a provider to offer different service characteristics for
traffic in different VPNs, or to subsets of traffic flows within a
single VPN.
The specific tunneling protocols considered in this section are GRE,
IP-in-IP, IPsec, and MPLS, as these are the most suitable for
carrying VPN traffic across the VPN backbone. Other tunneling
protocols, such as L2TP [RFC2661], may be used as access tunnels,
carrying traffic between a PE and a CE. As backbone tunneling is
independent of and orthogonal to access tunneling, protocols for the
latter are not discussed here.
4.3.1. Tunnel Encapsulations
All tunneling protocols use an encapsulation that adds additional
information to the encapsulated packet; this information is used for
forwarding across the VPN backbone. Examples are provided in section
4.3.6.
One characteristic of a tunneling protocol is whether per-tunnel
state is needed in the SP network in order to forward the
encapsulated packets. For IP tunneling schemes (GRE, IP-in-IP, and
IPsec) per-tunnel state is completely confined to the VPN edge
devices. Other routers are unaware of the tunnels, and forward
according to the IP header. For MPLS, per-tunnel state is needed,
since the top label in the label stack must be examined and swapped
by intermediate LSRs. The amount of state required can be minimized
by hierarchical multiplexing, and by use of multi-point to point
tunnels, as discussed below.
Another characteristic is the tunneling overhead introduced. With
IPsec the overhead may be considerable as it may include, for
example, an ESP header, ESP trailer and an additional IP header. The
other mechanisms listed use less overhead, with MPLS being the most
lightweight. The overhead inherent in any tunneling mechanism may
result in additional IP packet fragmentation, if the resulting packet
is too large to be carried by the underlying link layer. As such it
is important to report any reduced MTU sizes via mechanisms such as
path MTU discovery in order to avoid fragmentation wherever possible.
Yet another characteristic is something we might call "transparency
to the Internet". IP-based encapsulation can carry be used to carry
a packet anywhere in the Internet. MPLS encapsulation can only be
used to carry a packet on IP networks that support MPLS. If an
MPLS-encapsulated packet must cross the networks of multiple SPs, the
adjacent SPs must bilateral agreements to accept MPLS packets from
each other. If only a portion of the path across the backbone lacks
MPLS support, then an MPLS-in-IP encapsulation can be used to move
the MPLS packets across that part of the backbone. However, this
does add complexity. On the other hand, MPLS has efficiency
advantages, particularly in environments where encapsulations may
need to be nested.
Transparency to the Internet is sometimes a requirement, but
sometimes not. This depends on the sort of service which a SP is
offering to its customer.
4.3.2. Tunnel Multiplexing
When a tunneled packet arrives at the tunnel egress, it must be
possible to infer the packet’s VPN from its encapsulation header. In
MPLS encapsulations, this must be inferred from the packet’s label
stack. In IP-based encapsulations, this can be inferred from some
combination of the IP source address, the IP destination address, and
a "multiplexing field" in the encapsulation header. The multiplexing
field might be one which was explicitly designed for multiplexing, or
one that wasn’t originally designed for this but can be pushed into
service as a multiplexing field. For example:
o GRE: Packets associated to VPN by source IP address, destination IP
address, and Key field, although the key field was originally
intended for authentication.
o IP-in-IP: Packets associated to VPN by IP destination address in
outer header.
o IPsec: Packets associated to VPN by IP source address, IP
destination address, and SPI field.
o MPLS: Packets associated to VPN by label stack.
Note that IP-in-IP tunneling does not have a real multiplexing field,
so a different IP destination address must be used for every VPN
supported by a given PE. In the other IP-based encapsulations, a
given PE need have only a single IP address, and the multiplexing
field is used to distinguish the different VPNs supported by a PE.
Thus the IP-in-IP solution has the significant disadvantage that it
requires the allocation and assignment of a potentially large number
of IP addresses, all of which have to be reachable via backbone
routing.
In the following, we will use the term "multiplexing field" to refer
to whichever field in the encapsulation header must is used to
distinguish different VPNs at a given PE. In the IP-in-IP
encapsulation, this is the destination IP address field, in the other
encapsulations it is a true multiplexing field.
4.3.3. Tunnel Establishment
When tunnels are established, the tunnel endpoints must agree on the
multiplexing field values which are to be used to indicate that
particular packets are in particular VPNs. The use of "well known"
or explicitly provisioned values would not scale well as the number
of VPNs increases. So it is necessary to have some sort of protocol
interaction in which the tunnel endpoints agree on the multiplexing
field values.
For some tunneling protocols, setting up a tunnel requires an
explicit exchange of signaling messages. Generally the multiplexing
field values would be agreed upon as part of this exchange. For
example, if an IPsec encapsulation is used, the SPI field plays the
role of the multiplexing field, and IKE signaling is used to
distribute the SPI values; if an MPLS encapsulation is used, LDP,
CR-LDP or RSVP-TE can be used to distribute the MPLS label value used
as the multiplexing field. Information about the identity of the VPN
with which the tunnel is to be associated needs to be exchanged as
part of the signaling protocol (e.g., a VPN-ID can be carried in the
signaling protocol). An advantage of this approach is that
per-tunnel security, QoS and other characteristics may also be
negotiable via the signaling protocol. A disadvantage is that the
signaling imposes overhead, which may then lead to scalability
considerations, discussed further below.
For some tunneling protocols, there is no explicit protocol
interaction that sets up the tunnel, and the multiplexing field
values must be exchanged in some other way. For example, for MPLS
tunnels, MPLS labels can be piggybacked on the protocols used to
distribute VPN routes or VPN membership information. GRE and
IP-in-IP have no associated signaling protocol, and thus by necessity
the multiplexing values are distributed via some other mechanism,
such as via configuration, control protocol, or piggybacked in some
manner on a VPN membership protocol.
The resources used by the different tunneling establishment
mechanisms may vary. With a full mesh VPN topology, and explicit
signaling, each VPN edge device has to establish a tunnel to all the
other VPN edge devices for in each VPN. The resources needed for
this on a VPN edge device may be significant, and issues such as the
time needed to recover following a device failure may need to be
taken into account, as the time to recovery includes the time needed
to reestablish a large number of tunnels.
4.3.4. Scaling and Hierarchical Tunnels
If tunnels require state to be maintained in the core of the network,
it may not be feasible to set up per-VPN tunnels between all adjacent
devices that are adjacent in some VPN topology. This would violate
the principle that there is no per-VPN state in the core of the
network, and would make the core scale poorly as the number of VPNs
increases. For example, MPLS tunnels require that core network
devices maintain state for the topmost label in the label stack. If
every core router had to maintain one or more labels for every VPN,
scaling would be very poor.
There are also scaling considerations related to the use of explicit
signaling for tunnel establishment. Even if the tunneling protocol
does not maintain per tunnel state in the core, the number of tunnels
that a single VPN edge device needs to handle may be large, as this
grows according to the number of VPNs and the number of neighbors per
VPN. One way to reduce the number of tunnels in a network is to use
a VPN topology other than a full mesh. However this may not always
be desirable, and even with hub and spoke topologies the hubs VPN
edge devices may still need to handle large numbers of tunnels.
If the core routers need to maintain any per-tunnel state at all,
scaling can be greatly improved by using hierarchical tunnels. One
tunnel can be established between each pair of VPN edge devices, and
multiple VPN-specific tunnels can then be carried through the single
"outer" tunnel. Now the amount of state is dependent only on the
number of VPN edge devices, not on the number of VPNs. Scaling can
be further improved by having the outer tunnels be
multipoint-to-point "merging" tunnels. Now the amount of state to be
maintained in the core is on the order of the number of VPN edge
devices, not on the order of the square of that number. That is, the
amount of tunnel state is roughly equivalent to the amount of state
needed to maintain IP routes to the VPN edge devices. This is almost
(if not quite) as good as using tunnels which do not require any
state to be maintained in the core.
Using hierarchical tunnels may also reduce the amount of state to be
maintained in the VPN edge devices, particularly if maintaining the
outer tunnels requires more state than maintaining the per-VPN
tunnels that run inside the outer tunnels.
There are other factors relevant to determining the number of VPN
edge to VPN edge "outer" tunnels to use. While using a single such
tunnel has the best scaling properties, using more than one may allow
different QoS capabilities or different security characteristics to
be used for different traffic flows (from the same or from different
VPNs).
When tunnels are used hierarchically, the tunnels in the hierarchy
may all be of the same type (e.g., an MPLS label stack) or they may
be of different types (e.g., a GRE tunnel carried inside an IPsec
tunnel).
One example using hierarchical tunnels is the establishment of a
number of different IPsec security associations, providing different
levels of security between a given pair of VPN edge devices. Per-VPN
GRE tunnels can then be grouped together and then carried over the
appropriate IPsec tunnel, rather than having a separate IPsec tunnel
per-VPN. Another example is the use of an MPLS label stack. A
single PE-PE LSP is used to carry all the per-VPN LSPs. The
mechanisms used for label establishment are typically different. The
PE-PE LSP could be established using LDP, as part or normal backbone
operation, with the per-VPN LSP labels established by piggybacking on
VPN routing (e.g., using BGP) discussed in sections 3.3.1.3 and 4.1.
4.3.5. Tunnel Maintenance
Once a tunnel is established it is necessary to know that the tunnel
is operational. Mechanisms are needed to detect tunnel failures, and
to respond appropriately to restore service.
There is a potential issue regarding propagation of failures when
multiple tunnels are multiplexed hierarchically. Suppose that
multiple VPN-specific tunnels are multiplexed inside a single PE to
PE tunnel. In this case, suppose that routing for the VPN is done
over the VPN-specific tunnels (as may be the case for CE-based and VR
approaches). Suppose that the PE to PE tunnel fails. In this case
multiple VPN-specific tunnels may fail, and layer 3 routing may
simultaneously respond for each VPN using the failed tunnel. If the
PE to PE tunnel is subsequently restored, there may then be multiple
VPN-specific tunnels and multiple routing protocol instances which
also need to recover. Each of these could potentially require some
exchange of control traffic.
When a tunnel fails, if the tunnel can be restored quickly, it might
therefore be preferable to restore the tunnel without any response by
high levels (such as other tunnels which were multiplexed inside the
failed tunnels). By having high levels delay response to a lower
level failed tunnel, this may limit the amount of control traffic
needed to completely restore correct service. However, if the failed
tunnel cannot be quickly restored, then it is necessary for the
tunnels or routing instances multiplexed over the failed tunnel to
respond, and preferable for them to respond quickly and without
explicit action by network operators.
With most layer 3 provider-provisioned CE-based VPNs and the VR
scheme, a per-VPN instance of routing is running over the tunnel,
thus any loss of connectivity between the tunnel endpoints will be
detected by the VPN routing instance. This allows rapid detection of
tunnel failure. Careful adjustment of timers might be needed to
avoid failure propagation as discussed the above. With the
aggregated routing scheme, there isn’t a per-VPN instance of routing
running over the tunnel, and therefore some other scheme to detect
loss of connectivity is needed in the event that the tunnel cannot be
rapidly restored.
Failure of connectivity in a tunnel can be very difficult to detect
reliably. Among the mechanisms that can be used to detect failure
are loss of the underlying connectivity to the remote endpoint (as
indicated, e.g., by "no IP route to host" or no MPLS label), timeout
of higher layer "hello" mechanisms (e.g., IGP hellos, when the tunnel
is an adjacency in some IGP), and timeout of keep alive mechanisms in
the tunnel establishment protocols (if any). However, none of these
techniques provides completely reliable detection of all failure
modes. Additional monitoring techniques may also be necessary.
With hierarchical tunnels it may suffice to only monitor the
outermost tunnel for loss of connectivity. However there may be
failure modes in a device where the outermost tunnel is up but one of
the inner tunnels is down.
4.3.6. Survey of Tunneling Techniques
Tunneling mechanisms provide isolated communication between two CE-PE
devices. Available tunneling mechanisms include (but are not limited
to): GRE [RFC2784] [RFC2890], IP-in-IP encapsulation [RFC2003]
[RFC2473], IPsec [RFC2401] [RFC2402], and MPLS [RFC3031] [RFC3035].
Note that the following subsections address tunnel overhead to
clarify the risk of fragmentation. Some SP networks contain layer 2
switches that enforce the standard/default MTU of 1500 bytes. In
this case, any encapsulation whatsoever creates a significant risk of
fragmentation. However, layer 2 switch vendors are in general aware
of IP tunneling as well as stacked VLAN overhead, thus many switches
practically allow an MTU of approximately 1512 bytes now. In this
case, up to 12 bytes of encapsulation can be used before there is any
risk of fragmentation. Furthermore, to improve TCP and NFS
performance, switches that support 9K bytes "jumbo frames" are also
on the market. In this case, there is no risk of fragmentation.
4.3.6.1. GRE [RFC2784] [RFC2890]
Generic Routing Encapsulation (GRE) specifies a protocol for
encapsulating an arbitrary payload protocol over an arbitrary
delivery protocol [RFC2784]. In particular, it can be used where
both the payload and the delivery protocol are IP as is the case in
layer 3 VPNs. A GRE tunnel is a tunnel whose packets are
encapsulated by GRE.
o Multiplexing
The GRE specification [RFC2784] does not explicitly support
multiplexing. But the key field extension to GRE is specified in
[RFC2890] and it may be used as a multiplexing field.
o QoS/SLA
GRE itself does not have intrinsic QoS/SLA capabilities, but it