MPLS L3 VPN Walkthrough

Let me go over a basic MPLS L3 VPN using the topology below.

 

Here AS 2345 makes up the network that will provide the MPLS L3 VPN service to AS 6 and AS 78.

Building a MPLS L3 VPN service requires a couple of technologies for label exchange:

  • MPLS – LDP
  • BGP – VPNv4

For both LDP and BGP we need reachability of the loopback interfaces of the routers in AS 2345. I will go over the reason for this in a bit.

LDP

LDP (Label Distribution Protocol) is the protocol in charge of providing a transit label. It does so by having each router allocate a label for any IGP prefix it knows about and advertise those IP and label bindings to its neighbors. Let’s have a look at an example of this between R1 and R2.

! R1 config:
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
 ip ospf 1 area 0
!
interface GigabitEthernet1.12
 encapsulation dot1Q 12
 ip address 10.0.12.1 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
R1#

The configuration of R2 is very similar to that of R1 so I will not provide it to save space and avoid clutter.

Time for verification. We will look at it from R2’s perspective:

! R2 LDP verification:
R2#sh mpls ldp neighbor
   Peer LDP Ident: 1.1.1.1:0; Local LDP Ident 2.2.2.2:0
    TCP connection: 1.1.1.1.646 - 2.2.2.2.38200
    State: Oper; Msgs sent/rcvd: 426/427; Downstream
    Up time: 06:02:37
    LDP discovery sources:
     GigabitEthernet1.12, Src IP addr: 10.0.12.1
    Addresses bound to peer LDP Ident:
     10.0.12.1 10.0.13.1 10.0.14.1 10.0.15.1
     1.1.1.1
R2#

We have a lot of interesting information here. First of all what you always must look for in this output is the Peer LDP Ident. This is the LDP router-id of the neighbor. By default this router-id is also used to establish the TCP session between the routers. This is one of the reasons why I wrote earlier that the loopback must be reachable.

Next very important thing to verify is the State. A state of Oper (Operational) means that everything is good.

Lastly the Addresses bound to peer LDP Ident section is very important. I will touch on this a bit later.

So far so good. Now let’s see which labels we have generated and received on R2.

! R2 Label bindings:
R2#sh mpls ldp bindings
  lib entry: 1.1.1.1/32, rev 12
    local binding: label: 19
    remote binding: lsr: 1.1.1.1:0, label: imp-null
  lib entry: 2.2.2.0/24, rev 20
    local binding: label: imp-null
  lib entry: 2.2.2.2/32, rev 21(no route)
    no local binding
    remote binding: lsr: 1.1.1.1:0, label: 19
  lib entry: 3.3.3.3/32, rev 10
    local binding: label: 18
    remote binding: lsr: 1.1.1.1:0, label: 18
  lib entry: 4.4.4.4/32, rev 8
    local binding: label: 17
    remote binding: lsr: 1.1.1.1:0, label: 17
  lib entry: 5.5.5.5/32, rev 6
    local binding: label: 16
    remote binding: lsr: 1.1.1.1:0, label: 16
  lib entry: 10.0.12.0/24, rev 2
    local binding: label: imp-null
    remote binding: lsr: 1.1.1.1:0, label: imp-null
  lib entry: 10.0.13.0/24, rev 18
    local binding: label: 22
    remote binding: lsr: 1.1.1.1:0, label: imp-null
  lib entry: 10.0.14.0/24, rev 16
    local binding: label: 21
    remote binding: lsr: 1.1.1.1:0, label: imp-null
  lib entry: 10.0.15.0/24, rev 14
    local binding: label: 20
    remote binding: lsr: 1.1.1.1:0, label: imp-null
R2#

A lot of labels have been received for this small 5 router network! This is because every router generates labels for _ALL_ IGP prefixes. Even the once it just received labels for from its direct neighbor! This is why we for 1.1.1.1/32 see a local binding with label 19. This is somewhat analogous to how Distance Vector routing protocols. Unlike Distance Vector protocols LDP does not have split horizon, mening that R2 will advertise the binding for 1.1.1.1/32 right back at R1! You’ll see this below when looking a the binding on R1 for 1.1.1.1/32.

R1#sh mpls ldp bindings | be 1.1.1.1
  lib entry: 1.1.1.1/32, rev 2
    local binding: label: imp-null
    remote binding: lsr: 4.4.4.4:0, label: 17
    remote binding: lsr: 3.3.3.3:0, label: 18
    remote binding: lsr: 5.5.5.5:0, label: 16
    remote binding: lsr: 2.2.2.2:0, label: 19
! output omitted for brevity

So on R1 we receive label binding from all of our neighbors! Quite a bit of overhead with LDP. Why isn’t this an issue with loops then? Because of the way the labels are used in conjunction with the routing table. And it has to do with how a router decides which of all the binding labels to use. The router simply looks at the next-hop IP of a prefix, then it looks at the Addresses bound to peer LDP Ident from its LDP neighbors and see if there is a match. If one of them matches, the router knows the LDP Ident of the neighbor and can find the remote binding among all the label bindings it has learned. If for example we were to look at how R2 finds the correct label for reaching R3’s loopback using this method:

! R2 find NH to R3's loopback:
R2#sh ip route 3.3.3.3
Routing entry for 3.3.3.3/32
  Known via "ospf 1", distance 110, metric 3, type intra area
  Last update from 10.0.12.1 on GigabitEthernet1.12, 06:33:53 ago
  Routing Descriptor Blocks:
  * 10.0.12.1, from 3.3.3.3, 06:33:53 ago, via GigabitEthernet1.12
    Route metric is 3, traffic share count is 1
R2#

Now we know that 10.0.12.1 is the NH to reach 3.3.3.3/32. Next let’s see if we can find this NH address among any of our LDP neighbors interfaces (Addresses bound to peer LDP Ident):

R2#sh mpls ldp neighbor | in Ident:|10.0.12.1_ 
  Peer LDP Ident: 1.1.1.1:0; Local LDP Ident 2.2.2.2:0
     GigabitEthernet1.12, Src IP addr: 10.0.12.1
   Addresses bound to peer LDP Ident:
     10.0.12.1 10.0.13.1 10.0.14.1 10.0.15.1 
R2#

Bingo! We found the LDP Ident (1.1.1.1:0) of the neighbor who has 10.0.12.1 configured. Now we can look in the label binding tabel and find the label we must use when forwarding packets to 3.3.3.3/32 out Gi1.12:

R2#sh mpls ldp bindings 3.3.3.3 32
  lib entry: 3.3.3.3/32, rev 10
    local binding: label: 18
    remote binding: lsr: 1.1.1.1:0, label: 18
R2#

And the remote binding for 3.3.3.3/32 from 1.1.1.1:0 is telling us to use label 18.

You can see all of this by looking in CEF:

R2#sh ip cef 3.3.3.3/32
3.3.3.3/32
 nexthop 10.0.12.1 GigabitEthernet1.12 label 18
R2#

LDP Lingo

So far I have been using the term label binding table and stuff like that. Here is an overview of the various tables and their names:

  • FIB – The regular CEF table as we all know it.
    • sh ip cef
  • LIB – This is what I called the label binding table. It is somewhat like the topology table of EIGRP or the LSDB of OSPF. Think of it as a label RIB.
    • sh mpls ldp binding
  • LFIB – This is the actual MPLS forwarding table containing the labels used for forwarding.
    • sh mpls forwarding-table

BGP

Now that we know how LDP works and that it provides the transport labels needed for forwarding packets of our customers between the PE routers. We need a way of segmenting our customers which is the hole idea of MPLS L3 VPNs. This is where BGP comes into the picture. It too can allocate labels. These are the VPN labels used in the bottom of the label stack.

The VPNv4 BGP AFI is used for exchanging the prefixes between PE routers – either directly in smaller networks or through RRs. And for doing the segmentation we need something do distinguish the prefixes received from our customers. This is where route-targets (RTs) comes in. The RTs are attached to the prefixes using BGP extended communities. So we must enable sending of extended communities when we peer using VPNv4 AFI. RTs work by attaching the RT when exporting (from the Customer VRF to BGP) the prefix and on the remote PE we use the RT when importing (from BGP to the Customer VRF) the prefix. Now we also need a way of making all prefixes unique which allows for overlapping prefixes between customers. This is where the route distinguisher (RD) come is. This RD which is (of course) unique per VRF is prepended to the prefixes received from the customer and makes the prefix a VPNv4 prefix. Let’s have a look at all of this. First I will show the configuration of a VRF and BGP, then we’ll look at the verification putting the pieces together.

! R2 VRF config:
vrf definition a
 rd 2.2.2.2:6
 route-target export 2345:6
 route-target import 2345:6
 !
 address-family ipv4
 exit-address-family

Defining a VRF gives the ability to use it for both IPv4 and IPv6. If you create the VRF using the legacy ip vrf command, you’ll only have IPv4 capability. We define the RD and the RTs for importing and exporting prefixes.

Next we have the BGP config part.

R2#sh run | sec router bgp
router bgp 2345
bgp log-neighbor-changes
neighbor 5.5.5.5 remote-as 2345
neighbor 5.5.5.5 update-source Loopback0
!
address-family vpnv4
neighbor 5.5.5.5 activate
neighbor 5.5.5.5 send-community extended
exit-address-family
!
address-family ipv4 vrf a
neighbor 10.0.26.6 remote-as 6
neighbor 10.0.26.6 activate
exit-address-family
R2#

In BGP we first define the neighbor and how we establish a peering with them using the remote-as and update-source commands. With MPLS L3 VPNs the update-source is extremely important, because it is used as the NH on the remote PEs and for forwarding to work on Cisco routers it must be a /32 prefix used for the NH with MPLS L3 VPNs.

The neighbor is then activated under the VPNv4 AFI which in newer code also configured the sending of the extended community.

Lastly we have the peering to our customer in vrf a so we can import and export prefixes tying together the customers sites.

Let’s have a look at what we have received form the customer in VRF a:

R2#sh bgp vpnv4 u vrf a
BGP table version is 4, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 2.2.2.2:6 (default for vrf a)
 *> 6.6.6.6/32 10.0.26.6 0 0 6 ?
R2#

We’ll drill down into 6.6.6.6/32:

R2#sh bgp vpnv4 u vrf a 6.6.6.6/32
BGP routing table entry for 2.2.2.2:6:6.6.6.6/32, version 2
Paths: (1 available, best #1, table a)
 Advertised to update-groups:
   1
 Refresh Epoch 1
 6
  10.0.26.6 (via vrf a) from 10.0.26.6 (6.6.6.6)
    Origin incomplete, metric 0, localpref 100, valid, external, best
    Extended Community: RT:2345:6
    mpls labels in/out 23/nolabel
    rx pathid: 0, tx pathid: 0x0
R2#

So upon receiving the prefix from our customer in VRF a, we’ve attached the RD giving the VPNv4 prefix of 2.2.2.2:6:6.6.6.6/32 and the RT 2345:6. Also we’ve allocated the label 23 for this specific prefix. By default Cisco routers allocate a VPN label per prefix using this command:

mpls label mode all-vrfs protocol all-afs per-prefix

You could change this to be per-vrf or per-ce

A PE router that has no import RT matching those of the received VPNv4 prefixes will discard the updates. Let’s see this in action:

R4#deb bgp vpnv4 u updates
BGP updates debugging is on for address family: VPNv4 Unicast
R4#clear bgp vpnv4 u * in
R4#
BGP(4): 5.5.5.5 rcvd UPDATE w/ attr: nexthop 2.2.2.2, origin ?, localpref 100, metric 0, originator 2.2.2.2, clusterlist 5.5.5.5, merged path 6, AS_PATH , extended community RT:2345:6
BGP(4): 5.5.5.5 rcvd 2.2.2.2:6:6.6.6.6/32, label 23 -- DENIED due to: extended community not supported;
R4#sh bgp vpnv4 u all 6.6.6.6/32
% Network not in table
R4#

As we can see the update is DENIED (discarded) because we do not have the RT 2345:6 in any import statements under any VRFs.

This is true on all VPNv4 routers except RRs. It makes sense. How else are RRs to be able to reflect prefixes from PEs to other PEs? They have need for installing the customers prefixes. They just must reflect the prefixes which is why RRs will not deny VPNv4 updates, but hold them in the BGP table for further reflection.

AS Override

Let’s say a customer decides to build a template for the CE devices using the same ASN for all routers. This causes reachability issues when updates are received in on other sites, because of the AS_PATH loop prevention built in to BGP (rejecting prefixes containing our own AS). Two BGP features exist for preventing this issue. You can either use the allowas-in on the CE devices, or the SP can help by using the as-override feature.

The neighbor as-override command looks at prefixes before they are sent to the other sites of a customer (so on the egress PE node). It compares the AS_PATH to the remote-as configured for the neighbor. If they match, the ASN is overridden (replaced) with the AS of the SP before sending the update to the CE router.

If for example R6, R7, and R8 are in AS 78 and R6 sends an update for network 66.66.66.66/32 to the SP. This is of corse sent to R2 with 78 in the AS_PATH. The SP somehow forwards this update to the PE routers that will accept the update if they have an import statement under a VRF matching the extended community of the prefix. Let’s say R4 is the egress PE node we’re looking at. It will now check the remote-as configured for R7 comparing it to the AS_PATH of the prefix and overriding (replacing) any AS that matches that of the remote-as – even if R6 prepended its AS before sending it to R2. Let’s have a look at the prefix on R7:

R7#sh bgp 66.66.66.66/32
BGP routing table entry for 66.66.66.66/32, version 34
Paths: (2 available, best #2, table default)
  Advertised to update-groups:
     5
  Refresh Epoch 1
  2345 2345 66 2345
    10.0.58.5 from 10.0.78.8 (8.8.8.8)
     Origin IGP, metric 0, localpref 100, valid, internal
     rx pathid: 0, tx pathid: 0
  Refresh Epoch 2
  2345 2345 66 2345
    10.0.47.4 from 10.0.47.4 (4.4.4.4)
     Origin IGP, localpref 100, valid, external, best
     rx pathid: 0, tx pathid: 0x0
R7#

Here we see that R6 did a prepend of the 78 66 78. The 78 AS (configured for R7 on R4 using the remote-as command) has now been replaced with 2345 66 2345 and of course prepended with the SP AS before sending it to R7 making the AS_PATH complete with: 2345 2345 66 2345

BGP SoO

The as-override feature disables the AS_PATH loop prevention of BGP, because now the AS_PATH does not contain the receiving routes own AS. This can cause loops when a site is multi homed like R7 and R8 are in this case. This calls for another feature configured on the SP – BGP SoO (Site of Origin). Let’s see how this works.

router bgp 2345
!
address-family ipv4 vrf b
neighbor 10.0.47.7 remote-as 78
neighbor 10.0.47.7 activate
neighbor 10.0.47.7 as-override
neighbor 10.0.47.7 soo 2345:78
exit-address-family
R4#

Above is the BGP configuration for VRF b of R4. Here both as-override and SoO have been configured for R7. The SoO value must be the same on all PE routers that peer with this site (hence the name Site of Origin). When configured, BGP will check the SoO extended community value of a prefix and compare this value to the SoO value configured for the peer. If they match, the PE node will not send this prefix to the peer and a loop is avoided. We can verify if prefixes are being denied due to SoO by looking at the neighbor:

R4#sh bgp vpnv4 u vrf b neighbors 10.0.47.7
! output omitted for brevity
                               Outbound Inbound
 Local Policy Denied Prefixes: -------- -------
   AS_PATH loop:                    n/a       7
   SOO loop:                          3     n/a
   Bestpath from this peer:           8     n/a
   Total:                            11       7
#R4

 

 

 

 

Jacob Zartmann avatar
Jacob Zartmann
Passionate Network Engineer thriving for challenges and knowledge.