Label Distribution Protocol (LDP) is one of the protocols that can be used for MPLS to distribute labels. Other protocols are RSVP-TE, BGP, and IGPs (ISIS and OSPF). This short post addresses LDP and how it works.

I’m using two routers to talk about LDP in this post. R2 and R3 that are connected like this:


LDP is very simple to configure. It is basically just one command needed.

! R2 (IOS-XE)
interface GigabitEthernet1.23
 encapsulation dot1Q 23
 ip address
 ip router isis 1
 mpls ip
 isis network point-to-point

For IOS XR the configuration is this:

mpls ldp
 interface GigabitEthernet0/0/0/0.23


LDP establishes adjacencies between directly connected routers. It does so by using a multicast hello packet that looks like this:

This is an LDP Hello Message sent by LSR (Label Switching Router) R3. This we can see by looking at the IP header which is sourced from We can also look inside the MPLS header and find the LSR ID of which is R3’s Loopback0 address. The destination of the packet is which is the link local all routers multicast address. And the TTL (not shown) is set to 1. So this truly is a link local discovery message asking to see if any other LDP enabled routers are available on the link. The UDP port numbers are 646 both for the source and destination ports. Finally, in the LDP header, we see an IPv4 transport address. This lets other LDP enabled routers known to which address they should form a TCP session once they’ve discovered each other. So this address has to be routable.

Next in the process of establishing an adjacency the routers send LDP Initialization Messages:

These messages are sent after the three way TCP handshake and lets each know the capabilities of each other along with an authentication TLV.

Finally LDP exchanges labels using Address packets:

First (not expanded) is an LDP header that contains a keepalive. The last LDP header contains the addresses bound to ourselves (essentially all our interface addresses). Here we also see the various Label Mapping Messages that contains the actual labels to be used by the neighbor (R2). I’ve expanded one of them listing the prefix which is the loopback of R3 (the advertising router). The label value is 3 which is actually referred to as an implicit-null label. This means that the router receiving this label (R2) should pop (remove) the label before sending the packet to this router (R3). We call this process PHP (Penultimate Hop Popping).


We can verify LDP by looking at R2:

R2#sh mpls ldp bindings 32
  lib entry:, rev 6
        local binding:  label: 200
        remote binding: lsr:, label: imp-null

This is the LRIB (Label Routing Information Base) of R2 for prefix Here we see the label that we (R2) has allocated for (200) and also we see the received binding from R3 which is the imp-null label.

The LFIB (Label Forwarding Information Base) also verifies this:

R2#sh mpls forwarding-table 32 
Local      Outgoing   Prefix         Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id   Switched      interface              
200        Pop Label     0             Gi1.23   

If R2 had received multiple remote bindings for it would pick the label advertised by the LDP neighbor found by looking at the routing table and the LDP neighbor table. Let me demonstrate this. Again looking at R2:

R2#sh ip route
Routing entry for
  Known via "isis", distance 115, metric 10, type level-2
  Redistributing via isis 1
  Last update from on GigabitEthernet1.23, 3d20h ago
  Routing Descriptor Blocks:
  *, from, 3d20h ago, via GigabitEthernet1.23
      Route metric is 10, traffic share count is 1

We have a route for received from R3 ( The next hop is out Gi1.23. This information, in particular the next hop, can be used to decided from which neighbor we should pick a label for We do so by looking at all our LDP neighbors:

R2#sh mpls ldp neighbor
    Peer LDP Ident:; Local LDP Ident
        TCP connection: -
        State: Oper; Msgs sent/rcvd: 84/83; Downstream
        Up time: 01:04:08
        LDP discovery sources:
          GigabitEthernet1.23, Src IP addr:
        Addresses bound to peer LDP Ident:

The important piece of information here is the Addresses bound to peer LDP Ident. This is where we correlate the next hop to the LDP neighbor. In this case R3 as the LDP neighbor. Now we know that R2 must use the imp-null (pop) action when sending packets to

mVPN – Profile 0 aka. “Rosen Draft”

mVPN Profile 0 is the original way of doing mVPN when we had no extensions to LDP. This means we need PIM in the underlay – the SP core. Specifically the profile is called:

Profile 0 Default MDT - GRE - PIM C-mcast Signaling


I’ll use the following topology to go through and explain the concepts and workings of Rosen Draft with Data MDT.

Default MDT

MDT (Multicast Distribution Tree) is referring to the SPs global environment – the underlay. And the Default MDT is what is used to connect the PEs together in a LAN like NBMA fashion where every PE will become PIM neighbors with all other PEs. We need the Default MDT to provide the underlay for the customers multicast traffic that travels inside the mVRF (multicast VRF).

To be able to join the Default MDT the PEs need the sources of every other PEs. How does the PE know the source IPs of the other PEs? We use BGP IPv4 MDT for the signaling of the (S,G) that should be joined for the Default MDT. SSM is the only viable solution with the Default MDT although you could do ASM, but why would you? So the configuration in the cores is really simple:

! P routers
ip multicast-routing distributed
ip pim ssm default
int x/y
 ip pim sparse-mode
! PE routers
ip multicast-routing distributed
ip multicast-routing vrf x distributed
ip pim ssm default
int x/y
 ip pim sparse-mode 
int Loopback0
 ip pim sparse-mode
router bgp 65000
 neighbor rr peer-group
 neighbor rr remote-as 65000
 neighbor rr update-source Loopback0
 neighbor peer-group rr
 address-family vpnv4
  neighbor rr send-community extended
  neighbor activate
 address-family ipv4 mdt
  neighbor activate
 address-family ipv4 vrf a
  redistribute connected metric 0
! RR
router bgp 65000
 neighbor rr-client peer-group
 neighbor rr-client remote-as 65000
 neighbor rr-client update-source Loopback0
 neighbor peer-group rr-client
 neighbor peer-group rr-client
 address-family vpnv4
  neighbor rr-client send-community extended
  neighbor rr-client route-reflector-client
  neighbor activate
  neighbor activate
 address-family ipv4 mdt
  neighbor rr-client route-reflector-client
  neighbor activate
  neighbor activate

On the PE routers you will need the regular RP configuration or SSM configuration in the mVRF.

When you configure the Default MDT group in the VRF, a MTI (Multicast Tunnel Interface) is created. It is of type mGRE using the VPNv4 update-source as the source interface and it is also unnumbered to this interface. Here is what the configuration looks like:

interface Tunnel0
 ip unnumbered Loopback0
 no ip redirects
 ip mtu 1500
 tunnel source Loopback0
 tunnel mode gre multipoint

Although we can’t see that the tunnel is a member of a VRF from the derived-config, it is clearly seen when viewing the VRF:

R4#sh vrf
  Name         Default RD            Protocols   Interfaces
  a              ipv4        Gi1.46

It is a bit like having a backdoor VRF with a DMVPN tunnel interface and the underlay (source) in global (no frontdoor VRF).

The VRF is configured like this:

! VRF a on R4
vrf definition a
 route-target export 65000:10
 route-target import 65000:10
 address-family ipv4
  mdt default
  mdt data threshold 1
  mdt data threshold 1

Ignore the mdt data part for now. We see the mdt default configuration with the group You must use uniq groups per VRF.

To join the other PEs we need their addresses. This is received using BGP IPv4 MDT:

R4#sh bgp ipv4 mdt all
BGP routing table entry for version 2
Paths: (1 available, best #1, table IPv4-MDT-BGP-Table)
  Not advertised to any peer
  Refresh Epoch 2
  Local from (
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Originator:, Cluster list:,
      MDT group address:

      rx pathid: 0, tx pathid: 0x0

Here we see an update that originated from R2 and was reflected by R3. It has a next-hop matching its VPNv4 update-source which is Loopback0 with an IP address of Also it contains the MDT group address for the Default MDT. Now we can join the Default/MDT. This information is used by PIM:

R4#sh ip pim mdt bgp 
MDT (Route Distinguisher + IPv4)    Router ID    Next Hop
  MDT group     

And we can see in the multicast routing table (in global) that we in fact did join the Default MDT:

R4#sh ip mroute
IP Multicast Routing Table
Flags: s - SSM Group
T - SPT-bit set,
I - Received Source Specific Host Report, 
Z - Multicast Tunnel

(,, 00:00:30/00:02:59, flags: sT
  Incoming interface: Loopback0, RPF nbr
  Outgoing interface list:
    GigabitEthernet1.34, Forward/Sparse, 00:00:30/00:02:59
(,, 00:28:51/stopped, flags: sTIZ
  Incoming interface: GigabitEthernet1.34, RPF nbr
  Outgoing interface list:
    MVRF a, Forward/Sparse, 00:28:51/00:01:08


This should give us a PIM neighbor over the MTI in VRF a:

R4#sh ip pim vrf a nei
PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
      P - Proxy Capable, S - State Refresh Capable, G - GenID Capable,
      L - DR Load-balancing Capable
Neighbor        Interface             Uptime/Expires    Ver   DR
Address                                                            Prio/Mode         Tunnel0               00:02:29/00:01:43 v2    1 / S P G

A bit like Layer 3 MPLS VPNs where we transit the SP core using the global table. So the customer multicast traffic in the overlay travels over the MTI which uses the Default MDT (the underlay).

At this point we have no mroute entries in VRF a, but the network is ready to serve the customers multicast traffic via the Default MDT.

Data MDT

Why do we need a Data MDT? Well, the Default MDT is joined by all PEs, meaning that all PEs will receive all multicast packets. This is not very efficient and reminds us of the old days where we had PIM dense mode. The idea of the Data MDT is that once a certain threshold (in kbps) is passed, the ingress PE (so the PE with the source behind it) will send a PIM Data-MDT Join message over the Default MDT so all egress PEs receive it. The message contains (S,G,MDT) where MDT is a new multicast group taken from a range of addresses specified under the VRF. In this case that would be

The PIM Data-MDT Join message looks like this:

So the destination is the All PIM Routers address of the inner IP packet. Its a UDP packet and the Data part should hold the (S,G,MDT) information. The outer IP header uses the Default MDT group address as destination.

Immediately we receive a PIM Join message initiate by the egress PE. This is sent upstream towards the ingress PE. It looks like this:

After 3 seconds the ingress PE stops sending the stream on the Default MDT and now sends it only over the Data MDT, meaning that only those egress PEs with receivers behind them will receive the traffic. Here is the wireshark to confirm this:

The Data-MDT Join in the top (red) was at time 47.59. The next ICMP Echo sent is at 49.18 which isn’t after the 3 seconds, so this was sent over the Default MDT. Now the highlighted packet shows the new Data MDT Group which is (the first available IP in the Data MDT range). The switchover has happened and the Default MDT is no longer used for this communication to (the inner multicast destination – the customer packet).
The PEs that currently do not have receivers behind them, will cache the Data-MDT Join information for fast signaling if a receiver should send a report requesting the stream.

Data MDT groups can be reused. They time out after a period of inactivity.


As the profile name says we’re using GRE to transport the customers traffic across the SP core. Actually the packet is multicast in multicast. NO MPLS is used in the forwarding of multicast! But why do we need MPLS then? Well, multicast relies heavily on RPF (Reverse Path Forwarding) check to avoid loops and duplicate packets. So for the mVRF we must have a unicast IP address for RPF checks. For this we use normal Layer 3 MPLS VPNs that does require MPLS.


RPF in the mVRF can be a bit confusing. Here I’m thinking about how the egress PE (so the PE with receivers behind it) does RPF check when the multicast stream is received on the MTI.

R4#sh ip rpf vrf a
RPF information for ? (
  RPF interface: Tunnel0
  RPF neighbor: ? (
  RPF route/mask:
  RPF type: unicast (bgp 65000)
  Doing distance-preferred lookups across tables
  BGP originator:
  RPF topology: ipv4 multicast base, originated from ipv4 unicast base

The MTI isn’t used for unicast routing. So the RPF check for the customer source prefix will not use the MTI as the RPF interface hence RPF will fail. To go about this unfortunate situation, the RPF interface is set to the MTI that is associated with that mVRF. And if the RPF interface is set to the MTI, the RPF neighbor is set to the customer source (VPNv4 prefix) BGP next-hop which is the ingress PE. This next-hop must also be the PIM neighbor in the mVRF table.

Ubuntu netplan static IP address

It has been a few years since I last touched a Linux box. Tonight I decided to install an Ubuntu Server and wanted to give it a static IP address. As I went to /etc/network and wanted to edit the interfaces file, I realised that things have changed. Now you configure the NICs using netplan.

So I fired up putty after reading some examples on how to do this and got hit by this error when trying to apply the configuration:

Invalid YAML at /etc/netplan/eth.yaml line 10 column 0: found character that cannot start any token

Turns out that I had used tab to do indents of the config. I change my tabs to spaces and it all worked out. My config is this now:

    version: 2
          addresses: [ "" ]
            addresses: [,]

It is applied using:

sudo netplan apply

If you do not receive any errors verify with


Deploy ASAv using Ovftool

If you ever get stuck with the “A required disk image was missing” error when deploying an OVA directly on an ESXi host, you might want to check this out – ASAv specific, but should work for other OVAs like I wrote about regarding Deploy ISE PoV 2.3 OVA using ovftool

First of all you download the ASAv from CCO and unzip it. Second you create a text file with the command to deploy the OVA on the ESXi host. My file looks like this:

ovftool \
--name="ASAv_FW" \
--deploymentOption=ASAv5 \
--diskMode=thin \
--datastore=DS-Data-LUN \
--acceptAllEulas \
--noSSLVerify \
--allowExtraConfig \
--net:Management0-0="mgmt-net" \
--net:GigabitEthernet0-0="lab-net" \
--net:GigabitEthernet0-1="lab-net" \
--net:GigabitEthernet0-2="lab-net" \
--net:GigabitEthernet0-3="lab-net" \
--net:GigabitEthernet0-4="lab-net" \
--net:GigabitEthernet0-5="lab-net" \
--net:GigabitEthernet0-6="lab-net" \
--net:GigabitEthernet0-7="lab-net" \
--net:GigabitEthernet0-8="lab-net" \
asav-esxi.ovf \

I placed it in the unzipped directory of ASAv to avoid struggling with full paths.

I’m a mac user, so I’ll need to first edit the PATH variable to include the path to the ovftool executable. Again to avoid using a full path to the file. To do this edit the file ~/.bash_profile to look something like this:

# Add path to VMware Ovftool
PATH="/Applications/VMware OVF Tool:${PATH}"
export PATH

Close the Terminal and reopen it.

Now the text file above needs to be executable:

# chmod 744 deploy_asav.cmd

Now all you need to do is run the file:

# ./deploy_asav.cmd

You’ll get prompted for the root password of the ESXi host and deployment will begin. It looks like this:After deployment you’ll need to change the Memory of the ASAv to 2GB or you’ll receive an error about starting webvpn when configuring ASDM (http ip mask interface).

Also you might want to configure serial access to the VM. This is done like this:

ciscoasa# cd coredumpinfo
ciscoasa# copy coredump.cfg disk0:/use_ttyS0

Shutdown the ASA and add a serial port to the VM for network, server with telnet://ip-of-esxi-host:port

You must enable the firewall rule on the ESXi host:

That’s it! Enjoy!



The DNAC is currently sold as an appliance (part number DN1-HW-APL). It costs a whopping $80k list per box! So why do you need three of them when doing a HA setup?

It is because of Quorum. The definition of quorum is:

"The number of members of a group or organization required to be present to transact business legally, usually a majority."

 - source: dictionary.com

Say you only have two hosts in a cluster. If they get partitioned, who should take the role of servicing the network and maintaining the database? This is where quorum comes into play. When you have a three node cluster and one of them loses network connectivity to the other two nodes, the “majority” of nodes lies with the two nodes that can see each other. Quorum is obtained and DNAC continues to function. Now data consistency is also ensured when the network to the isolated single host is restored. This is also the reason why you can only survive losing a single DNAC host. If all three gets cut off from each other, you will need to isolate one of them and reinitialize the other two – one at a time.

This fact about quorum is very important when deciding where to place the nodes. First of all they must be layer 2 adjacent. Second they should be physically close to each other. At least two of them. If you have a data center with two sites, perhaps it would be a good idea to place one of the DNAC boxes at one site and two of them together at the other site.

For more information, see:
Split Brain and Network Partition
Recover Failed Hosts in a Cluster

DNAC – External Authentication with ISE Radius

You probably want to use an existing Identity Store such as Active Directory when managing your network infrastructure – including DNAC. Below is a guide on how to configure this functionality.

When you enable external authentication in DNAC it will not exempt you from using the locally defined users on DNAC – at least not the built-in admin user.

DNAC External Authentication Configuration

Locate the “External Authentication” page in Settings -> System Settings -> Users

Here you define your ISE server IP address and the shared secret. Lastly you tick off the “Enable External User”. Do NOT modify the “AAA Attribute” default setting of “Cisco-AVPair”.

ISE Radius Configuration

I assume you already have ISE integrated with Active Directory. We must add the Active Directory group to ISE for use in the policy set later.

Go to Administration -> Identity Management -> External Identity Store -> AD (or whatever you called your Active Directory store) -> Groups

Here I have added a DNAC-Admins group:

To configure ISE to let DNAC use it as a AAA server, you must first add DNAC  as a Network Device in ISE.

Go to Administration -> Network Resources -> Network Devices

Here you add the DNAC by filling out the Name, IP address, Device Profile (Cisco), and Shared Secret:

Next go to Policy -> Policy Elements -> Results -> Authentication -> Allowed Protocols

Add a new Allowed Protocol Service like this:

Under Policy -> Policy Elements -> Results -> Authorization -> Authorization Profiles you add a new authorization profile for the ACCESSS-ACCEPT message we’ll use later in our policy set.

I called mine DNAC_Super_Admin, but the important part is the Advanced Attributes Settings where you must select Cisco:cisco-av-pair=Role=SUPER-ADMIN-ROLE

Attributes Details in the bottom should read:

cisco-av-pair = Role=SUPER-ADMIN-ROLE

Finally we should be able to create the policy set. Go to Policy -> Policy Sets and add a new policy for our DNAC-Admin policy:

Once created using the DNAC IP address as a condition, save it and modify it by clicking on the sign to the far right of the policy.

Now we can put in the Identity Store AD under the Authentication Policy and add an Authorization Policy containing the DNAC-Admins group as a condition and our DNAC_Super_Admin profile:

Hit save and you should be good to go.

Now in DNAC under Settings -> System Settings -> Users -> External Authentication you should see the external Users that have successfully logged on.


CCIE Lab Passed

April 4 2018 I did my 4th attempt of the CCIE R&S Lab in Brussels. This time I passed!

Now I know how to approach the lab and through my failed attempts I learned what works for me in terms of strategy. I will go through it here:

First of all you need to master all topics in the blueprint. This goes without saying. Not only do you need to know the technologies inside out, but you must also master how to troubleshoot, diagnose, and configure them – fast!

Second (for me at least) you should keep track of the troubleshooting tickets as well as tasks on the configuration section. What works for me is to create a table with these columns:

T    P   R   V   Notes

T: Ticket/Task
P: Points
R: Resolved
V: Verified

I only use the notes column in the configuration section where task inter-dependencies can be tricky. For example a note in L3 might read:


UL is short for underlay.

The strategy I used for TS was to make a note of the overall lab start time. For each ticket I would also note the start time and give myself 8-10 minutes. If I didn’t have a solution or was close and confident that I could do it quickly, I would make a note about it and move forward. This is key throughout the lab – DO NOT GET STUCK!

In the configuration section the strategy was to do L2 without reading through it. Then I would read the rest of the tasks and fill out the table to create an overview of what is required. Look at the drawings and get yourself familiar with the topology. Then pick the tasks that makes sense to do first. For example you can’t configure an overlay without first having configured the underlay. Or you can’t match certain outputs they ask of you without solving other tasks. It is up to you to understand and make this happen.

The CCIE R&S lab is a performance test.

I spend hundreds of hours studying for this exam and it took me 3-4 years. Finally I can rightfully use this logo:




DNA Center – Reinstall

If you messed up your DNAC or just want to start over, you can do so by downloading the ISO for the appliance. Get the ISO on the below link:

Download DNA Center ISO from CCO

After downloading the ISO, you must create a USB installer by using Etcher

If you use Rufus or any other tools for the USB installer creation, it might not work due to insanely long file names in the ISO.

You must also NOT mount the ISO with CIMC and install over the network. Although it might work, Cisco discourages using this approach, because they’ve seen many installs go bad. I’ve been there, too!

Be sure to read the release notes carefully! Especially regarding the order of which you install packages after the initial ISO install.

That’s it for this post. I hope you get your DNAC up and running! And remember that SDA is a journey and it will take time – a lot of time!


DNAC Integration with ISE using a self-signed Certificate

NOTE! Using ISE 2.3p3 is not an option due to CSCvi94778

If you’re deploying a DNAC and you want to integrate with ISE, you might have read the following documents:

Perform Post-Installation Tasks
Cisco ISE Integration Limitations

I did and ended up with this error in DNAC when adding ISE:

Clearly this is a certificate error. The thing is that Cisco mentions that SAN (Subject Alternate Name) is essential for the trust between DNAC and ISE. They state this:

The IP address or FQDN of both Cisco ISE and DNA Center should be present in either the Subject Name field or the Subject Alt Name field of the corresponding certificates.

So I decided to use DNS for my SAN and got the above error! A colleague of mine decided to go with IPs instead which worked! Here is how you do it.

Create a file named req.conf with the following content:

[ ca ]
default_ca = CA_default

[ req ]
prompt = no
distinguished_name = req_distinguished_name
x509_extensions = v3_ca

C = DK
O = MyCompany
CN = dnac.sda.domain.lab

IP.1 =
IP.2 =
DNS.1 = dnac.sda.domain.lab

basicConstraints = CA:TRUE
subjectAltName = @alternate_names
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment

copy_extensions = copy

This serves as a configuration file for openssl. Specify both the real IP of the DNAC(s) and the VIP.

Next, use openssl to generate a self-signed certificate like this:

openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 1825 -config req.conf

The only thing needed is to upload the new certificate to DNAC. Browse to Settings -> Certificate and click Replace Certificate.

Click Upload/Activate and wait for 5-10 minutes. Refresh the page and you should be prompte to accept the newly (untrusted) certificate in your browser.

Now you can add ISE under Settings -> Authentication and Policy Servers

If it works, you should see this:

Now you must approve DNAC in ISE. Go to your ISE Web UI under Administration -> pxGrid Services

Here you’ll see this:

Select Total Pending Approval and select Approve all (click OK to confirm).

Now back to DNAC and you’ll see this:

All done. Success!

Deploy ISE PoV 2.3 OVA using ovftool

When you want to deploy the ISE Proof of Value OVA in a ESXi 6.5 this happens:

We create a new VM, specify the name and select the ova:

In the last step, you’ll receive an error that “A required disk image was missing.”

Most likely due to CSCvf26967

Instead of combining the 5 zip files you downloaded from box.cisco.com (.001-.005), you should extract them and use ovftool to deploy the vm.

You now have these files:

Deploy using ovftool:

ovftool --noSSLVerify --acceptAllEulas -ds="datastore-name" --network=network-for-vm -n=ISE-PoV-2.3 "e:\ISE 2.3-POV.ovf" vi://root:passw0rd!@ip-of-esxi-host/

This is what you’ll see:

After the deployment has completed successfully use the Web UI to change these values:

  • VM Compatibility
  • Guest OS to RHEL 7 (64-bit)
  • CPUs
  • Memory
  • NIC to E1000