mVPN – Profile 0 aka. “Rosen Draft”

mVPN Profile 0 is the original way of doing mVPN when we had no extensions to LDP. This means we need PIM in the underlay – the SP core. Specifically the profile is called:

Profile 0 Default MDT - GRE - PIM C-mcast Signaling

Topology

I’ll use the following topology to go through and explain the concepts and workings of Rosen Draft with Data MDT.

Default MDT

MDT (Multicast Distribution Tree) is referring to the SPs global environment – the underlay. And the Default MDT is what is used to connect the PEs together in a LAN like NBMA fashion where every PE will become PIM neighbors with all other PEs. We need the Default MDT to provide the underlay for the customers multicast traffic that travels inside the mVRF (multicast VRF).

To be able to join the Default MDT the PEs need the sources of every other PEs. How does the PE know the source IPs of the other PEs? We use BGP IPv4 MDT for the signaling of the (S,G) that should be joined for the Default MDT. SSM is the only viable solution with the Default MDT although you could do ASM, but why would you? So the configuration in the cores is really simple:

! P routers
ip multicast-routing distributed
ip pim ssm default
!
int x/y
 ip pim sparse-mode
! PE routers
ip multicast-routing distributed
ip multicast-routing vrf x distributed
ip pim ssm default
!
int x/y
 ip pim sparse-mode 
!
int Loopback0
 ip pim sparse-mode
!
router bgp 65000
 neighbor rr peer-group
 neighbor rr remote-as 65000
 neighbor rr update-source Loopback0
 neighbor 3.3.3.3 peer-group rr
 !
 address-family vpnv4
  neighbor rr send-community extended
  neighbor 3.3.3.3 activate
 exit-address-family
 !
 address-family ipv4 mdt
  neighbor 3.3.3.3 activate
 exit-address-family
 !
 address-family ipv4 vrf a
  redistribute connected metric 0
 exit-address-family
! RR
router bgp 65000
 neighbor rr-client peer-group
 neighbor rr-client remote-as 65000
 neighbor rr-client update-source Loopback0
 neighbor 2.2.2.2 peer-group rr-client
 neighbor 4.4.4.4 peer-group rr-client
 !
 address-family vpnv4
  neighbor rr-client send-community extended
  neighbor rr-client route-reflector-client
  neighbor 2.2.2.2 activate
  neighbor 4.4.4.4 activate
 exit-address-family
 !
 address-family ipv4 mdt
  neighbor rr-client route-reflector-client
  neighbor 2.2.2.2 activate
  neighbor 4.4.4.4 activate
 exit-address-family

On the PE routers you will need the regular RP configuration or SSM configuration in the mVRF.

When you configure the Default MDT group in the VRF, a MTI (Multicast Tunnel Interface) is created. It is of type mGRE using the VPNv4 update-source as the source interface and it is also unnumbered to this interface. Here is what the configuration looks like:

interface Tunnel0
 ip unnumbered Loopback0
 no ip redirects
 ip mtu 1500
 tunnel source Loopback0
 tunnel mode gre multipoint

Although we can’t see that the tunnel is a member of a VRF from the derived-config, it is clearly seen when viewing the VRF:

R4#sh vrf
  Name         Default RD            Protocols   Interfaces
  a            4.4.4.4:10            ipv4        Gi1.46
                                                 Tu0

It is a bit like having a backdoor VRF with a DMVPN tunnel interface and the underlay (source) in global (no frontdoor VRF).

The VRF is configured like this:

! VRF a on R4
vrf definition a
 rd 4.4.4.4:10
 route-target export 65000:10
 route-target import 65000:10
 !
 address-family ipv4
  mdt default 232.1.0.10
  mdt data 232.1.1.0 0.0.0.255 threshold 1
  mdt data threshold 1
 exit-address-family

Ignore the mdt data part for now. We see the mdt default configuration with the group 232.1.0.10. You must use uniq groups per VRF.

To join the other PEs we need their addresses. This is received using BGP IPv4 MDT:

R4#sh bgp ipv4 mdt all 2.2.2.2/32
BGP routing table entry for 2.2.2.2:10:2.2.2.2/32 version 2
Paths: (1 available, best #1, table IPv4-MDT-BGP-Table)
  Not advertised to any peer
  Refresh Epoch 2
  Local
    2.2.2.2 from 3.3.3.3 (3.3.3.3)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Originator: 2.2.2.2, Cluster list: 3.3.3.3,
      MDT group address: 232.1.0.10

      rx pathid: 0, tx pathid: 0x0
R4#

Here we see an update that originated from R2 and was reflected by R3. It has a next-hop matching its VPNv4 update-source which is Loopback0 with an IP address of 2.2.2.2/32. Also it contains the MDT group address 232.1.0.10 for the Default MDT. Now we can join the Default/MDT. This information is used by PIM:

R4#sh ip pim mdt bgp 
MDT (Route Distinguisher + IPv4)    Router ID    Next Hop
  MDT group 232.1.0.10
   4.4.4.4:10:2.2.2.2               3.3.3.3      2.2.2.2
R4#

And we can see in the multicast routing table (in global) that we in fact did join the Default MDT:

R4#sh ip mroute 232.1.0.10
IP Multicast Routing Table
Flags: s - SSM Group
T - SPT-bit set,
I - Received Source Specific Host Report, 
Z - Multicast Tunnel

(4.4.4.4, 232.1.0.10), 00:00:30/00:02:59, flags: sT
  Incoming interface: Loopback0, RPF nbr 0.0.0.0
  Outgoing interface list:
    GigabitEthernet1.34, Forward/Sparse, 00:00:30/00:02:59
(2.2.2.2, 232.1.0.10), 00:28:51/stopped, flags: sTIZ
  Incoming interface: GigabitEthernet1.34, RPF nbr 10.0.34.3
  Outgoing interface list:
    MVRF a, Forward/Sparse, 00:28:51/00:01:08

R4#

This should give us a PIM neighbor over the MTI in VRF a:

R4#sh ip pim vrf a nei
PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
      P - Proxy Capable, S - State Refresh Capable, G - GenID Capable,
      L - DR Load-balancing Capable
Neighbor        Interface             Uptime/Expires    Ver   DR
Address                                                            Prio/Mode
2.2.2.2         Tunnel0               00:02:29/00:01:43 v2    1 / S P G
R4#

A bit like Layer 3 MPLS VPNs where we transit the SP core using the global table. So the customer multicast traffic in the overlay travels over the MTI which uses the Default MDT (the underlay).

At this point we have no mroute entries in VRF a, but the network is ready to serve the customers multicast traffic via the Default MDT.

Data MDT

Why do we need a Data MDT? Well, the Default MDT is joined by all PEs, meaning that all PEs will receive all multicast packets. This is not very efficient and reminds us of the old days where we had PIM dense mode. The idea of the Data MDT is that once a certain threshold (in kbps) is passed, the ingress PE (so the PE with the source behind it) will send a PIM Data-MDT Join message over the Default MDT so all egress PEs receive it. The message contains (S,G,MDT) where MDT is a new multicast group taken from a range of addresses specified under the VRF. In this case that would be 232.1.1.0/24.

The PIM Data-MDT Join message looks like this:

So the destination is the All PIM Routers address 224.0.0.13 of the inner IP packet. Its a UDP packet and the Data part should hold the (S,G,MDT) information. The outer IP header uses the Default MDT group address as destination.

Immediately we receive a PIM Join message initiate by the egress PE. This is sent upstream towards the ingress PE. It looks like this:

After 3 seconds the ingress PE stops sending the stream on the Default MDT and now sends it only over the Data MDT, meaning that only those egress PEs with receivers behind them will receive the traffic. Here is the wireshark to confirm this:

The Data-MDT Join in the top (red) was at time 47.59. The next ICMP Echo sent is at 49.18 which isn’t after the 3 seconds, so this was sent over the Default MDT. Now the highlighted packet shows the new Data MDT Group which is 232.1.1.0 (the first available IP in the Data MDT range). The switchover has happened and the Default MDT is no longer used for this communication to 232.5.5.5 (the inner multicast destination – the customer packet).
The PEs that currently do not have receivers behind them, will cache the Data-MDT Join information for fast signaling if a receiver should send a report requesting the stream.

Data MDT groups can be reused. They time out after a period of inactivity.

MPLS

As the profile name says we’re using GRE to transport the customers traffic across the SP core. Actually the packet is multicast in multicast. NO MPLS is used in the forwarding of multicast! But why do we need MPLS then? Well, multicast relies heavily on RPF (Reverse Path Forwarding) check to avoid loops and duplicate packets. So for the mVRF we must have a unicast IP address for RPF checks. For this we use normal Layer 3 MPLS VPNs that does require MPLS.

RPF

RPF in the mVRF can be a bit confusing. Here I’m thinking about how the egress PE (so the PE with receivers behind it) does RPF check when the multicast stream is received on the MTI.

R4#sh ip rpf vrf a 10.0.12.1
RPF information for ? (10.0.12.1)
  RPF interface: Tunnel0
  RPF neighbor: ? (2.2.2.2)
  RPF route/mask: 10.0.12.0/24
  RPF type: unicast (bgp 65000)
  Doing distance-preferred lookups across tables
  BGP originator: 2.2.2.2
  RPF topology: ipv4 multicast base, originated from ipv4 unicast base
R4#

The MTI isn’t used for unicast routing. So the RPF check for the customer source prefix will not use the MTI as the RPF interface hence RPF will fail. To go about this unfortunate situation, the RPF interface is set to the MTI that is associated with that mVRF. And if the RPF interface is set to the MTI, the RPF neighbor is set to the customer source (VPNv4 prefix) BGP next-hop which is the ingress PE. This next-hop must also be the PIM neighbor in the mVRF table.

Ubuntu netplan static IP address

It has been a few years since I last touched a Linux box. Tonight I decided to install an Ubuntu Server and wanted to give it a static IP address. As I went to /etc/network and wanted to edit the interfaces file, I realised that things have changed. Now you configure the NICs using netplan.

So I fired up putty after reading some examples on how to do this and got hit by this error when trying to apply the configuration:

Invalid YAML at /etc/netplan/eth.yaml line 10 column 0: found character that cannot start any token

Turns out that I had used tab to do indents of the config. I change my tabs to spaces and it all worked out. My config is this now:

network:
    version: 2
    ethernets:
        ens160:
          addresses: [ "10.10.10.22/22" ]
          gateway4: 10.10.10.1
          nameservers:
            addresses: [8.8.8.8, 9.9.9.9]

It is applied using:

sudo netplan apply

If you do not receive any errors verify with

ifconfig

Deploy ASAv using Ovftool

If you ever get stuck with the “A required disk image was missing” error when deploying an OVA directly on an ESXi host, you might want to check this out – ASAv specific, but should work for other OVAs like I wrote about regarding Deploy ISE PoV 2.3 OVA using ovftool

First of all you download the ASAv from CCO and unzip it. Second you create a text file with the command to deploy the OVA on the ESXi host. My file looks like this:

ovftool \
--name="ASAv_FW" \
--deploymentOption=ASAv5 \
--diskMode=thin \
--datastore=DS-Data-LUN \
--acceptAllEulas \
--noSSLVerify \
--allowExtraConfig \
--net:Management0-0="mgmt-net" \
--net:GigabitEthernet0-0="lab-net" \
--net:GigabitEthernet0-1="lab-net" \
--net:GigabitEthernet0-2="lab-net" \
--net:GigabitEthernet0-3="lab-net" \
--net:GigabitEthernet0-4="lab-net" \
--net:GigabitEthernet0-5="lab-net" \
--net:GigabitEthernet0-6="lab-net" \
--net:GigabitEthernet0-7="lab-net" \
--net:GigabitEthernet0-8="lab-net" \
asav-esxi.ovf \
vi://root@ip-of-esxi-host/

I placed it in the unzipped directory of ASAv to avoid struggling with full paths.

I’m a mac user, so I’ll need to first edit the PATH variable to include the path to the ovftool executable. Again to avoid using a full path to the file. To do this edit the file ~/.bash_profile to look something like this:

# Add path to VMware Ovftool
PATH="/Applications/VMware OVF Tool:${PATH}"
export PATH

Close the Terminal and reopen it.

Now the text file above needs to be executable:

# chmod 744 deploy_asav.cmd

Now all you need to do is run the file:

# ./deploy_asav.cmd

You’ll get prompted for the root password of the ESXi host and deployment will begin. It looks like this:After deployment you’ll need to change the Memory of the ASAv to 2GB or you’ll receive an error about starting webvpn when configuring ASDM (http ip mask interface).

Also you might want to configure serial access to the VM. This is done like this:

ciscoasa# cd coredumpinfo
ciscoasa# copy coredump.cfg disk0:/use_ttyS0

Shutdown the ASA and add a serial port to the VM for network, server with telnet://ip-of-esxi-host:port

You must enable the firewall rule on the ESXi host:

That’s it! Enjoy!

 

DNAC HA

The DNAC is currently sold as an appliance (part number DN1-HW-APL). It costs a whopping $80k list per box! So why do you need three of them when doing a HA setup?

It is because of Quorum. The definition of quorum is:

"The number of members of a group or organization required to be present to transact business legally, usually a majority."

 - source: dictionary.com

Say you only have two hosts in a cluster. If they get partitioned, who should take the role of servicing the network and maintaining the database? This is where quorum comes into play. When you have a three node cluster and one of them loses network connectivity to the other two nodes, the “majority” of nodes lies with the two nodes that can see each other. Quorum is obtained and DNAC continues to function. Now data consistency is also ensured when the network to the isolated single host is restored. This is also the reason why you can only survive losing a single DNAC host. If all three gets cut off from each other, you will need to isolate one of them and reinitialize the other two – one at a time.

This fact about quorum is very important when deciding where to place the nodes. First of all they must be layer 2 adjacent. Second they should be physically close to each other. At least two of them. If you have a data center with two sites, perhaps it would be a good idea to place one of the DNAC boxes at one site and two of them together at the other site.

For more information, see:
Split Brain and Network Partition
Recover Failed Hosts in a Cluster

DNAC – External Authentication with ISE Radius

You probably want to use an existing Identity Store such as Active Directory when managing your network infrastructure – including DNAC. Below is a guide on how to configure this functionality.

When you enable external authentication in DNAC it will not exempt you from using the locally defined users on DNAC – at least not the built-in admin user.

DNAC External Authentication Configuration

Locate the “External Authentication” page in Settings -> System Settings -> Users

Here you define your ISE server IP address and the shared secret. Lastly you tick off the “Enable External User”. Do NOT modify the “AAA Attribute” default setting of “Cisco-AVPair”.

ISE Radius Configuration

I assume you already have ISE integrated with Active Directory. We must add the Active Directory group to ISE for use in the policy set later.

Go to Administration -> Identity Management -> External Identity Store -> AD (or whatever you called your Active Directory store) -> Groups

Here I have added a DNAC-Admins group:

To configure ISE to let DNAC use it as a AAA server, you must first add DNAC  as a Network Device in ISE.

Go to Administration -> Network Resources -> Network Devices

Here you add the DNAC by filling out the Name, IP address, Device Profile (Cisco), and Shared Secret:

Next go to Policy -> Policy Elements -> Results -> Authentication -> Allowed Protocols

Add a new Allowed Protocol Service like this:

Under Policy -> Policy Elements -> Results -> Authorization -> Authorization Profiles you add a new authorization profile for the ACCESSS-ACCEPT message we’ll use later in our policy set.

I called mine DNAC_Super_Admin, but the important part is the Advanced Attributes Settings where you must select Cisco:cisco-av-pair=Role=SUPER-ADMIN-ROLE

Attributes Details in the bottom should read:

Access Type = ACCESS_ACCEPT
cisco-av-pair = Role=SUPER-ADMIN-ROLE

Finally we should be able to create the policy set. Go to Policy -> Policy Sets and add a new policy for our DNAC-Admin policy:

Once created using the DNAC IP address as a condition, save it and modify it by clicking on the sign to the far right of the policy.

Now we can put in the Identity Store AD under the Authentication Policy and add an Authorization Policy containing the DNAC-Admins group as a condition and our DNAC_Super_Admin profile:

Hit save and you should be good to go.

Now in DNAC under Settings -> System Settings -> Users -> External Authentication you should see the external Users that have successfully logged on.

 

CCIE Lab Passed

April 4 2018 I did my 4th attempt of the CCIE R&S Lab in Brussels. This time I passed!

Now I know how to approach the lab and through my failed attempts I learned what works for me in terms of strategy. I will go through it here:

First of all you need to master all topics in the blueprint. This goes without saying. Not only do you need to know the technologies inside out, but you must also master how to troubleshoot, diagnose, and configure them – fast!

Second (for me at least) you should keep track of the troubleshooting tickets as well as tasks on the configuration section. What works for me is to create a table with these columns:

T    P   R   V   Notes

T: Ticket/Task
P: Points
R: Resolved
V: Verified
Notes

I only use the notes column in the configuration section where task inter-dependencies can be tricky. For example a note in L3 might read:

Site1 OSPF - MPLS UL

UL is short for underlay.

The strategy I used for TS was to make a note of the overall lab start time. For each ticket I would also note the start time and give myself 8-10 minutes. If I didn’t have a solution or was close and confident that I could do it quickly, I would make a note about it and move forward. This is key throughout the lab – DO NOT GET STUCK!

In the configuration section the strategy was to do L2 without reading through it. Then I would read the rest of the tasks and fill out the table to create an overview of what is required. Look at the drawings and get yourself familiar with the topology. Then pick the tasks that makes sense to do first. For example you can’t configure an overlay without first having configured the underlay. Or you can’t match certain outputs they ask of you without solving other tasks. It is up to you to understand and make this happen.

The CCIE R&S lab is a performance test.

I spend hundreds of hours studying for this exam and it took me 3-4 years. Finally I can rightfully use this logo:

 

 

 

DNA Center – Reinstall

If you messed up your DNAC or just want to start over, you can do so by downloading the ISO for the appliance. Get the ISO on the below link:

Download DNA Center ISO from CCO

After downloading the ISO, you must create a USB installer by using Etcher

If you use Rufus or any other tools for the USB installer creation, it might not work due to insanely long file names in the ISO.

You must also NOT mount the ISO with CIMC and install over the network. Although it might work, Cisco discourages using this approach, because they’ve seen many installs go bad. I’ve been there, too!

Be sure to read the release notes carefully! Especially regarding the order of which you install packages after the initial ISO install.

That’s it for this post. I hope you get your DNAC up and running! And remember that SDA is a journey and it will take time – a lot of time!

 

DNAC Integration with ISE using a self-signed Certificate

NOTE! Using ISE 2.3p3 is not an option due to CSCvi94778

If you’re deploying a DNAC and you want to integrate with ISE, you might have read the following documents:

Perform Post-Installation Tasks
Cisco ISE Integration Limitations

I did and ended up with this error in DNAC when adding ISE:

Clearly this is a certificate error. The thing is that Cisco mentions that SAN (Subject Alternate Name) is essential for the trust between DNAC and ISE. They state this:

The IP address or FQDN of both Cisco ISE and DNA Center should be present in either the Subject Name field or the Subject Alt Name field of the corresponding certificates.

So I decided to use DNS for my SAN and got the above error! A colleague of mine decided to go with IPs instead which worked! Here is how you do it.

Create a file named req.conf with the following content:

[ ca ]
default_ca = CA_default

[ req ]
prompt = no
distinguished_name = req_distinguished_name
x509_extensions = v3_ca

[req_distinguished_name]
C = DK
O = MyCompany
CN = dnac.sda.domain.lab

[alternate_names]
IP.1 = 10.0.0.101
IP.2 = 10.0.0.201
DNS.1 = dnac.sda.domain.lab

[v3_ca]
basicConstraints = CA:TRUE
subjectAltName = @alternate_names
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment

[CA_default]
copy_extensions = copy

This serves as a configuration file for openssl. Specify both the real IP of the DNAC(s) and the VIP.

Next, use openssl to generate a self-signed certificate like this:

openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 1825 -config req.conf

The only thing needed is to upload the new certificate to DNAC. Browse to Settings -> Certificate and click Replace Certificate.

Click Upload/Activate and wait for 5-10 minutes. Refresh the page and you should be prompte to accept the newly (untrusted) certificate in your browser.

Now you can add ISE under Settings -> Authentication and Policy Servers

If it works, you should see this:

Now you must approve DNAC in ISE. Go to your ISE Web UI under Administration -> pxGrid Services

Here you’ll see this:

Select Total Pending Approval and select Approve all (click OK to confirm).

Now back to DNAC and you’ll see this:

All done. Success!

Deploy ISE PoV 2.3 OVA using ovftool

When you want to deploy the ISE Proof of Value OVA in a ESXi 6.5 this happens:

We create a new VM, specify the name and select the ova:

In the last step, you’ll receive an error that “A required disk image was missing.”

Most likely due to CSCvf26967

Instead of combining the 5 zip files you downloaded from box.cisco.com (.001-.005), you should extract them and use ovftool to deploy the vm.

You now have these files:

Deploy using ovftool:

ovftool --noSSLVerify --acceptAllEulas -ds="datastore-name" --network=network-for-vm -n=ISE-PoV-2.3 "e:\ISE 2.3-POV.ovf" vi://root:passw0rd!@ip-of-esxi-host/

This is what you’ll see:

After the deployment has completed successfully use the Web UI to change these values:

  • VM Compatibility
  • Guest OS to RHEL 7 (64-bit)
  • CPUs
  • Memory
  • NIC to E1000

Enjoy!

ISR4321 Switch Module Not Working – CPLD Incompatibility

I recently had the pleasure of upgrading a ISR4321 router to Denali (16.3.5). If you have a NIM-ES2-8 for example you might want to be careful and check the CPLD version before doing the upgrade! Here is why.

Here the CPLD version is 14101324

The Firmware Version is the ROMMON version.

As of writing there is no way of correlating the CPLD version show in the output of show platform and the one you can download on CCO. And for ISR4321 there is only one version on CCO.

Why is this important? Well, if you upgrade to Denali this will show up in the log after the upgrade:

*Jan 17 2018 10:07:34.633 CET: %SPA_OIR-3-RECOVERY_RELOAD: subslot 0/1: Attempting recovery by reloading SPA
*Jan 17 2018 10:07:34.637 CET: %SPA_OIR-6-OFFLINECARD: SPA (NIM-ES2-8) offline in subslot 0/1

Router#sh hw-module subslot 0/1 oir
Module        Model                Operational Status
------------- -------------------- ------------------------
subslot 0/1   NIM-ES2-8            out of service(failed too many times)

After you upgrade the CPLD version the module will work again.

Router#show hw-module subslot 0/1 oir
Module        Model                Operational Status
------------- -------------------- ------------------------
subslot 0/1   NIM-ES2-8            ok

And as of writing this was not documented in the release notes or anywhere else in Cisco Documentation!