Advanced Cisco Routing: DMVPN -- Point-to-Multipoint VPN Tunneling

A few years ago, I used to work for a service provider that operated in rural Alaska. By lower-48 standards, our network wasn’t terribly large — or at least, the logical topology wasn’t terribly large; the physical topology covered a rather large geographical region. Our major hub was a huge, bustling metropolis of about 5,000 (!) people.

This site also was where we located our hub router for the network. We had an extension site in Anchorage (naturally, since that was where most of our employees lived and worked), the hub site, a PoP at the hub site hanging off the hub router, and then multiple PoPs scattered across our service area, also linked off of the hub router. Because our own management network was built across our own service-provider network, we set up VPN tunnels from the hub router to each and every one of our sub-tended sites to provide secure management access to our network. Conceptually, it was a very simple model (and honestly, it might have been the only model our equipment would support at the time), but if you think that configuring a separate VPN tunnel for each site could be a bit of a chore, you are exactly right.

As I’m sure you’ve guessed by now, there is A Better Way to achieve these goals, a way that makes configuring and managing multiple sites sub-tended off of a single hub much less time consuming. Allow me to introduce you to DMVPN’s (Dynamic Multipoint VPN’s). As always, we’ll start with our network diagram:



R1 through R4 will be our management network, with R1 being the hub and R2 — R4 being the spokes. R5, R6 and R7 are the service provider network. In the real network that I managed, we had static, default routes on R1 through R4, and ran OSPF on our provider network. In this lab, we will run OSPF internally on both networks, and peer the management and provider networks with BGP, since that is a more common scenario for most people (being both provider and customer is fairly unusual). Also, running OSPF over a DMVPN topology introduces a few wrinkles that are worth covering, but I’m getting ahead of myself ;)

For addressing, I’ll be using 100.64.x.x addresses in place of public IP address ranges, and 192.168.x.0/24 for the inside interfaces on my management network. I’ll use 172.16.x.x IP space for the tunnel addressing. On the "Internet" routers, I’ll use 100.64.254.x/32 for the Loopback IP addresses, while 10.254.254.x/32 will be the Loopback IP addresses on my management routers.

Still with me? Good! Let’s start by setting up basic connectivity to each router, starting with the Internet routers (since there is nothing new on them):

R5:
interface Loopback0
ip address 100.64.254.5 255.255.255.255
no shut
!
interface FastEthernet0/0
ip address 100.64.0.1 255.255.255.252
no shut
!
interface FastEthernet0/1
ip address 100.64.0.5 255.255.255.252
no shut
!
interface FastEthernet1/0
ip address 100.64.0.9 255.255.255.252
no shut
!
router ospf 1138
router-id 100.64.254.5
log-adjacency-changes
passive-interface Loopback0
network 100.64.0.0 0.0.0.3 area 0.0.0.0
network 100.64.0.4 0.0.0.3 area 0.0.0.0
network 100.64.0.8 0.0.0.3 area 0.0.0.0
network 100.64.254.5 0.0.0.0 area 0.0.0.0
!

R6 and R7 are similar, and since there is nothing new here, I’ll skip those configs.

We’ll go ahead and configure the IP addressing on the FastEthernet and Loopback interfaces of R1, R2, R3 and R4 next. Again, nothing new, and nothing exciting, so I won’t belabor the config here, but make sure R1, R2, R3 and R4 can ping their respective gateways before proceeding. Once point-to-point connectivity between the management network and the service provider network is working, we’ll set up BGP peering between R1 and R5, R2 and R6, R3 and R5, and finally, R4 and R7:

R1:
router bgp 65511
bgp router-id 100.64.0.2
network 100.64.0.0 mask 255.255.255.252
neighbor 100.64.0.1 remote-as 65512
neighbor 100.64.0.1 activate

R5: router bgp 65512
bgp router-id 100.64.0.1
network 100.64.0.0 mask 255.255.255.252
neighbor 100.64.0.2 remote-as 65511
neighbor 100.64.0.2 activate
redist ospf 1138 metric 120
!
router ospf 1138 redist bgp 65512 sub metric 120
!

As you can see, the configurations are almost identical, aside from swapping the AS’ in the "router bgp..." and "neighbor..." statements, and swapping the IP addresses in the "bgp router-id..." and "neighbor..." statements. Also, on R5, we are redistributing the routes learned via BGP into OSPF. We are also redistributing OSPF routes into the BGP process. R6 and R7 will be configured similarly to R5, and R2, R3 and R4 will be configured similarly to R1. Again, nothing new so far.

But now, things will start to get interesting. Let’s set up the GRE tunnel on R1:

interface Tunnel0
ip address 172.16.0.1 255.255.255.0
ip nhrp map multicast dynamic
ip nhrp network-id 1
tunnel source 100.64.0.2
tunnel mode gre multipoint
no shut
!

Just like a normal GRE tunnel, we start with "interface Tunnel <blah>", and assign an IP address to the tunnel interface. Unlike a normal tunnel interface, we are assigning a /24. You can use whatever size subnet you want, but since it is a multipoint tunnel, it should probably be larger than a /30. The "tunnel source..." statement should look familiar also (if not, see the GRE Tunnel lab for a refresher).

However, there are a few differences between a DMVPN tunnel config and a standard, point-to-point tunnel config. One of the first things you’ll likely notice is that we have not specified any of the opposite endpoints. Instead, we used the command "tunnel mode gre multipoint" to explicitly state that we are creating a point-to-multipoint (hub-and-spoke) network. That’s the "dynamic" portion of the "Dynamic Multipoint VPN. Basically, the hub accepts tunnel requests from multiple spoke routers, and automatically establishes the tunnels on demand.

You'll also notice that, even though there are three spoke routers, the hub only has one tunnel interface. That's the "Multipoint" portion of the acronym ;) This raises a very interesting question. In a point-to-point circuit, it is trivial to determine the IP address of the next hop (if you are on a /30 or /31 network, there are only two usable IP addresses, and you are using one of them, right?). However, in a multipoint network, your tunnel interfaces are in a larger subnet. In our example, we are using a /24, meaning the other end of the tunnel could be any one of 253 possible IP addresses! How does the hub router know which IP address corresponds to which tunnel? If you look at the next two lines of the tunnel config, you’ll see the two "ip nhrp..." statements. NHRP ("Next Hop Resolution Protocol," see also CCIE or Null! for a good discussion on the topic) is the protocol that we use to determine the IP address of the other side of the multipoint tunnel. In much the same way that ARP maps IP addresses to Ethernet addresses, NHRP allows our routers to dynamically map IP addressing to the multipoint tunnels. In the "ip nhrp map multicast dynamic" statement, we are telling NHRP to dynamically create these mappings for our multipoint tunnels. However, you might have multiple tunnels on any given router, so by specifying different network ID's with the "ip nhrp network-id ..." statement, you can create multiple hub-and-spoke networks without them conflicting with one another. That’s it for the hub router. That wasn't too bad, was it?

We’ll use R2 as an example of the spoke router configuration; R3 and R4 will be very similar:

R2:
interface Tunnel0
ip address 172.16.0.2 255.255.255.0
ip nhrp map 172.16.0.1 100.64.0.2
ip nhrp map multicast 100.64.0.2
ip nhrp network-id 1
ip nhrp nhs 172.16.0.1
tunnel source 100.64.0.14
tunnel mode gre multipoint
no shut
!

Like the hub router, the spoke router contains the "ip address...," "tunnel source..." and "tunnel mode gre multipoint commands." It also contains a handful of "ip nhrp..." statements, but they are slightly more complex. First, the spoke router must know how to reach the hub router in order to send the tunnel request, so we start by telling the tunnel to create a connection to the IP address of the hub router’s outside interface (the tunnel source on the hub router). In other words, to reach 172.16.0.1 (the tunnel IP address on R1) use 100.64.0.2 (Fa1/0 on R1). Next, "ip nhrp map multicast 100.64.0.2" sets 100.64.0.2 (Fa1/0 on R1) as the destination for multicast or broadcast packets sent across the non-broadcast, multi-access, or NBMA, (ie., the DMVPN) network. If multicast or broadcast packets are sent across the NBMA network, R1 is responsible for forwarding them to other hosts participating in the network, so we are telling the tunnel interface to forward those packets to R1. The last new command on the spoke router is the "ip nhrp nhs 172.16.0.1" statement. With this line, we are telling the spoke router to use the "next-hop server" to forward traffic across the NBMA network.

Substitute the appropriate values for the IP address and tunnel source on R3 and R4, and you should have working tunnels between R1 and each of the spoke routers. To verify this, use the "sho dmvpn" command:

R2#sho dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
    N - NATed, L - Local, X - No Socket
    # Ent --> Number of NHRP entries with same NBMA peer
    NHS Status: E --> Expecting Replies, R --> Responding
    UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel0, IPv4 NHRP Details

IPv4 NHS: 172.16.0.1 RE
Type:Spoke, Total NBMA Peers (v4/v6): 1

# Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb    Target Network
----- --------------- --------------- ----- -------- ----- -----------------
    1     100.64.0.2      172.16.0.1    UP 01:21:34    S      172.16.0.1/32


R2#

As you can see from the snippet of output above, R2 has now established a tunnel connection to R1 (tunnel state is "Up" and next to "IPV4 NHS, we have the IP address of int tunnel0 on R1, followed by the flags "RE," verifying that the tunnel is responding and expecting replies). After duplicating the tunnel config on R3 and R4, you should see similar output on those routers, although each router will only show the connection to R1. This is a point-to-multipoint network, meaning that R2 cannot talk directly to R3 without going through R1 (sort of...actually R1 can broker connections between the spokes, but honestly, I’m not comfortable enough with the topic to go there yet). Assuming that you have copied the modified version of R2’s tunnel config to R3 and R4, you should have a completed point-to-multipoint VPN network now (w00t!). However, if you try to ping from the inside interface of R2, R3 or R4 to the inside interface of R1, you will most likely not be thrilled with the result:

R4#ping 192.168.1.1 source 192.168.4.1
<...snip...>
.....
Success rate is 0 percent (0/5)
R4#

Any ideas why? Of course! We haven’t set up any routing between the inside networks. When we configured BGP peering between the management network and the service provider network, we only advertised the outside interfaces of R1, R2, R3 and R4, since our service provider should not be aware of the inner workings of our network (unless we are using MPLS). In order to actually send traffic across the VPN tunnels, we need to enable a routing protocol over the tunnels. Easy enough, right? It should look something like this...:

R1:
router ospf 42
router-id 10.254.254.1
passive-interface default
no passive-interface Tunnel0
network 10.254.254.1 0.0.0.0 area 0.0.0.0
network 172.16.0.0 0.0.0.255 area 0.0.0.0
network 192.168.1.0 0.0.0.255 area 0.0.0.0

Again, after making the appropriate substitutions for the router-id and the advertised networks, we’ll make the same changes on the spoke routers, and...what is going on here?

R1(config)#
*Dec 13 15:44:15.947: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.3 on Tunnel0 from LOADING to FULL, Loading Done
*Dec 13 15:44:16.391: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.3 on Tunnel0 from FULL to DOWN, Neighbor Down: Adjacency forced to reset
R1(config)#
*Dec 13 15:44:20.483: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.2 on Tunnel0 from INIT to DOWN, Neighbor Down: Adjacency forced to reset
*Dec 13 15:44:20.535: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.3 on Tunnel0 from EXSTART to DOWN, Neighbor Down: Adjacency forced to reset
*Dec 13 15:44:20.699: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.4 on Tunnel0 from LOADING to FULL, Loading Done
R1(config)#
*Dec 13 15:44:22.735: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.4 on Tunnel0 from FULL to DOWN, Neighbor Down: Adjacency forced to reset
R1(config)#
*Dec 13 15:44:25.307: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.3 on Tunnel0 from LOADING to FULL, Loading Done
*Dec 13 15:44:25.555: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.3 on Tunnel0 from FULL to DOWN, Neighbor Down: Adjacency forced to reset
R1(config)#

Why is OSPF flapping?!?!

This is where I start to get in a little bit over my head. If I understand correctly, the issue is that OSPF is aware of the type of network configured across each OSPF-aware link. In this case, we have configured a NBMA network via multipoint GRE tunnels, but OSPF considers GRE tunnels to be a point-to-point network. While we had OSPF only running between R1 and R2, this was fine, but as soon as OSPF sees two neighbors across a single "point-to-point" link, it gets confused (understandably so!) and drops the neighbor relationship. To resolve this problem, at least under certain circumstances, make the following change to R1, R2, R3 and R4:
int tunnel 0
ip ospf network point-to-multipoint

Now that OSPF understands that int Tunnel0 is actually part of a multipoint network, it will allow all three spoke routers to participate in OSPF across the tunnel. Run "sho ip route" and "sho ip ospf neighbor" to verify that everything is working as expected (it should be), and you should be good to go!

Advanced Cisco Routing: GRE Tunnels

Suppose you had a network that spanned an untrusted carrier's network -- say, the Internet, for example. A wise admin probably would not be comfortable broadcasting company data across this untrusted network; any number of common LAN protocols, such as POP3, CIFS, HTTP, Telnet, etc. could potentially leak confidential data, like user names and passwords, to anyone who happened to be listening in. Even on relatively trustworthy networks such as leased lines or Metro-Ethernet E-Line or E-Tree services, a simple configuration error could result in propagating company data to other organizations on that network.

One solution to this problem is "tunneling" -- that is, encapsulating the raw TCP/IP traffic into another format. This is often used to create VPN's, or "Virtual Private Networks." A tunnel does not necessarily have to be encrypted, although it frequently is.

One protocol that is used on Cisco devices to create tunnels is GRE, which will be the focus of today's post.

Consider the following network:

In this example, I have created two LAN segments, one in Anchorage (R3 and R5) and one in Bethel(R4 and R6), connected by a service provider's network (R1 and R2). The configuration on R1 and R2 are very straightforward: turn up the necessary network interfaces, set up OSPF to share routes between R1 and R2, then configure BGP to peer with the Anchorage and Bethel routers (R3 and R4), all of which we've done before. R5 and R6 are also very straightforward: turn up FA1/0 on both routers, assign the appropriate IP addresses to each router, then set up a default route through R3 for R5 and through R4 for R6. As usual, these two routers are simply simulating PCs on the internal LAN.

R3 and R4 are the routers that where the magic happens in this lab. Essentially, a tunnel interface is just a logical interface on the router that is configured much like a physical interface. For GRE, the interface name is "Tunnel1" and the IP address of the tunnel interface is configured exactly like the IP address of any other interface on the router. On R3...:
interface Tunnel1
ip address 10.255.255.1 255.255.255.252


R4, of course, is configured with a tunnel IP address of 10.255.255.2/30.

Next, you must configure a tunnel source and a tunnel destination. That is, you must tell the router what interface you want the tunnel to terminate at on the local router ("tunnel source") and what router the far end of the interface should exist upon ("tunnel destination"). On R3, also under the "interface Tunnel1" configuration...:
tunnel source 172.16.0.2
tunnel destination 172.16.1.2


There is one additional caveat to consider, and this is one that is easy to forget. On TCP/IP networks, there is a parameter known as MTU size. Essentially, this is the largest size of a packet that a network device can forward without having to break the packet into smaller pieces. Consider it like this: if you have a shipment that is delivered to you in a semi truck, but you need to carry that shipment in your pickup truck, you will have to make several trips to deliver the load, right? Same thing in networking. If one network device can handle a 1500 byte packet, but the next device can only handle a 1400 byte packet, then the second network device will have to break the packet into a 1400 byte chunk, then send the remaining 100 bytes in a second chunk. This is called "fragmentation." MTU size becomes an issue with GRE tunnels, because GRE adds several bytes to the packet size when it encapsulates a TCP packet. Consequently, if your network can only support 1500 byte MTUs, and you are receiving 1500 byte packets from, say, a LAN interface, then you will have a GRE-encapsulated packet that is larger than 1500 bytes exiting through your WAN interface. You can avoid this by setting the MTU size and adjusting the maximum packet size that the tunnel will encapsulate. This is done with two commands in the tunnel interface setting:
ip mtu 1460
ip tcp adjust-mss 1420


The complete tunnel interface configuration looks like this:
interface Tunnel1
ip address 10.255.255.1 255.255.255.252
ip mtu 1460
ip tcp adjust-mss 1420
tunnel source 172.16.0.2
tunnel destination 172.16.1.2

There is one last factor to consider. Since tunnels are used to isolate certain types of traffic, for example, LAN traffic, you probably don't want to advertise all of your routes when tunneling between two networks. In this lab, I modeled a very common scenario: two LAN networks with a tunnel to allow them to communicate over a public, untrusted network. This is, in fact, exactly how a network I manage in my day job is built: two LAN networks with a VPN tunnel across the Internet to bridge them together. Obviously, I don't want to advertise routes to my internal, RFC-1918 networks to the Internet, nor do I want the entire Internet routing tables stored on my LAN routers. To address this, I set up BGP peering between R3 and R1, and between R4 and R2, to provide routes to public (Internet) hosts (note: yes, I'm using private IP space on all of these hosts. Just pretend that 192.168.0.0/16 is private IP space, but 10.0.0.0/8 and 172.16.0.0/12 are public for this lab). Then, I set up OSPF to advertise routes to my LAN networks only across the GRE tunnel. R3 and R4 are NOT sharing their OSPF routes with R1 or R2.

To do this, I first created the access list and route map to match the LAN network on each router. The config is identical on R3 and R4:
access-list 10 permit 192.168.0.0 0.0.255.255
!
route-map lan permit 10
match ip address 10

Then, I applied the route maps to the OSPF configuration on each router. On R3:
router ospf 2112
router-id 10.255.255.1
log-adjacency-changes
redistribute connected subnets route-map lan
network 10.255.255.0 0.0.0.3 area 0

...and on R4:
router ospf 2112
router-id 10.255.255.2
log-adjacency-changes
redistribute connected subnets route-map lan
network 10.255.255.0 0.0.0.3 area 0

For BGP, R3 and R4 are essentially set up to receive routes only. On R3:
router bgp 64513
bgp log-neighbor-changes
neighbor 172.16.0.1 remote-as 64512
!
address-family ipv4
neighbor 172.16.0.1 activate
no auto-summary
no synchronization
exit-address-family

...and on R4:
router bgp 64515
bgp log-neighbor-changes
neighbor 172.16.1.1 remote-as 64514
!
address-family ipv4
neighbor 172.16.1.1 activate
no auto-summary
no synchronization
exit-address-family

With that, I have a virtual connection from R3 to R4, as shown by the traceroute from R5 to R6:
R5#traceroute 192.168.1.2

Type escape sequence to abort.
Tracing the route to 192.168.1.2

1 192.168.0.1 0 msec 0 msec 8 msec
2 10.255.255.2 28 msec 24 msec 40 msec
3 192.168.1.2 32 msec 24 msec *
R5#

Notice that the traceroute does not show that it crossed 172.16.0.0/30, 10.0.0.0/30 or 172.16.1.0/30? It appears to go directly from R3 to R4. That is due to the tunnel that bridges R3 and R4 together. Even though, in reality, the traffic is indeed passing through the "ISP" network, to the LAN hosts, it is as if R3 has a direct physical connection to R4.