A few years ago, I used to work for a service provider that operated in rural Alaska. By lower-48 standards, our network wasn’t terribly large — or at least, the logical topology wasn’t terribly large; the physical topology covered a rather large geographical region. Our major hub was a huge, bustling
metropolis of about 5,000 (!) people.
This site also was where we located our hub router for the network. We had an extension site in Anchorage (naturally, since that was where most of our employees lived and worked), the hub site, a
PoP at the hub site hanging off the hub router, and then multiple PoPs scattered across our service area, also linked off of the hub router. Because our own management network was built across our own service-provider network, we set up VPN tunnels from the hub router to
each and every one of our sub-tended sites to provide secure management access to our network. Conceptually, it was a very simple model (and honestly, it might have been the only model our equipment would support at the time), but if you think that configuring a separate VPN tunnel for each site could be a bit of a chore, you are exactly right.
As I’m sure you’ve guessed by now, there is
A Better Way to achieve these goals, a way that makes configuring and managing multiple sites sub-tended off of a single hub much less time consuming. Allow me to introduce you to DMVPN’s (Dynamic Multipoint VPN’s). As always, we’ll start with our network diagram:
R1 through R4 will be our management network, with R1 being the hub and R2 — R4 being the spokes. R5, R6 and R7 are the service provider network. In the real network that I managed, we had static, default routes on R1 through R4, and ran OSPF on our provider network. In this lab, we will run OSPF internally on both networks, and peer the management and provider networks with BGP, since that is a more common scenario for most people (being both provider and customer is fairly unusual). Also, running OSPF over a DMVPN topology introduces a few wrinkles that are worth covering, but I’m getting ahead of myself ;)
For addressing, I’ll be using 100.64.x.x addresses in place of public IP address ranges, and 192.168.x.0/24 for the inside interfaces on my management network. I’ll use 172.16.x.x IP space for the tunnel addressing. On the "Internet" routers, I’ll use 100.64.254.x/32 for the Loopback IP addresses, while 10.254.254.x/32 will be the Loopback IP addresses on my management routers.
Still with me? Good! Let’s start by setting up basic connectivity to each router, starting with the Internet routers (since there is nothing new on them):
R5:
interface Loopback0
ip address 100.64.254.5 255.255.255.255
no shut
!
interface FastEthernet0/0
ip address 100.64.0.1 255.255.255.252
no shut
!
interface FastEthernet0/1
ip address 100.64.0.5 255.255.255.252
no shut
!
interface FastEthernet1/0
ip address 100.64.0.9 255.255.255.252
no shut
!
router ospf 1138
router-id 100.64.254.5
log-adjacency-changes
passive-interface Loopback0
network 100.64.0.0 0.0.0.3 area 0.0.0.0
network 100.64.0.4 0.0.0.3 area 0.0.0.0
network 100.64.0.8 0.0.0.3 area 0.0.0.0
network 100.64.254.5 0.0.0.0 area 0.0.0.0
!
R6 and R7 are similar, and since there is nothing new here, I’ll skip those configs.
We’ll go ahead and configure the IP addressing on the FastEthernet and Loopback interfaces of R1, R2, R3 and R4 next. Again, nothing new, and nothing exciting, so I won’t belabor the config here, but make sure R1, R2, R3 and R4 can ping their respective gateways before proceeding. Once point-to-point connectivity between the management network and the service provider network is working, we’ll set up BGP peering between R1 and R5, R2 and R6, R3 and R5, and finally, R4 and R7:
R1:
router bgp 65511
bgp router-id 100.64.0.2
network 100.64.0.0 mask 255.255.255.252
neighbor 100.64.0.1 remote-as 65512
neighbor 100.64.0.1 activate
R5:
router bgp 65512
bgp router-id 100.64.0.1
network 100.64.0.0 mask 255.255.255.252
neighbor 100.64.0.2 remote-as 65511
neighbor 100.64.0.2 activate
redist ospf 1138 metric 120
!
router ospf 1138
redist bgp 65512 sub metric 120
!
As you can see, the configurations are almost identical, aside from swapping the AS’ in the "router bgp..." and "neighbor..." statements, and swapping the IP addresses in the "bgp router-id..." and "neighbor..." statements. Also, on R5, we are redistributing the routes learned via BGP into OSPF. We are also redistributing OSPF routes into the BGP process. R6 and R7 will be configured similarly to R5, and R2, R3 and R4 will be configured similarly to R1. Again, nothing new so far.
But now, things will start to get interesting. Let’s set up the GRE tunnel on R1:
interface Tunnel0
ip address 172.16.0.1 255.255.255.0
ip nhrp map multicast dynamic
ip nhrp network-id 1
tunnel source 100.64.0.2
tunnel mode gre multipoint
no shut
!
Just like a normal GRE tunnel, we start with "interface Tunnel <blah>", and assign an IP address to the tunnel interface. Unlike a normal tunnel interface, we are assigning a /24. You can use whatever size subnet you want, but since it is a multipoint tunnel, it should probably be larger than a /30. The "tunnel source..." statement should look familiar also (if not, see
the GRE Tunnel lab for a refresher).
However, there are a few differences between a DMVPN tunnel config and a standard, point-to-point tunnel config. One of the first things you’ll likely notice is that we have not specified any of the opposite endpoints. Instead, we used the command "tunnel mode gre multipoint" to explicitly state that we are creating a point-to-multipoint (hub-and-spoke) network. That’s the "dynamic" portion of the "Dynamic Multipoint VPN. Basically, the hub accepts tunnel requests from multiple spoke routers, and automatically establishes the tunnels on demand.
You'll also notice that, even though there are three spoke routers, the hub only has one tunnel interface. That's the "Multipoint" portion of the acronym ;) This raises a very interesting question. In a point-to-point circuit, it is trivial to determine the IP address of the next hop (if you are on a /30 or /31 network, there are only two usable IP addresses, and you are using one of them, right?). However, in a multipoint network, your tunnel interfaces are in a larger subnet. In our example, we are using a /24, meaning the other end of the tunnel could be any one of 253 possible IP addresses! How does the hub router know which IP address corresponds to which tunnel? If you look at the next two lines of the tunnel config, you’ll see the two "ip nhrp..." statements.
NHRP ("Next Hop Resolution Protocol," see also
CCIE or Null! for a good discussion on the topic) is the protocol that we use to determine the IP address of the other side of the multipoint tunnel. In much the same way that
ARP maps IP addresses to Ethernet addresses, NHRP allows our routers to dynamically map IP addressing to the multipoint tunnels. In the "ip nhrp map multicast dynamic" statement, we are telling NHRP to dynamically create these mappings for our multipoint tunnels. However, you might have multiple tunnels on any given router, so by specifying different network ID's with the "ip nhrp network-id ..." statement, you can create multiple hub-and-spoke networks without them conflicting with one another. That’s it for the hub router. That wasn't too bad, was it?
We’ll use R2 as an example of the spoke router configuration; R3 and R4 will be very similar:
R2:
interface Tunnel0
ip address 172.16.0.2 255.255.255.0
ip nhrp map 172.16.0.1 100.64.0.2
ip nhrp map multicast 100.64.0.2
ip nhrp network-id 1
ip nhrp nhs 172.16.0.1
tunnel source 100.64.0.14
tunnel mode gre multipoint
no shut
!
Like the hub router, the spoke router contains the "ip address...," "tunnel source..." and "tunnel mode gre multipoint commands." It also contains a handful of "ip nhrp..." statements, but they are slightly more complex. First, the spoke router must know how to reach the hub router in order to send the tunnel request, so we start by telling the tunnel to create a connection to the IP address of the hub router’s outside interface (the tunnel source on the hub router). In other words, to reach 172.16.0.1 (the tunnel IP address on R1) use 100.64.0.2 (Fa1/0 on R1). Next, "ip nhrp map multicast 100.64.0.2" sets 100.64.0.2 (Fa1/0 on R1) as the destination for multicast or broadcast packets sent across the
non-broadcast, multi-access, or NBMA, (ie., the DMVPN) network. If multicast or broadcast packets are sent across the NBMA network, R1 is responsible for forwarding them to other hosts participating in the network, so we are telling the tunnel interface to forward those packets to R1. The last new command on the spoke router is the "ip nhrp nhs 172.16.0.1" statement. With this line, we are telling the spoke router to use the "next-hop server" to forward traffic across the NBMA network.
Substitute the appropriate values for the IP address and tunnel source on R3 and R4, and you should have working tunnels between R1 and each of the spoke routers. To verify this, use the "sho dmvpn" command:
R2#sho dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================
Interface: Tunnel0, IPv4 NHRP Details
IPv4 NHS: 172.16.0.1 RE
Type:Spoke, Total NBMA Peers (v4/v6): 1
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network
----- --------------- --------------- ----- -------- ----- -----------------
1 100.64.0.2 172.16.0.1 UP 01:21:34 S 172.16.0.1/32
R2#
As you can see from the snippet of output above, R2 has now established a tunnel connection to R1 (tunnel state is "Up" and next to "IPV4 NHS, we have the IP address of int tunnel0 on R1, followed by the flags "RE," verifying that the tunnel is responding and expecting replies). After duplicating the tunnel config on R3 and R4, you should see similar output on those routers, although each router will only show the connection to R1. This is a point-to-multipoint network, meaning that R2 cannot talk directly to R3 without going through R1 (sort of...actually R1 can broker connections between the spokes, but honestly, I’m not comfortable enough with the topic to go there yet). Assuming that you have copied the modified version of R2’s tunnel config to R3 and R4, you should have a completed point-to-multipoint VPN network now (w00t!). However, if you try to ping from the inside interface of R2, R3 or R4 to the inside interface of R1, you will most likely not be thrilled with the result:
R4#ping 192.168.1.1 source 192.168.4.1
<...snip...>
.....
Success rate is 0 percent (0/5)
R4#
Any ideas why? Of course! We haven’t set up any routing between the inside networks. When we configured BGP peering between the management network and the service provider network, we only advertised the outside interfaces of R1, R2, R3 and R4, since our service provider should not be aware of the inner workings of our network (unless we are using
MPLS). In order to actually send traffic across the VPN tunnels, we need to enable a routing protocol over the tunnels. Easy enough, right? It should look something like this...:
R1:
router ospf 42
router-id 10.254.254.1
passive-interface default
no passive-interface Tunnel0
network 10.254.254.1 0.0.0.0 area 0.0.0.0
network 172.16.0.0 0.0.0.255 area 0.0.0.0
network 192.168.1.0 0.0.0.255 area 0.0.0.0
Again, after making the appropriate substitutions for the router-id and the advertised networks, we’ll make the same changes on the spoke routers, and...what is going on here?
R1(config)#
*Dec 13 15:44:15.947: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.3 on Tunnel0 from LOADING to FULL, Loading Done
*Dec 13 15:44:16.391: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.3 on Tunnel0 from FULL to DOWN, Neighbor Down: Adjacency forced to reset
R1(config)#
*Dec 13 15:44:20.483: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.2 on Tunnel0 from INIT to DOWN, Neighbor Down: Adjacency forced to reset
*Dec 13 15:44:20.535: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.3 on Tunnel0 from EXSTART to DOWN, Neighbor Down: Adjacency forced to reset
*Dec 13 15:44:20.699: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.4 on Tunnel0 from LOADING to FULL, Loading Done
R1(config)#
*Dec 13 15:44:22.735: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.4 on Tunnel0 from FULL to DOWN, Neighbor Down: Adjacency forced to reset
R1(config)#
*Dec 13 15:44:25.307: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.3 on Tunnel0 from LOADING to FULL, Loading Done
*Dec 13 15:44:25.555: %OSPF-5-ADJCHG: Process 42, Nbr 10.254.254.3 on Tunnel0 from FULL to DOWN, Neighbor Down: Adjacency forced to reset
R1(config)#
Why is OSPF flapping?!?!
This is where I start to get in a little bit over my head. If I understand correctly, the issue is that OSPF is aware of the type of network configured across each OSPF-aware link. In this case, we have configured a NBMA network via multipoint GRE tunnels, but OSPF considers GRE tunnels to be a point-to-point network. While we had OSPF only running between R1 and R2, this was fine, but as soon as OSPF sees two neighbors across a single "point-to-point" link, it gets confused (understandably so!) and drops the neighbor relationship. To resolve this problem, at least under
certain circumstances, make the following change to R1, R2, R3 and R4:
int tunnel 0
ip ospf network point-to-multipoint
Now that OSPF understands that int Tunnel0 is actually part of a multipoint network, it will allow all three spoke routers to participate in OSPF across the tunnel. Run "sho ip route" and "sho ip ospf neighbor" to verify that everything is working as expected (it should be), and you should be good to go!
