Several years ago, I worked for a telecommunications provider that was rolling out what I believe was one of the very first -- if the
THE very first --
MPLS network in the state of Alaska. The network admins often joked that their network was not just cutting edge; it was
bleeding edge. At that time, the network admins worked very closely with Cisco to build and troubleshoot the network, since Cisco was still making sure the tech worked as advertised. Since then, technology has changed, and MPLS has become much more common, even though one of the major drivers that led to the development of MPLS has become more or less a moot point: hardware has improved to the point, where it is no longer significantly faster to switch or route traffic based upon labels than IP or MAC addresses.
For example, the company I currently work for has implemented a ring (!) network with an MPLS core, simply because using MPLS makes it easier for us to configure customer networks across our service provider network. Rather than visiting each node in the ring to create a VLAN for that customer, then adding the VLAN to a topology group (again, on every node in the network), we simply create a
VLL or
VPLS on the endpoints, and we're done. It's a configuration change on two nodes in the case of a VLL, rather than every node in the ring (with VPLS, it depends upon how many endpoints the client requires, so in some cases, there isn't much to be gained by using MPLS rather than a plain VLAN, and a VPLS introduces some overhead into the network that a VLAN doesn't).
In this lab, we'll create a "service-provider" network in GNS3 consisting of three routers, R1, R2 and R3, and two customer networks, R4 and R5 belonging to one customer, and R6 and R7 belonging to a second customer. The network topology looks like this:
Disclaimer: This lab was created using Cisco 2600-series routers with IOS version 12.3. This is a really, really old version of code, on really, really old (virtual) hardware, since that's all I have access to, currently. Consequently, Cisco's implementation of MPLS may be somewhat different on modern hardware and firmware versions. Hopefully, this will be close enough to get you started, however.
Since both customer networks are Layer-3 (routed) networks, we will use
VRF's to logically separate the two customer networks from each other. In this sense, a VRF can more or less be considered to be the Layer-3 equivalent of a VLAN: the VRF's will partition the service provider network into to separate logical networks, including routing tables, that keep addressing and routing from the customer networks from interfering with each other. This has two major implications: first, the two customers in our lab don't have to coordinate IP addressing with each other (in fact, in our example, R4 and R7 have the exact same IP address and subnet!); second, the
service provider will participate in the customer's routing!
Let's get started. First, we'll set up the service provider network, including the routing process for the network core. On R1:
interface Loopback0
ip address 10.10.10.1 255.255.255.255
!
interface Serial0/0
ip address 192.168.12.1 255.255.255.0
!
interface Serial0/1
ip address 192.168.13.1 255.255.255.0
!
router ospf 1138
router-id 10.10.10.1
log-adjacency-changes
network 10.10.10.1 0.0.0.0 area 0.0.0.0
network 192.168.12.0 0.0.0.255 area 0.0.0.0
network 192.168.13.0 0.0.0.255 area 0.0.0.0
!
R2 and R3 have similar configurations, and none of this is new, so I won't repeat their configs here.
Next, on a Cisco router, we have to enable
CEF (Cisco Express Forwarding). It's trivial to do:
R1(config)#ip cef
Next, we create the two VRFs. Technically, you don't need MPLS to create a VRF (Cisco calls this "VRF Lite"), nor do you need VRFs to run MPLS (for instance, if you are using VLLs or VPLS'). However, VRFs and MPLS work very well together, and for this lab, they solve the problem we have quite handily:
ip vrf left
rd 100:1
route-target export 100:1
route-target import 100:1
!
ip vrf right
rd 200:1
route-target export 200:1
route-target import 200:1
!
As you can see in the configuration above, I was highly original with the name of the VRFs, using "left" for routers R4 and R5, and "right" for R6 and R7. When implementing your own MPLS networks, I'd recommend keeping the VRF names short, easy to type, and easy to remember, as you'll be using them quite a bit in the router configurations. After creating the two VRFs, we use the "rd <
AS:nn" command to create the "route distinguisher," using AS 100 for VRF left and AS 200 for VRF right. This is the label that you will be using in various places to differentiate between the two customer routing tables that you will be using in this network. According to Cisco's context-sensitive help function, you can also use an IP address rather than an AS number for the route distinguisher, but why type more than you have to? AS numbers are shorter! ;)
After creating your two VRFs, you next enable MPLS on the network interfaces that will be part of your MPLS core. In this case, that is S0/0 and S0/1 on R1, R2 and R3:
interface Serial0/0
mpls ip
Just for giggles, exit out of configuration mode (or stay in config mode and prepend the following command with the word "do") and run "sho run":
interface Serial0/0
ip address 192.168.12.1 255.255.255.0
tag-switching ip
!
interface Serial0/1
ip address 192.168.13.1 255.255.255.0
tag-switching ip
!
Wait a minute...we didn't type "tag-switching ip" in the Serial interface configuration! We used "mpls ip!" What gives? Well, way back when MPLS was still being developed, Cisco was doing essentially the same thing, but called it "tag switching." When the standard was formalized, the technology was given the name MPLS, and Cisco implemented both the tag-switching and mpls commands on the command line. As I understand, there are some slight differences between the tag-switching and mpls command -- and I'm not sure what, exactly those difference might be, so I won't elaborate on that rather vague statement, except to say that you should probably stick with the mpls command, if that's what you are actually trying to build -- but the "show run" output rewrites either command as "tag-switching."
At this point, you know have a working MPLS core, as can be proven by running the "show mpls ldp neighbor" command:
R1#sho mpls ldp neighbor
Peer TDP Ident: 10.10.10.3:0; Local TDP Ident 10.10.10.1:0
TCP connection: 10.10.10.3.11008 - 10.10.10.1.711
State: Oper; PIEs sent/rcvd: 165/165; Downstream
Up time: 02:19:39
TDP discovery sources:
Serial0/1, Src IP addr: 192.168.13.3
Addresses bound to peer TDP Ident:
192.168.13.3 10.10.10.3 192.168.23.3
Peer TDP Ident: 10.10.10.2:0; Local TDP Ident 10.10.10.1:0
TCP connection: 10.10.10.2.11004 - 10.10.10.1.711
State: Oper; PIEs sent/rcvd: 167/166; Downstream
Up time: 02:19:38
TDP discovery sources:
Serial0/0, Src IP addr: 192.168.12.2
Addresses bound to peer TDP Ident:
192.168.12.2 10.10.10.2 192.168.23.2
R1#
Now, we need to tell MPLS which interfaces will be accepting traffic that should be encapsulated inside the MPLS label. This is done with the "ip vrf forwarding..." command. Let's also configure the IP addresses on the customer-facing Ethernet interfaces as we do this:
R1(config)#int eth0/0
R1(config-if)#ip vrf forwaring left
R1(config-if)#ip address 172.16.0.1 255.255.255.252
R1(config-if)#no shut
The only difference between a typical interface configuration and what we did here is that we used the "ip vrf forwarding left" command to tell the MPLS process that this interface is an endpoint of the VRF "left" tunnel(s). Do the same thing for Eth0/1 on R1, and for both E0/0 and E0/1 on R2 and R3, as well.
Be careful here! If you apply the IP address before you run the "ip vrf forwarding..." command, IOS will display an error message telling you that the IP address
has been removed from the interface configuration! If you don't see the warning, you can spend quite a while trying to figure out why you aren't seeing the routes you expect across your MPLS core (
DAMHIKT!)
This is where things start to get fun. You've basically got a complete MPLS configuration now -- you've enabled MPLS on the various network interfaces, you've got MPLS distributing labels for the various routes, and you can see the MPLS peering is working as it should. However, there is one last piece of the puzzle left: now, you need a way to distribute routes across your MPLS core. First, apply the IP addresses and routing processes to your "customer" routers just as you normally would if R4/R5 and R6/R7 were directly connected to each other. For example, on R4:
interface Loopback0
ip address 10.10.10.4 255.255.255.255
!
interface Ethernet0/0
ip address 172.16.0.2 255.255.255.252
half-duplex
!
router ospf 42
router-id 10.10.10.4
log-adjacency-changes
redistribute connected subnets
network 172.16.0.0 0.0.0.3 area 0.0.0.0
!
If you're paying attention, this is the point where you might scratch your head and object, "Wait a minute...with the 'network 172.16.0.0 0.0.0.3 area 0.0.0.0' command, I'm advertising my OSPF routing process to the service provider!" And you'd be exactly right. With MPLS, the service provider participates in your routing process, but because your traffic is being encapsulated in MPLS, that routing process is only participating with your other endpoints. In this example, R4 and R5 will share routes through the service provider's network, but will not see routes from R6 and R7 (and vice versa).
To accomplish this task on the service provider's network, the service provider will need to turn up separate OSPF process
for each customer. In the lab, I also created separate loopback interfaces for each OSPF process because OSPF will complain if you try to enable OSPF without including any local interfaces in the process -- for example, on R2, if I try to turn up an OSPF process for VRF "right" (
Edit: I'm not sure it's necessary to turn up OSPF for VRF "right" on R2 or VRF "left" on R3, as we will be setting up BGP to share routes between the OSPF processes; I'll investigate this in a later post). However, when turning up a separate OSPF process, you must tell OSPF which VRF this routing process belongs to:
interface Loopback1
description loopback for OSPF 100 -- VRF LEFT
ip vrf forwarding left
ip address 10.0.100.1 255.255.255.255
!
interface Loopback2
description loopback for OSPF 200 -- VRF RIGHT
ip vrf forwarding right
ip address 10.0.200.1 255.255.255.255
!
router ospf 100 vrf left
router-id 10.0.100.1
network 172.16.0.0 0.0.0.3 area 0.0.0.0
!
router ospf 200 vrf right
router-id 10.0.200.1
network 172.16.0.0 0.0.0.3 area 0.0.0.0
!
Notice that I included the "ip vrf forwaring..." command on the loopback interfaces!
If you execute the "sho route" command on any of the customer routers now, however, you will see that they are only aware of the locally connected routes and the loopback interface of the locally connected service provider router (i.e., R4 will only see the IP address of loopback1 on R1, R7 will only see the IP address of loopback2 on R1, etc.). This is because the OSPF processes that we just turned up are only sharing routes with the customer router connected to their Ethernet interface (OSPF 100 on R1 is only sharing routes through eth0/0 to R4, etc.). To enable routing across the MPLS core, we need to enable another routing process (!) to redistribute routes between these routing processes. If that sounds like a perfect job for BGP, you are exactly right:
router bgp 65510
no synchronization
bgp log-neighbor-changes
neighbor 10.10.10.2 remote-as 65510
neighbor 10.10.10.2 update-source Loopback0
neighbor 10.10.10.3 remote-as 65510
neighbor 10.10.10.3 update-source Loopback0
no auto-summary
!
address-family vpnv4
neighbor 10.10.10.2 activate
neighbor 10.10.10.2 send-community extended
neighbor 10.10.10.3 activate
neighbor 10.10.10.3 send-community extended
exit-address-family
!
address-family ipv4 vrf right
redistribute ospf 200
no auto-summary
no synchronization
exit-address-family
!
address-family ipv4 vrf left
redistribute ospf 100
no auto-summary
no synchronization
exit-address-family
!
Notice the "address-family ipv4 vrf left" and "address-family ipv4 vrf right" statements. Under these statements, we have "redistribute ospf <process ID>" commands. These commands tell BGP to share routes learned via these OSPF process to the appropriate VRFs. That is, BGP should share routes learned via OSPF process 100 with VRF "left" and OSPF process 200 with VRF "right." This is good, but not sufficient. BGP will now know about any routes that OSPF knows, but OSPF doesn't yet know about all of the routes that BGP has in its routing table (you can verify this by running the "show ip route vrf left" and "show ip route vrf right" commands on any of the service provider routers). Let's take care of that now:
R1(config)#router ospf 100
R1(config-router)#redistribute bgp 65510 subnets
R1(config-router)#router ospf 200
R1(config-router)#redistribute bgp 65510 subnets
With this final configuration command, you should now have routes across the MPLS core between the customer routers. R4 should be able to ping/telnet/etc. to R5 (and vice versa) and R6 should be able to ping/telnet/etc. to R7 (and vice versa). However, neither R4 nor R5 should be able to communicate with R6 or R7.
Troubleshooting:
Enabling MPLS introduces a couple of complications to troubleshooting. As I mentioned earlier, I was fortunate enough to be an admin at a telco that was building out an MPLS core back in my early days of system and network administration. On one occasion, I saw an alarm for a customer's network, and logged into one of the MPLS edge routers to try to troubleshoot. However, when I tried to ping the customer's router, I was dismayed to find that my router apparently didn't have a route to that host. You can see this for yourself by attempting to ping a customer router from R1, R2 or R3:
R1#ping 172.16.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.2.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R1#
If you try "show route," you'll see a similar issue:
R1#sho ip route
<...snip...>
C 192.168.12.0/24 is directly connected, Serial0/0
C 192.168.13.0/24 is directly connected, Serial0/1
10.0.0.0/32 is subnetted, 3 subnets
O 10.10.10.2 [110/65] via 192.168.12.2, 01:09:36, Serial0/0
O 10.10.10.3 [110/65] via 192.168.13.3, 01:09:36, Serial0/1
C 10.10.10.1 is directly connected, Loopback0
O 192.168.23.0/24 [110/128] via 192.168.13.3, 01:09:36, Serial0/1
[110/128] via 192.168.12.2, 01:09:36, Serial0/0
R1#
If you'll notice, all of the customer routers were configured with 172.16.x.y IP addresses, but there are no 172.16.x.y IP addresses in the routing table on R1. Yet, if you try a ping from R4 to R5 (or R6 to R7), you will see that it works:
R4#ping 172.16.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.2.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/19/52 ms
R4#
How is this possible?
If you'll recall, I commented earlier that MPLS is rather like the Layer-3 equivalent of a VLAN, in that MPLS encapsulates the routing tables of the various VRFs into their own logical partitions, and that these routing tables are not shared amongst other VRFs. Nor are they shared with the global routing table of the service provider routers. If you want to view the routing tables inside a VRF or ping across the VRF, you have to explicitly tell the router so:
R1#ping vrf left 172.16.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.2.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/36/140 ms
R1#sho ip route vrf left
Routing Table: left
<...snip...&rt;
Gateway of last resort is not set
172.16.0.0/30 is subnetted, 2 subnets
C 172.16.0.0 is directly connected, Ethernet0/0
B 172.16.2.0 [200/0] via 10.10.10.2, 03:01:06
10.0.0.0/32 is subnetted, 2 subnets
O E2 10.10.10.4 [110/20] via 172.16.0.2, 01:17:46, Ethernet0/0
C 10.0.100.1 is directly connected, Loopback1
R1#
As a fellow network admin and blogger so eloquently put it, "Pretty much any command that you would normally use, add 'vrf <vrf name>' to it." This tells the router that you are specifically asking about the routing tables inside the specified VRF, which makes sense, if you think about it. In this lab, both R4 and R7 have the same IP address. From the service provider router's standpoint, how would it know whether it should ping R4 or R7 if all you did was "ping 172.16.0.2?" Consequently, you have to explicitly tell the router that it is to ping 172.16.0.2 inside VRF "left" (for example).
For more on the subject, check out the following web pages that I found to be helpful while trying to learn how to configure MPLS on Cisco routers:
Packet Life: Creating MPLS VPNs
Cisco Dreamer: MPLS Basics
Router Jockey: Cisco MPLS VRF Configuration and Demo
