Advanced Cisco Routing -- BGP MED (Multi-Exit Discriminator)

Suppose we have two connections to our upstream ISP: a high-speed link from Cust-A to ISP-1, and a low-speed link from Cust-A to ISP-2 (Cust-B is just a random Internet host):


Here are the subnets in use on this network:

Subnet	Endpoint A	Endpoint A Interface	Endpoint B	Endpoint B Interface
10.254.254.1	ISP-1	Lo0	N/A	N/A
10.254.254.2	ISP-2	Lo0	N/A	N/A
100.64.1.254	Cust-A	Lo0	N/A	N/A
100.64.2.254	Cust-A	Lo0	N/A	N/A
10.0.0.0/30	ISP-1	Gig-E 1/0	Cust-A	Gig-E 1/0
10.0.0.4/30	ISP-1	Gig-E 2/0	Cust-B	Gig-E 1/0
10.0.0.8/30	ISP-1	Gig-E 3/0	ISP-2	Gig-E 3/0
10.0.0.12/30	ISP-2	Gig-E 1/0	Cust-B	Gig-E 2/0
100.64.1.0/26	Cust-A	Fast-E 0/0	Knoppix-32	Eth 1/0
100.64.2.0/26	Cust-B	Fast-E 0/0	CentOS7_1	Eth 1/0

Obviously, we would typically want traffic to flow across the high-speed link rather than the low-speed link. However, BGP doesn't consider bandwidth when determining the "best" path from one host to another:


As you can see, BGP has selected a route via the low-speed circuit from the host Knoppix-32 PC to the CentOS7_1 web server in Cust_B's network. To solve this problem, it's easy enough to set a weight on the outbound link to force traffic to use the circuit connected to ISP-1. All we have to do is set a sufficiently high metric on the route we want to take:

Cust-A
router bgp 65512
neighbor 10.0.0.1 weight 30

Since higher weights take priority over lower weights, this will force outbound traffic to use ISP-1 rather than ISP-2. However, that only has an effect on our outbound traffic. BGP may still provide a route from Cust-B back to us through ISP-2 (the low-bandwidth circuit). This potentially causes two problems: first, we'd rather have our traffic go through the faster circuit (for obvious reasons); and second, this can cause "asymmetric routing." Some applications and network devices (stateful firewalls, for example) really don't like asymmetric routing. Unfortunately, trying to troubleshoot a problem caused by asymmetric routing can be a real PITA, and no, not the tasty kind :( To force other networks to prefer the path via ISP-1, we will adjust BGP's "MED" ("Multi Exit Discriminator"), one of the metrics that BGP uses to calculate the "best" route between endpoints. First, on our router, we'll create an access list to identify our internal networks:

Cust-A(config)#ip access-list standard BGP_Internal_Nets
Cust-A(config-std-nacl)#permit 100.64.1.0 0.0.0.63
Cust-A(config-std-nacl)#permit host 100.64.1.254

Next, we create a route map:

Cust-A(config)#route-map BGP_MED 10
Cust-A(config-route-map)#match ip addr BGP_Internal_Nets
Cust-A(config-route-map)#set metric 110

Finally, we apply the route map to the LESS-PREFERRED neighbor (ISP-2) in our BGP configuration:

Cust-A(config)#router bgp 65512
Cust-A(config-router)#neighbor 172.16.0.1 route-map BGP_MED out
Cust-A(config-router)#exit
Cust-A(config)#exit
Cust-A#clear ip bgp 65511

Unlike weight, a lower MED is preferable to a higher MED, and therefore, by advertising a higher-than-default MED to ISP-2's BGP process, we are effectively telling it to prefer an alternate route to our network.

After BGP re-converges, we should see that both ISP-1 and ISP-2 are using the higher-bandwidth link via ISP-1 to reach 100.64.1.x:

ISP-1#sho ip bgp | inc 65512
*  10.0.0.0/30      10.0.0.2                 0             0 65512 i
*> 100.64.1.0/26    10.0.0.2                 0             0 65512 i
*> 100.64.1.254/32  10.0.0.2                 0             0 65512 i
*  172.16.0.0/30    10.0.0.2                 0             0 65512 i
ISP-1#

...and...:

ISP-2(config)#do sho ip bgp | inc 65512
*>i100.64.1.0/26    10.0.0.2                 0    100      0 65512 i
*                   172.16.0.2             110             0 65512 i
*>i100.64.1.254/32  10.0.0.2                 0    100      0 65512 i
*                   172.16.0.2             110             0 65512 i
ISP-2(config)#

Perfect! Both routers are now advertising a preferred route via ISP-1, just as we wanted (">" indicates a preferred route). You can verify this by a traceroute from CentOS7_1:


By setting the MED in our BGP config, we have redundant links to our ISP, but will still prefer the high-bandwidth circuit unless there is a problem. I'll leave testing fail-over as an exercise for the reader ;)

Advanced Cisco Routing: OSPF Metrics

Suppose there are multiple routes between hosts on a network? How does OSPF choose a preferred route? I'm a big fan of practical experience, so to answer this question, I fired up GNS3 and started building networks to find out:

There are two endpoints, R5 and R6 in this network. There is a core network consisting of R1, R2, R3 and R4 which provide the transport between these two endpoints. Looking at this network diagram, you can see that there are two routes that traffic between these endpoints might take: first, there is the route R1-R2-R3-R4; second, there is the much shorter route of R1-R4. You can make a logical argument for either of these routes being better. From R1, there are two paths to R4, one of which is across a FastEthernet interface, and the other of which is across a 10M Ethernet interface. From R1's perspective, route R1-R2-R3-R4, going through the FastEthernet interface, might appear to be better, since the FastEthernet interface has ten times the bandwidth of the Ethernet interface connecting R1 and R4. However, there are more hops on that route, and the Serial link between R2 and R3 is much slower than the Ethernet interface between R1 and R4. Consequently, a human with the complete picture of the network can see that R5-R1-R4-R6 is clearly the better path. Does OSPF see that, also?

R5#traceroute 192.168.2.2

Type escape sequence to abort.
Tracing the route to 192.168.2.2

1 192.168.1.1 4 msec 4 msec 4 msec
2 172.16.1.2 12 msec 8 msec 8 msec
3 192.168.2.2 28 msec 24 msec *
R5#

Yep. Like a human with a network diagram, OSPF forms a complete picture of the network topology and makes intelligent decisions about the best route between endpoints.

However, suppose the topology were different. What if R1-R4 were connected by a serial link and R1-R2-R3-R4 were all FastEthernet links, like so?:

In this case, there is only one hop on the R1-R4 path, whereas there are four hops on the R1-R2-R3-R4 path. However, the bandwidth across the FastEthernet links is much better than the bandwidth across the serial link. A human would most likely choose R1-R2-R3-R4 to pass the traffic. What does OSPF choose as the best path?
R5#traceroute 192.168.2.2

Type escape sequence to abort.
Tracing the route to 192.168.2.2

1 192.168.1.1 4 msec 8 msec 0 msec
2 172.16.2.2 16 msec 12 msec 12 msec
3 172.16.3.2 24 msec 24 msec 24 msec
4 172.16.4.2 36 msec 24 msec 28 msec
5 192.168.2.2 48 msec 40 msec *
R5#

Once again, OSPF uses its comprehensive view of the network to choose the best path between endpoints. However, if you need to force OSPF to choose a particular path, it is possible to do so by manually assigning a cost to a link, but that's a topic for another post ;)