Thursday, October 27, 2016

Cisco Advanced Routing Lab: IP Service Level Agreements

So you've taken all of the pieces that we've discussed so far -- Multicast, QoS, etc. -- and built a solid network for your customers. How do you prove to your customers that they are getting the quality that they are paying for? Arguably more important, how do you keep tabs on what's happening on your network so that you can fix any problems before your customer's begin complaining about them? As I'm sure you've guessed, there is a solution that is built in to most reasonably modern versions of Cisco IOS, and that solution is called "ip sla" (service level agreements). To mock up this lab, I had to upgrade my virtual routers slightly by adding IOS images for Cisco 7200 routers, as the IOS versions I had for the 3640 and 2651 routers I've been modeling don't support the ip sla feature.

Basically, the ip sla feature sends traffic across the network to measure packet loss, latency, jitter, etc. What's nice is that ip sla's allow you to craft the packets (or datagrams) that will be sent to mimic whatever kind of traffic you want to monitor. For example, you can set the TOS markings to monitor priority traffic across the network. You can tell the IP SLA engine to send data inside the G729 codec to measure how voice traffic is affected by the network. You can specify how often to send this traffic across the network, and how many packets to send at a time. It's kind of like the ping tool on steroids ;)

Suppose you have the following network...:


Because we have clients connected to R5, and a streaming server connected through R3, we want to make sure that jitter and latency is within acceptable limits between these two routers. We could build a Smokeping server (and in fact, Smokeping is running on the server labeled "CentOS6"), but Smokeping just sends standard ICMP packets. What if we want to measure latency of UDP traffic with specific QoS markings? Smokeping can't do that. Fortunately, this is exactly the kind of problem that the ip sla feature was designed to address, so we will enable ip sla on R3 and R5:
R3#sho run | begin ip sla
ip sla responder
ip sla 10015
udp-jitter 10.254.254.5 32766 source-ip 10.254.254.3 num-packets 20
tos 184
frequency 300
ip sla schedule 10015 life forever start-time now
ip sla 10025
udp-echo 10.254.254.5 32764 source-ip 10.254.254.3
ip sla schedule 10025 life forever start-time now

...and on R5:
R5#sho run | begin ip sla
ip sla responder
ip sla 10013
udp-jitter 10.254.254.3 32766 source-ip 10.254.254.5 num-packets 20
tos 184
frequency 300
ip sla schedule 10013 life forever start-time now
ip sla 10023
udp-echo 10.254.254.3 32764 source-ip 10.254.254.5
ip sla schedule 10023 life forever start-time now

The first line, "ip sla responder" tells the router to respond to IP SLA traffic. Next, we create an IP SLA process to send traffic to remote destination. The 5-digit identifier is simply a unique identifier on that router; in my case, the first three digits were pretty much arbitrary, the fourth digit I used to separate the UDP jitter test from the UDP echo test, and the last digit was taken from the router number (i.e., xxxx3 was a query for R3, xxxx5 was a query for R5, etc.). This is perhaps overkill with the examples I've shown, but on R1, for example, I set up similar IP SLA probes for R2-R5. Next, I specified what kind of test I wanted to run ("udp-jitter, udp-echo -- there are many, many more options to choose from), the destination IP address, a random port number to probe on the destination, the source IP address to use in the outbound datagram, and the optional "num-packets" flag to specify how many packets (grrr...not to be pedantic, but UDP -> datagram, Cisco! In technical fields, precise terminology often is important </rant>). In the UDP jitter test, the next line specifies that we want to apply a TOS marking so that this packet receives the same kind of priority handling that VoIP traffic (DSCP EF) would receive. After that, I specified a frequency of one test every 5 minutes for the UDP jitter test (I left the UDP echo test at the default settings). Finally, I specified the schedule for the IP SLA probes, start now and run indefinitely. However, if you wanted to monitor performance over a weekend or for a short period during backups at night, you could specify different values:
R3(config)#ip sla schedule 10023 life ?
  <0-2147483647>  Life seconds (default 3600)
  forever         continue running forever

R3(config)#ip sla schedule 10023 life 18600 ?
  ageout      How long to keep this Entry when inactive
  recurring   Probe to be scheduled automatically every day
  start-time  When to start this entry
  <cr>

R3(config)#ip sla schedule 10023 life 18600 recurring ?
  ageout      How long to keep this Entry when inactive
  start-time  When to start this entry
  <cr>

R3(config)#ip sla schedule 10023 life 18600 recurring start-time ?
  after     Start after a certain amount of time from now
  hh:mm     Start time (hh:mm)
  hh:mm:ss  Start time (hh:mm:ss)
  now       Start now
  pending   Start pending

R3(config)#ip sla schedule 10023 life 18600 recurring start-time 23:00 ?
  <1-31>  Day of the month
  MONTH   Month of the year
  ageout  How long to keep this Entry when inactive
  <cr>

R3(config)#$dule 10023 life 18600 recurring start-time 23:00 November 1 ?
  ageout  How long to keep this Entry when inactive
  <cr>

Cool. So...what happens when you enable IP SLA monitoring on your routers?

R3#sho ip sla statistics
IPSLAs Latest Operation Statistics

IPSLA operation id: 10015
Type of operation: udp-jitter
        Latest RTT: 17 milliseconds
Latest operation start time: *00:20:17.115 AKDT Wed Oct 26 2016
Latest operation return code: OK
RTT Values:
        Number Of RTT: 19               RTT Min/Avg/Max: 1/17/51 milliseconds
Latency one-way time:
        Number of Latency one-way Samples: 0
        Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds
        Destination to Source Latency one way Min/Avg/Max: 0/0/0 milliseconds
Jitter Time:
        Number of SD Jitter Samples: 17
        Number of DS Jitter Samples: 18
        Source to Destination Jitter Min/Avg/Max: 0/11/39 milliseconds
        Destination to Source Jitter Min/Avg/Max: 0/8/22 milliseconds
Packet Loss Values:
        Loss Source to Destination: 1           Loss Destination to Source: 0
        Out Of Sequence: 0      Tail Drop: 0
        Packet Late Arrival: 0  Packet Skipped: 0
Voice Score Values:
        Calculated Planning Impairment Factor (ICPIF): 0
        Mean Opinion Score (MOS): 0
Number of successes: 6
Number of failures: 0
Operation time to live: Forever



IPSLA operation id: 10025
Type of operation: udp-echo
        Latest RTT: 8 milliseconds
Latest operation start time: *00:24:23.535 AKDT Wed Oct 26 2016
Latest operation return code: OK
Number of successes: 58
Number of failures: 0
Operation time to live: Forever


R3#

That is at once a lot of information and not nearly as much information as you might hope to receive. While there is a lot of output showing quite a bit of detail, this is only the results of the last test. If you want any kind of historical perspective, you will need to put in some additional work. For example, I believe that you can have the results of the IP SLA monitoring logged to a syslog server (Splunk, perhaps?), although I haven't tried that. Alternatively, you can use SNMP to pull the results, although finding the right OIDs to query is a bit of a chore.

For now, let's focus on the CLI output. First, the UDP jitter test: the executive summary of the IP SLA monitoring is presented at the very end of the output:
Voice Score Values:
        Calculated Planning Impairment Factor (ICPIF): 0
        Mean Opinion Score (MOS): 0
Number of successes: 6
Number of failures: 0

A quick Google search provides a little insight into this output. Both ICPIF and MOS attempt to provide a metric for evaluating how suitable a network is for Voice over IP telephony, with lower scores being better than higher scores.
ICPIF values numerically less than 20 are generally considered "adequate."

MOS, on the other hand, is rated on a scale from 1 to 5, with 1 being very poor quality and 5 being very good quality. Note: In my output, the given MOS score was zero. Per Cisco, this means that the "...MOS data could not be generated for the given operation."

That should be enough to get you started with IP SLA monitoring on IOS. It's definitely a feature worth checking out if you will be using Cisco routers on any kind of production network.

IPv6 Intro: BGP, OSPF and IPv6...or Maybe Just EIGRP and IPv6, smh

Being a network engineer is kind of like being Sisyphus. Just when you think you you're starting to get to the top of your game, someone moves the target on you. In fact, a writer by the name of Spencer Johnson (M.D.) wrote a book on that subject not quite 20 years ago, and even though I've never read the book, I'd guess that it's at least as relevant today as it was then. Case in point, even though I've used IPv4, OSPF, and EIGRP professionally for years, I don't have a lot of professional experience with IPv6 or BGP. To address that problem, I set up the following network in GNS3 for playing with IPv4 and IPv6 with multiple routing protocols on emulated Cisco 7200 routers:

We have R1, R2 and R3 as routers within an autonomous system, R4 a random (IPv4-only) Internet router and R5 as another (IPv6-only, this time) random Internet router. My intent was to set up BGP peering between R1, R4 and R5, and to have R1, R2 and R3 share routes via OSPF. Sounds easy enough, right?

Hahaha...no.

In a previous lab, we set up OSPFv3 (OSPF for IPv6) on Cisco 3640 routers, so I used those instruction to (try to) set up OSPFv3 on the 7200 routers:
R1(config)#int gig0/0
R1(config-if)#ipv6 ospf 42 area 0.0.0.0
                   ^
% Invalid input detected at '^' marker.

R1(config-if)#ipv6 o?
% Unrecognized command
R1(config-if)#ipv6 ?
IPv6 interface subcommands:
  address             Configure IPv6 address on interface
  authentication      authentication subcommands
...
  multicast           multicast
  nat                 Enable IPv6 NAT on interface
  nd                  IPv6 interface Neighbor Discovery subcommands
  next-hop-self       Configures IP-EIGRP next-hop-self
  policy              Enable IPv6 policy routing
  redirects           Enable sending of ICMP Redirect messages
  rip                 Configure RIP routing protocol
...

Okay...is OSPFv3 not supported on this router? As it turns out, I think it actually is, but I'll save that for another lab.

Edit: no, it's not. I mean, it is, but it isn't. The "hooks" are there to configure OSPFv3 using "ipv6 ospf <process ID>" in global configuration, but you have to have an Advanced IP Services image to run it. The SP Services image I am running isn't licensed for it, because after all, what service provider would run OSPF on their network (answer: every one I've ever worked at), grrr...

For now, I decided to try to set up EIGRP for IPv6 since, 1) it *is* supported on the 7200, and 2) it did not seem to be supported on the 3640's:
R1(config)#int gig0/0
R1(config-if)#ipv6 router eigrp 10
R1(config-rtr)#int gig1/0
R1(config-if)#ipv6 router eigrp 10
R1(config-rtr)#exit
R1(config)#exit
R1#sho run int gig0/0
Building configuration...

Current configuration : 201 bytes
!
interface GigabitEthernet0/0
ip address 66.223.227.5 255.255.255.252
duplex full
speed 1000
media-type gbic
negotiation auto
ipv6 address 2001:C0:FFEE:2::1/126
ipv6 enable
ipv6 eigrp 10
end
R1#sho ipv6 route
IPv6 Routing Table - Default - 8 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
       M - MIPv6, R - RIP, I1 - ISIS L1, I2 - ISIS L2
       IA - ISIS interarea, IS - ISIS summary, D - EIGRP, EX - EIGRP external
C   2001:C0:FFEE::/126 [0/0]
     via GigabitEthernet1/0, directly connected
L   2001:C0:FFEE::1/128 [0/0]
     via GigabitEthernet1/0, receive
C   2001:C0:FFEE:2::/126 [0/0]
     via GigabitEthernet0/0, directly connected
L   2001:C0:FFEE:2::1/128 [0/0]
     via GigabitEthernet0/0, receive
LC  2001:C0:FFEE:254::1/128 [0/0]
     via Loopback0, receive
C   2016:FA:1::/64 [0/0]
     via FastEthernet6/0.20, directly connected
L   2016:FA:1::1/128 [0/0]
     via FastEthernet6/0.20, receive
L   FF00::/8 [0/0]
     via Null0, receive
R1#

Weird...why are none of my EIGRP routes showing up? I could ping across the interfaces and my IPv4 routing protocols were working as expected, but I could not get EIGRP in IPv6 to form neighbor adjacencies. What gives?

I started troubleshooting EIGRP using essentially the same toolkit I would use for IPv4...:
R1#sho ipv6 eigrp 10 neigh
IPv6-EIGRP neighbors for process 10
% EIGRP 10 is in SHUTDOWN
R1#sho ipv6 eigrp 10 int
IPv6-EIGRP interfaces for process 10
% EIGRP 10 is in SHUTDOWN
R1#

"EIGRP...is in SHUTDOWN?" I'm not familiar with that error message. WWGS ("What Would Google Say")? I quickly found a couple of tutorials on-line which showed that setting up EIGRP in IPv6 on a 7200 is a little different than setting OSPFv3 on a 3640 (go figure). Whereas OSPFv3 on a 3640 is entirely configured on the interface, EIGRP for IPv6 is a mix of interface-level commands and global config commands:
R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#ipv6 router eigrp 10
R1(config-rtr)#router-id 10.254.254.1
R1(config-rtr)#redistribute connected
R1(config-rtr)#passive-int default
R1(config-rtr)#no passive-int gig0/0
R1(config-rtr)#no passive-int gig1/0
R1(config-rtr)#no shut
02:35:27: %DUAL-5-NBRCHANGE: IPv6-EIGRP(0) 10: Neighbor FE80::C802:7FF:FE00:70 (GigabitEthernet1/0) is up: new adjacency
R1(config-rtr)#
02:35:38: %DUAL-5-NBRCHANGE: IPv6-EIGRP(0) 10: Neighbor FE80::C801:6FF:FEF0:70 (GigabitEthernet0/0) is up: new adjacency
R1(config-rtr)#exit
R1(config)#exit

After making eseentially the same changes on R2 and R3 (the interface names were different, but...), I saw my routes as expected:
R1#sho ipv6 route
IPv6 Routing Table - Default - 12 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
       M - MIPv6, R - RIP, I1 - ISIS L1, I2 - ISIS L2
       IA - ISIS interarea, IS - ISIS summary, D - EIGRP, EX - EIGRP external
C   2001:C0:FFEE::/126 [0/0]
     via GigabitEthernet1/0, directly connected
L   2001:C0:FFEE::1/128 [0/0]
     via GigabitEthernet1/0, receive
C   2001:C0:FFEE:2::/126 [0/0]
     via GigabitEthernet0/0, directly connected
L   2001:C0:FFEE:2::1/128 [0/0]
     via GigabitEthernet0/0, receive
D   2001:C0:FFEE:3::/64 [90/28416]
     via FE80::C802:7FF:FE00:70, GigabitEthernet1/0
LC  2001:C0:FFEE:254::1/128 [0/0]
     via Loopback0, receive
D   2001:C0:FFEE:254::2/128 [90/130816]
     via FE80::C801:6FF:FEF0:70, GigabitEthernet0/0
D   2001:C0:FFEE:254::3/128 [90/130816]
     via FE80::C802:7FF:FE00:70, GigabitEthernet1/0
D   2001:C0:FFEE:2222::/64 [90/28416]
     via FE80::C801:6FF:FEF0:70, GigabitEthernet0/0
C   2016:FA:1::/64 [0/0]
     via FastEthernet6/0.20, directly connected
L   2016:FA:1::1/128 [0/0]
     via FastEthernet6/0.20, receive
L   FF00::/8 [0/0]
     via Null0, receive
R1#

Well that was more cumbersome than it should have been, but <shrug>. At least we've got EIGRP working now. BGP via IPv4 is nothing new, so I won't waste a lot of time discussing the BGP configuration for R1-R4. However, the IPv6 configuration between R1 and R5 had me swearing at Cisco:
R1(config)#router bgp 65511
R1(config-router)# neighbor 2016:FA:1::5 remote-as 65515
R1(config-router)# address-family ipv6
R1(config-router-af)#  network 2001:C0:FFEE:254:0:0:0:1/128
R1(config-router-af)#  network 2001:C0:FFEE:254:0:0:0:2/128
R1(config-router-af)#  network 2001:C0:FFEE:254:0:0:0:3/128
R1(config-router-af)#  network 2001:C0:FFEE:2:0:0:0:0/126
R1(config-router-af)#  network 2001:C0:FFEE:0:0:0:0:0/126
R1(config-router-af)#  network 2001:C0:FFEE:2222:0:0:0:0/64
R1(config-router-af)#  network 2001:C0:FFEE:3:0:0:0:0/64
R1(config-router-af)#  neighbor 2016:FA:1::5 activate
% BGP context not been initialized properly.
R1(config-router-af)# exit
R1(config-router)#exit
R1(config)#exit
R1#sho bgp ipv6 unicast neighbors

R1#sho run | begin router bgp
router bgp 65511
bgp router-id 10.254.254.1
bgp log-neighbor-changes
neighbor 2016:FA:1::5 remote-as 65515
neighbor 209.193.4.4 remote-as 65514
!
address-family ipv4
  neighbor 209.193.4.4 activate
  no auto-summary
  no synchronization
  network 10.254.254.1 mask 255.255.255.255
  network 10.254.254.2 mask 255.255.255.255
  network 10.254.254.3 mask 255.255.255.255
  network 66.223.224.0 mask 255.255.255.224
  network 66.223.224.32 mask 255.255.255.224
  network 66.223.227.0 mask 255.255.255.252
  network 66.223.227.4 mask 255.255.255.252
  network 209.193.4.0
exit-address-family
!
ip forward-protocol nd
...

Wait, where's my "address-family ipv6" entries, and what's with that "BGP context has not been initialized properly" error message? I went back to the Great Oracle of Google, where I found this little tidbit of information:
Q. Error message: "% BGP context not been initialized properly." when Configuring neighbor under address-family IPv6

A. The issue is with the feature set. If the feature set is SP services, the following services are not supported.

  • IPv6 Routing: Multiprotocol BGP Extensions for IPv6
  • IPv6 Routing: Multiprotocol BGP Link-local Address Peering

To use these features,change the feature set to Advanced Enterprise Services.

Okay, let's check the code version on my routers:
R1#sho ver
Cisco IOS Software, 7200 Software (C7200-SPSERVICESK9-M), Version 12.4(24)T4, RELEASE SOFTWARE (fc2)

Well, <expletive deleted>! Since I don't have an Advanced IP Services image laying around, that pretty much kills the BGP portion of this lab for now.

I went ahead and removed the BGP portion and played with EIGRP across the network, but I'm slightly miffed by the fact that I couldn't do any testing with BGP or OSPF under IPv6, since IPv6 is now a part of certification testing. With adoption of IPv6 "in the wild" still lagging, it would be nice to be able to mock such networks up in a lab without spending a fortune in hardware and software licensing.

Sunday, October 23, 2016

Lesson 16: VRRP

In the last lesson that I wrote while working on my CCNA certification, I introduced the concept of router redundancy via a Cisco proprietary protocol known as HSRP, or "Hot Standby Router Protocol." However, HSRP is not the only way to create a redundant data connection for your office. In this lab, we'll look at a second, similar protocol known as VRRP, or "Virtual Router Redundancy Protocol."

Disclaimer:
The configuration document I used to play with VRRP in this lab didn't work exactly as advertised on the routers I was emulating. In fairness, Cisco 3640 routers are decidedly, ummmm, "old-school" (read that: obsolete), so it's entirely possible that the syntax has changed on more modern platforms that are running more recent versions of IOS. However, what I present here should be close enough to get you started. Here (pdf) is the link to the Cisco document with the slightly different syntax.

As usual, we'll start with the network diagram:


We'll set up lo0 and fa1/0 on R1 and R2 as normal, R4 exists only to act as a DHCP server, and R3 serves as a destination network provider. We'll establish OSPF between R1, R2 and R3, using network statements for 100.64.1.0/30 and 100.64.2.0/30 and using "redistribute connected subnets." On our client, "Knoppix Clone 1," we'll set the default gateway to 100.64.0.1/29. So far, nothing unexpected, right?

Just to recap, the problem we want to solve is, what happens when fa0/0 goes down on our default gateway? If R2 did not exist in this network, then R1 is our single point of failure. If we lose R1, then the clients on our LAN can no longer reach the servers hanging off of R3. To address this, we set up two routers in parallel so that we have a redundant path to R3. However, there is no way to tell a client PC (or router or...) to use multiple default gateways. HSRP and VRRP were designed to address this problem. In both scenarios, you configure a single default gateway on your client network, then use either HSRP or VRRP to shuffle that default gateway address between multiple routers. To set it up, you...:

  1. Enter configuration mode;
  2. Switch to the interface facing your client LAN;
  3. Add an IP address within the subnet of your client LAN;
  4. Configure a meaningful description of the VRRP group;
  5. Configure the client's default gateway address in the VRRP group;
  6. Set the VRRP priority for the router (a higher value takes priority over a lower value);
  7. Set the VRRP advertisement and preempt delay timers.


Here's how the configuration looks on R1...:
interface FastEthernet0/0
ip address 100.64.0.2 255.255.255.248
vrrp 10 description VRRP Group
vrrp 10 ip 100.64.0.1
vrrp 10 preempt delay minimum 3
vrrp 10 priority 254
end

...and on R2:
interface FastEthernet0/0
ip address 100.64.0.3 255.255.255.248
vrrp 10 description VRRP Group
vrrp 10 ip 100.64.0.1
vrrp 10 preempt delay minimum 3
vrrp 10 priority 128
end


NOTE:
I also added the following line to the config...:
R1(config-if)#vrrp 10 timers advertise 1

...to set VRRP to send an "advertisement" every second. However, this is the default behaviour for VRRP, and therefore, it didn't show up in the config until I changed it for testing. Anyway, does it work?

Let's traceroute from R4 to R3:
R4#traceroute 10.254.254.3

Type escape sequence to abort.
Tracing the route to 10.254.254.3

  1 100.64.0.2 4 msec 4 msec 4 msec
  2 100.64.1.2 12 msec 8 msec 8 msec
R4#

Now, if I shut down fa0/0 on R1, I should see a short interruption in service, followed by R2 picking up the traffic:
R4#ping 10.254.254.3 repeat 100

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.254.254.3, timeout is 2 seconds:
!!!!!!!!!!!!!..!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 97 percent (97/100), round-trip min/avg/max = 4/11/36 ms
R4#traceroute 10.254.254.3

Type escape sequence to abort.
Tracing the route to 10.254.254.3

1 100.64.0.3 8 msec 4 msec 8 msec
2 100.64.2.2 8 msec 12 msec 8 msec
R4#

Notice how the first hop originally was 100.64.0.2, but now it's 100.64.0.3? VRRP has failed over the virtual 100.64.0.1 IP address from R1 to R2, which is reflected in the traceroute output.

With a little effort, we can see what's happening at the Ethernet level, too, and it's even more interesting. We'll start by verifying our configuration. R1 is currently the VRRP master (i.e., it's hosting the IP address 100.64.0.1), and R2 is the backup:
R1#sho vrrp brief
Interface          Grp Pri Time  Own Pre State   Master addr     Group addr
Fa0/0              10  254 3007       Y  Master  100.64.0.2      100.64.0.1    
R1#
----------------------------------------------------------------------------

R2#sho vrrp brief
Interface          Grp Pri Time  Own Pre State   Master addr     Group addr
Fa0/0              10  128 3500       Y  Backup  100.64.0.2      100.64.0.1    
R2#

First, we'll clear the arp table on Knoppix Client 1:

Now, we'll ping 10.254.254.3 (lo0 on R3) from Knoppix Clone 1:

Next, we shut down fa0/0 on R1, and verify that R2 is now the VRRP master:
R1#sho vrrp brief
Interface          Grp Pri Time  Own Pre State   Master addr     Group addr
Fa0/0              10  254 3007       Y  Init    0.0.0.0         100.64.0.1    
R1#

----------------------------------------------------------------------------

R2#sho vrrp brief
Interface          Grp Pri Time  Own Pre State   Master addr     Group addr
Fa0/0              10  128 3500       Y  Master  100.64.0.3      100.64.0.1    
R2#

...and ping R3 again:

Hmmm...since the VRRP virtual MAC address moves with the router, that doesn't give us much insight into what was actually happening here. Fortunately, I was running tcpdump to capture the Ethernet frames while running this test. After exporting the PCAP file to Wireshark, we can get a little better understanding of what happened here.

Note:
To keep the Wireshark screen captures relevant, I filtered out some of the chatter. We configured VRRP to send advertisements every second, for example, so I filtered out the VRRP protocol data. These routers were also running CDP, so I filtered that as well.

At the very beginning of the capture, we can see Knoppix Client 1 ("CadmusCo_d3:7c:8f") send an arp request for 100.64.0.1, and we can see R1 (cc:00:2a:af:00:00) send an arp reply, stating that the VRRP virtual MAC address "00:00:5e:00:01:0a" is associated with 100.64.0.1:

Then, we ping R3 through R1. As you can see, we sent the ICMP request to the VRRP virtual MAC...:

...but received the reply from the MAC of fa0/0 on R1 (cc:00:2a:af:00:00):

At this point, we shut down fa0/0 on R1, and allowed R2 to take ownership of 100.64.0.1. Since VRRP also transports the virtual MAC, our next ping will still be sent to 00:00:5e:00:01:0a...:

...but this time, our reply has come from the MAC address of fa0/0 on R2 (cc:01:2a:af:00:00):

Because the MAC address doesn't change, we don't have to wait until the arp cache on connected devices times out for traffic to use the new path. This can be a serious problem, in some cases. For example, if you are using a Cisco ASA to connect to a (non-VRRP) "highly-available" system, the default arp cache timeout period is FOUR HOURS, which means it can take up to four hours for your "highly-available" (cough) system to recover from a failover! This isn't just an academic, theoretical point, either. I am currently working a trouble ticket in my day job where this is exactly what's happening. Unfortunately, just shortening the arp cache timeout period can drive up CPU load and memory requirements on busy devices, so there is a balancing act to be found between automatic fail-over times and system resource utilization. VRRP neatly solves that problem by sidestepping the whole issue.

Wednesday, October 5, 2016

IPv6 Intro: Routing with OSPFv3 for IPv6

Continuing on with our last lab, let's remove the RIPng config, and set up OSPFv3, instead. Here's the network diagram again, just so you don't have to jump back to the RIPng lab again:



Just like configuring RIPng, we will start by enabling IPv6 OSPF in the interface configurations. Once again, only R1 is shown, but the other routers are configured in exactly the same way:
R1(config)#int fa0/0
R1(config-if)#ipv6 ospf 42 area 0.0.0.0
R1(config-if)#int s0/0
R1(config-if)#ipv6 ospf 42 area 0.0.0.1
R1(config-if)#exit
R1(config)#do sho ipv6 route
IPv6 Routing Table - 8 entries
Codes: C - Connected, L - Local, S - Static, R - RIP, B - BGP
       U - Per-user Static route
       I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea
       O - OSPF intra, OI - OSPF inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
LC  2001:C0:FFEE::1/128 [0/0]
     via ::, Loopback0
C   2001:C0:FFEE::1:0/125 [0/0]
     via ::, FastEthernet0/0
L   2001:C0:FFEE::1:1/128 [0/0]
     via ::, FastEthernet0/0
C   2001:C0:FFEE::1111:0/126 [0/0]
     via ::, Serial0/0
L   2001:C0:FFEE::1111:1/128 [0/0]
     via ::, Serial0/0
OI  2001:C0:FFEE::2222:0/126 [110/65]
     via FE80::CA03:3CFF:FEAE:0, FastEthernet0/0
L   FE80::/10 [0/0]
     via ::, Null0
L   FF00::/8 [0/0]
     via ::, Null0
R1(config)#

If this looks very familiar from the RIPng lab, it should. Any guesses on how to configure redistribution of connected networks?
R1(config)#ipv6 router ospf 42
R1(config-rtr)#router-id 10.254.254.1
R1(config-rtr)#redist conn
R1(config-rtr)#exit
R1(config)#

At this point, you might be scratching your head, wondering WHY I used an IPv4 address for the router ID. Actually...I didn't. The RFC for OSPF specifies that the router ID is a 32-bit, unique (at least within the AS) identifier. Since IPv4 addresses are ALSO 32-bit, unique identifiers, convention (and, unless a different router ID is specified in the config, Cisco's default behaviour) has set the loopback interface's IPv4 address as the router ID. I also chose to set an IPv4 address on the loopback interface, and I used it as the router ID in this lab. However, technically the OSPFv3 router ID is any random 32-bit number (see the footnote at the end of this lab for more).

After setting up redistribution, the routing table on R1 looks a little more populated:
R1(config)#do sho ipv6 route
IPv6 Routing Table - 12 entries
Codes: C - Connected, L - Local, S - Static, R - RIP, B - BGP
       U - Per-user Static route
       I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea
       O - OSPF intra, OI - OSPF inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
LC  2001:C0:FFEE::1/128 [0/0]
     via ::, Loopback0
OE2  2001:C0:FFEE::2/128 [110/20]
     via FE80::CA03:3CFF:FEAE:0, FastEthernet0/0
OE2  2001:C0:FFEE::3/128 [110/20]
     via FE80::CA04:3CFF:FEAE:0, FastEthernet0/0
OE2  2001:C0:FFEE::4/128 [110/20]
     via FE80::CA00:3CFF:FEAE:0, Serial0/0
OE2  2001:C0:FFEE::5/128 [110/20]
     via FE80::CA03:3CFF:FEAE:0, FastEthernet0/0
C   2001:C0:FFEE::1:0/125 [0/0]
     via ::, FastEthernet0/0
L   2001:C0:FFEE::1:1/128 [0/0]
     via ::, FastEthernet0/0
C   2001:C0:FFEE::1111:0/126 [0/0]
     via ::, Serial0/0
L   2001:C0:FFEE::1111:1/128 [0/0]
     via ::, Serial0/0
OI  2001:C0:FFEE::2222:0/126 [110/65]
     via FE80::CA03:3CFF:FEAE:0, FastEthernet0/0
L   FE80::/10 [0/0]
     via ::, Null0
L   FF00::/8 [0/0]
     via ::, Null0
R1(config)#

There you go: we have a basic OSPFv3 (IPv6) configuration, sharing routes between each of the five routers in our network. We even set up three separate areas: the backbone, consisting of R1, R2 and R3; Area 1, consisting of R1 and R4; and Area 2, consisting of R2 and R5.

Footnote:
At this point, you may be wondering -- well, I am, anyway -- what OSPFv3 uses for a router ID if there are no IPv4 interfaces in use on the router, and there is no router ID specified in the OSPFv3 config. Let's find out, shall we?
R1(config)#int lo0
R1(config-if)#no ip addr
R1(config-if)#shut
R1(config-if)#no shut
R1(config-if)#no ipv6 router ospf 42
R1(config)#int fa0/0
R1(config-if)#ipv6 ospf 42 area 0.0.0.0
R1(config-if)#int s0/0
R1(config-if)#ipv6 ospf 42 area 0.0.0.1
R1(config-if)#exit
R1(config-if)#ipv6 router ospf 42
R1(config-rtr)# redistribute connected
R1(config-rtr)#exit
R1(config)#exit
R1#
00:36:49: %OSPFv3-4-NORTRID: OSPFv3 process 42 could not pick a router-id,
please configure manually
R1#sho ipv6 ospf neigh
%OSPFv3: Router process 42 is not running, please configure a router-id
R1#

Well, that's interesting! What happens if you don't have an IPv4 address on the loopback interface, but you have an IPv4 address on another interface?
R1(config-if)#int vlan 10
R1(config-if)#ip addr 10.254.254.1 255.255.255.252
R1(config-if)#no shut
R1(config-if)#no ipv6 router ospf 42
R1(config)#ipv6 router ospf 42
R1(config-rtr)#redist conn
R1(config-rtr)#exit
R1(config)#int fa0/0
R1(config-if)#ipv6 ospf 42 area 0.0.0.0
R1(config-if)#int s0/0
R1(config-if)#ipv6 ospf 42 area 0.0.0.1
R1(config-if)#exit
R1(config)#exit
R1#sho ipv6 ospf 42 neigh

Neighbor ID    Pri  State      Dead Time  Interface ID  Interface
10.254.254.2     1  FULL/BDR   00:00:39   4             FastEthernet0/0
10.254.254.3     1  FULL/DR    00:00:36   4             FastEthernet0/0
10.254.254.4     1  FULL/  -   00:00:38   5             Serial0/0
R1#

There you have it. If there is an IPv4 address configured anywhere on the router, OSPFv3 will use it for the router ID. If not, then you MUST configure a router ID in the OSPFv3 configuration.

IPv6 Intro: Routing with RIP for IPv6 (RIPng)

In our last lab, we set up a small network with IPv6 and static routes. Let's up the ante a little bit this time, and set up slightly larger network using IPv6 and RIPng.

True Confession: I wanted to start with EIGRP for IPv6, but it isn't supported on either of the routers I have available in GNS3, grrr... However, RIPng is, so we'll go with RIPng for now. I now return you to your regularly scheduled blog post...

Here's the network I am using for this lab:


We start by enabling unicast routing on the routers, and assigning the IPv6 addresses to all of the interfaces (R1 shown, others similar):
R1(config)#ipv6 unicast-routing
R1(config)#int lo0
R1(config-if)#ip addr 10.254.254.1 255.255.255.255
R1(config-if)#ipv6 enable
R1(config-if)#ipv6 address 2001:00C0:FFEE:0000:0000:0000:0000:1/128
R1(config-if)#no shut
R1(config-if)#int fa0/0
R1(config-if)#ipv6 enable
R1(config-if)#ipv6 address 2001:00C0:FFEE:0000:0000:0000:0001:0001/125
R1(config-if)#no shut
R1(config-if)#int s/0/0
R1(config-if)#ipv6 enable
R1(config-if)#ipv6 address 2001:00C0:FFEE:0000:0000:0000:1111:0001/126
R1(config-if)#no shut

Once you have assigned IPv6 addresses to the respective interfaces on all of the routers, you should be able to ping across each subnet, but (for example) R5 shouldn't be able to ping R4:
R5(config)#do ping 2001:C0:FFEE::2222:1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:C0:FFEE::2222:1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/9/16 ms
R5(config)#do ping 2001:C0:FFEE::1111:2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:C0:FFEE::1111:2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R5(config)#

This is not at all unexpected. In IPv4, you wouldn't be able to ping from R5 to R4 either, unless you had static routes in place or a routing protocol to create the routes between routers that are not directly connected. Let's turn up RIPng on all five routers to share routes across this network (again, R1 shown, but the other routers are similar):
R1(config)#int fa0/0
R1(config-if)#ipv6 rip 10 enable
R1(config-if)#int s0/0
R1(config-if)#ipv6 rip 10 enable
R1(config-if)#exit

That's it! No "network" statements -- just edit the interface configuration to tell the router to start the RIPng process on that interface, and the router will begin sending advertisements for connected networks through that interface. Let's look at the routing table on R1:
R1(config)#do sho ipv6 route
IPv6 Routing Table - 8 entries
Codes: C - Connected, L - Local, S - Static, R - RIP, B - BGP
       U - Per-user Static route
       I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea
       O - OSPF intra, OI - OSPF inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
LC  2001:C0:FFEE::1/128 [0/0]
     via ::, Loopback0
C   2001:C0:FFEE::1:0/125 [0/0]
     via ::, FastEthernet0/0
L   2001:C0:FFEE::1:1/128 [0/0]
     via ::, FastEthernet0/0
C   2001:C0:FFEE::1111:0/126 [0/0]
     via ::, Serial0/0
L   2001:C0:FFEE::1111:1/128 [0/0]
     via ::, Serial0/0
R   2001:C0:FFEE::2222:0/126 [120/2]
     via FE80::CA01:33FF:FE30:0, FastEthernet0/0
L   FE80::/10 [0/0]
     via ::, Null0
L   FF00::/8 [0/0]
     via ::, Null0
R1(config)#

Hmmm...notice that the loopback interfaces for R2 through R5 aren't showing up? Let's edit R1 and R2 to see if enabling RIPng on the loopback interfaces resolves that problem:
R1(config)#int lo0
R1(config-if)#ipv6 rip 10 enable
R1(config-if)#exit
...
R2(config)#int lo0
R2(config-if)#ipv6 rip 10 enable
R2(config-if)#exit

Does R1 see the loopback address on R2 now?
R1(config)#do sho ipv6 route | inc 2001:C0:FFEE::2/128
R 2001:C0:FFEE::2/128 [120/2]
R1(config)#

Yep, that works...but is there a better way to enable RIPng on an interface that isn't actually connected to another host? Let's try something on R3:
R3(config)#ipv6 router rip 10
R3(config-rtr)#redist connected

Does R1 see R3's loopback now?
R1(config)#do sho ipv6 route | inc 2001:C0:FFEE::3/128
R 2001:C0:FFEE::3/128 [120/2]
R1(config)#

Success! On the downside, it's a little hokey that you enable RIPng on the interface, but have to enable redistribution of connected routes in a global configuration, but I suppose that's better than having to enable redistribution on each interface <shrug>

One other thing to look at: let's look at one other thing in the routing table on R1:
R1(config)#do sho ipv6 route
IPv6 Routing Table - 8 entries
<...snip...>
L   FF00::/8 [0/0]
     via ::, Null0
R1(config)#

Once again, there is an odd-looking IPv6 address showing up in the routing table. Let's check our interfaces for anything similar:
R1(config)#do sho ipv6 int brie | inc FF00
R1(config)#

No...nothing there. What gives?

As it turns out, FF02::9 is a multicast address reserved by IANA for RIPng route updates, as defined in RFC 2080. And with that interesting (?) little tidbit of information, we will wrap up this lab on IPv6 routing with RIPng. Next time, we'll dive into OSPFv3.

Tuesday, October 4, 2016

Cisco Routing Oddities -- Interaction Between EIGRP, Distribute-List, and Static Routes

I ran into something at work today that I didn't quite understand, and which I finally had to mock up in GNS-3 to figure out. While trying to track down a routing problem, I found a distribute-list statement on an EIGRP configuration that referenced an access list which denied anything from 10.0.0.0/8. That's not terribly unusual, except that it looked like there were still a lot of 10-dot subnets being propagated by EIGRP on this router. Digging into the configuration a bit more, I saw that there was a "redistribute static" command inside the EIGRP configuration, and I found several static routes to 10-dot subnets. To understand the interaction between EIGRP, static routes, and the "distribute-list" statement, I created the following, simple network in GNS-3:



In this network, R1, R3, R4 and R5 will all use EIGRP to share routes with each other. R2 will have a default route pointing to R1, and R1 will have a single static route to R2's loopback address:
R1(config)#ip route 10.254.254.0 255.255.255.0 10.0.0.2
R1(config)#router eigrp 10
R1(config-router)#redistribute static metric 1000000 1 255 1 1500

After turning up EIGRP, R1 shared all of its known routes with R3, R4 and R5, exactly as you would expect:
R2#ping 100.64.254.1
Sending 5, 100-byte ICMP Echos to 100.64.254.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/6/8 ms
R2#

-----------------------------------------------------------------------

R3#ping 10.0.0.2
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/17/44 ms
R3#

-----------------------------------------------------------------------

R4#ping 10.0.0.2
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/29/72 ms
R4#

-----------------------------------------------------------------------

R5#ping 10.0.0.2
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/27/44 ms
R5#

Then, I added the following configuration statements to R1:
R1(config)#ip access-list standard FILTER-NULL-ROUTES
R1(config-std-nacl)#deny 10.0.0.0
R1(config-std-nacl)#permit any
R1(config-std-nacl)#!
R1(config-std-nacl)#router eigrp 10
R1(config-router)#distribute-list FILTER-NULL-ROUTES out
R1(config-router)#!

Running the exact same ping tests...:
R2#ping 100.64.254.1
Sending 5, 100-byte ICMP Echos to 100.64.254.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/12/40 ms
R2#

-----------------------------------------------------------------------

R3#ping 10.0.0.2
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R3#

-----------------------------------------------------------------------

R4#ping 10.0.0.2
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R4#

-----------------------------------------------------------------------

R5#ping 10.0.0.2
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R5#

This is to be expected -- 10.0.0.0/30 is a control to verify that the distribute-list is working as expected (that is, that it is filtering 10.0.0.0/8 from the EIGRP updates). Let's see if the distribute-list overrides the static route, or if the static route overrides the distribute-list:
R3#ping 10.254.254.9
Sending 5, 100-byte ICMP Echos to 10.254.254.9, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/23/40 ms
R3#

-----------------------------------------------------------------------

R4#ping 10.254.254.9
Sending 5, 100-byte ICMP Echos to 10.254.254.9, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/35/72 ms
R4#

-----------------------------------------------------------------------

R5#ping 10.254.254.9
Sending 5, 100-byte ICMP Echos to 10.254.254.9, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/28/48 ms
R5#

Notice how we can still reach 10.254.254.9, even though we can no longer reach 10.0.0.2? The distribute-list is filtering 10.0.0.0/8 addresses from the EIGRP routing updates sent by R1. However, we have explicitly stated that we want static routes to be redistributed by EIGRP, and we have a static route for 10.254.254.0/24 via 10.0.0.2. Consequently, R1 is still advertising a route for 10.254.254.9, despite the distribute-list filter. This explains how my router at work was still sharing 10-dot routes with other routers in its EIGRP area, even though there was a distribute-list statement filtering 10-dot subnets.