Lesson 15 -- HSRP

You finally get caught up on all of your work and schedule a well-deserved week of PTO. The big day comes, and you relish in the chance to sleep in a little bit. You finally get up, then load up your motorcycle with gear (hey, it's my story, and with a week off, I'd go someplace cool on my motorcycle, lol) and race out of town to go camping in the mountains. You have just picked out a nice camping spot in the woods near a calming, babbling stream -- very Zen, you decide -- when you suddenly hear the most dreadful, blood-curdling, adrenaline-inducing noise known to man. The horrible wail sends chills up your spine, and you cringe with fear, wondering what is about to happen. You take a deep breath, summon all of your courage and...

...answer your cell phone.

It's the manager of the branch office. "I'm so sorry to bother you on the first day of your vacation, but we've just lost the network to the main office."

"No problem," you lie, sighing. You walk her through a few initial troubleshooting steps -- pings, traceroutes and such -- and confirm that the branch office router is off-line. You ask her to go to the wire closet and look for a power light on the front of the router. It takes a few tries to help her find the correct device, but soon enough, she has located it, and as you suspected, it is powered off. She cycles the power switch to no avail, so you have her check the circuit breakers in the wall. Sure enough, the breaker has tripped, so you have her reset it, and she tells you that she now sees "blinky lights" on the front of the router. By the time she has returned to her desk, the router has booted and she verifies that she can now reach the main office network again.

"Thanks again for the help," she tells you, as she hangs up, while you silently vow to camp tomorrow night in a remote valley farther back in the mountains where there is no cell service.

The rest of the week passes with no more interruptions, and you return to the office the following Monday morning with a plan to buy a second router and configure HSRP so that a failure of a router will cause the network to route all traffic through a standby router, reducing the number of after-hours calls you receive and providing better uptime to your users.

But mostly to reduce the number of after-hours calls you receive... :)

When the new router arrives, you copy the config from the existing router, then change the hostname on the new router and change the IP address of the serial line back to the main office. Once you've verified that the serial line to the main office is working, you configure HSRP on both routers. On the old router, you delete the IP address on the Fa0/0 port, then add these lines:

lab3640rtra(config)#interface FastEthernet0/0
lab3640rtra(config-if)#ip address 192.168.2.2 255.255.255.0
lab3640rtra(config-if)#standby 42 ip 192.168.2.1
lab3640rtra(config-if)#standby 42 priority 64
lab3640rtra(config-if)#standby 42 name Branch_Office_HSRP_Group


...and on the new router:

lab3640rtrb(config)#interface FastEthernet0/0
lab3640rtrb(config-if)#ip address 192.168.2.3 255.255.255.0
lab3640rtrb(config-if)#standby 42 ip 192.168.2.1
lab3640rtrb(config-if)#standby 42 priority 128
lab3640rtrb(config-if)#standby 42 name Branch_Office_HSRP_Group


A couple of things are important in this configuration. First, the old router (3640a) used to have the IP address 192.168.2.1./24 assigned to fa0/0, but I have now assigned that IP address to the standby group ("standby 42 ip 192.168.2.1"). Next, both the old router and the new router have new IP addresses in the same subnet as the IP address for the standby group (192.168.2.2, 192.168.2.3 and 192.168.2.1, respectively). The two routers will be negotiating responsibility for forwarding LAN traffic to the WAN port over their new IP addresses, and will be listening for traffic from the LAN on 192.168.2.1 (which the LAN hosts will use as the default gateway address). If the active router goes off-line for some reason, the standby router will notice that it is no longer receiving keep-alive messages, and will assume the role of the default gateway. Finally, notice that all of the lines that activate HSRP begin, "standby 42..." The number 42 is the standby group number, and can be anything that you desire (well, within the bounds of 0 and 255, anyway). I picked 42 because it is "a completely ordinary number, a number not just divisible by two but also six and seven. In fact it's the sort of number that you could without any fear introduce to your parents." Or something like that, anyway, lol.

<...shoots the rabbit, gets off of the bunny trail and gets back on topic...>

After setting up HSRP, you tap out a few more commands on the keyboard to make sure the routers really are providing fail-over protection:

lab3640artr#sho standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp Prio P State    Active addr     Standby addr    Group addr     
Fa0/0       42  64     Standby  192.168.2.3     local           192.168.2.1    
lab3640artr#


...and...:

lab3640brtr#sho standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp Prio P State    Active          Standby         Virtual IP     
Fa0/0       42  128    Active   local           192.168.2.2     192.168.2.1    
lab3640brtr#


Yep, looks good! You hook up a laptop to an unused switchport, then start pinging the main office router. Since 3640b is the active and 3640a is the standby, you decide to pull the Ethernet cable to 3640b to verify that the failover occurs as expected, and to see how many pings you drop during the switch:

64 bytes from 192.168.2.1: icmp_req=8 ttl=255 time=1.35ms
64 bytes from 192.168.2.1: icmp_req=9 ttl=255 time=1.38ms
64 bytes from 192.168.2.1: icmp_req=18 ttl=255 time=1.17ms
64 bytes from 192.168.2.1: icmp_req=19 ttl=255 time=1.14ms


Hmmm...about 9 seconds to fail over. Users will probably notice the drop, if they are paying attention, but the network will be back up before they can dial your cell phone. You check the HSRP status on the two routers again:

lab3640artr#sho standby brief                      P indicates configured to preempt.                      | Interface   Grp Prio P State    Active addr     Standby addr    Group addr      Fa0/0       42  64     Active   local           unknown         192.168.2.1     lab3640artr#

...and...:

lab3640brtr#sho standby brief                      P indicates configured to preempt.                      | Interface   Grp Prio P State    Active          Standby         Virtual IP      Fa0/0       42  128    Init     unknown         unknown         192.168.2.1     lab3640brtr#

Looks good. The output shows that the routers have switched roles, due to the disconnected network cable on the 3640b. Unless the WAN link on the active router is flapping, that should provide some redundancy between the main office and branch office networks for the users in the branch office. You pat yourself on the back and call it a day.

Lesson 14 -- Advanced WAN Configurations, Part 1

In Lesson 5 you set up a very simple WAN between your main office 2651 router and the remote site's 3640 router. The configuration for this link was pretty minimalist:

interface Serial0/1
description T1 to lab3640rtr
bandwidth 1544
ip address 192.168.3.5 255.255.255.252


This configuration used HDLC -- the default on Cisco routers -- to establish the WAN link to the 3640 router, set the bandwidth (used by routing protocols that consider bandwidth in their metrics) to 1.544Mbps, and establishes a /30 (two host) subnet between the routers, one IP address for each endpoint. However, you decide that you would like to implement PPP rather than HDLC, since PPP can be configured to use PAP or CHAP to authenticate the routers on each side of the link, and because if you manage to scrounge a couple of new T1 cards (and the funds to buy a second leased line) you can use Multilink PPP to give you a 3Mbps channel between sites. You telnet to the remote router, issue a "reload in 30" command to return the 3640 to its original configuration if you botch the changes (it's saved my backside more than once), then get started on configuring PPP:

lab3640rtr#conf t
lab3640rtr(config)#int serial 0/1
lab3640rtr(config-if)#encapsulation ppp


Your telnet session locks up, but you expected that. Now, configure the local side:

lab2651rtr#conf t
lab2651rtr(config-if)#encapsulation ppp
lab2651rtr(config-if)#exit
lab2651rtr(config)#exit
lab2651rtr#sho int ser0/1
Serial0/1 is up, line protocol is up


You return to your telnet session to the 3640 router, and hit the "Enter" key a couple of times to make sure it's responding again, and it is. Woohoo! You've enabled PPP successfully. You type "reload cancel" and "write mem" on both routers to save the current configurations, then set up PPP authentication using CHAP. To do this, you'll need to configure a user name on each router that matches the host name of the other router and assign the same password to both user names. That is, on the lab3640rtr router, you will create a user "lab2651rtr" and on the lab2651rtr, you will create a user "lab3640rtr":

lab2651rtr#conf t
lab2651rtr(config)#username lab3640rtr password myChapPw


...and on the remote router:

lab3640rtr#conf t
lab3640rtr(config)#username lab2651rtr password myChapPw


Next, in the Serial 0/1 interface configurations, you will tell the routers to authenticate using CHAP (substitute the word "pap" for "chap" in the config if you would rather use PAP):

lab3640rtr(config)#int ser0/1
lab3640rtr(config-if)#ppp authentication chap


...and on the local router:

lab2651rtr(config)#int serial 0/1
lab2651rtr(config-if)#ppp authentication chap


...and that's it!

Note 1: No joke about using the "reload in 30" before making config changes to the management interfaces on a remote router. It can be the difference between being the hero of the day and the idiot who had the misfortune to execute a career-limiting move. True story from the IT trenches: I work in Anchorage, Alaska, but manage network devices in Bethel, Alaska, 500 miles to the west (and there are no roads to Bethel from Anchorage -- it's a one hour flight via Alaska Airlines if you have to go in person). One day, I was working on my LAN router/firewall in Bethel, and fortunately, I heard that little voice inside my head telling me to run the "/sbin/shutdown -r +15 &" command (it's a Linux-based router, but that's the equivalent of Cisco's "reload in 15" command). I did, even though I was sure what I was going to do wouldn't lock me out of the router. I started making changes, and when I restarted the network interfaces, I lost my SSH connection to the router. I tried to create a new SSH session, but no joy. I tried to ping the router...nothing. I even ran "nmap -sP" just to make sure it was really off-line, but I already knew it would be (and it was). About two minutes later, my boss hollers at me from his office, "I think we just lost the Bethel router!" I calmly replied, "Yep, but it should be back in...<...checks watch...>...maybe ten minutes." I explained what happened, and he just said, "Let me know when it's back on-line." It's never good to take a LAN/WAN router off-line in the middle of the business day without arranging a scheduled outage, but, well, (*cough*) excrement occurs sometimes. However, had I not had the foresight ("luck") to tell the router reboot itself in fifteen minutes before I started working on it, I would have been in a much less comfortable position when my boss asked about the router being off-line.

Note 2: Notice how I made changes to the remote router before making changes to the local router every time? That's not by coincidence :) I lost my connection to the remote router when I removed HDLC encapsulation and added PPP encapsulation. If I had made the changes to lab2651rtr first, I would not have been able to reach lab3640rtr to make the changes that would restore the connection. It's obvious in hindsight, but maybe not so obvious the first time you are making changes in the middle of the night so as to not take users off-line during the business day. Yep, been there and done that, too :)

Note 3: Extra-credit: want to make sure that PPP really is doing what you think it's doing? Delete the "username lab3640rtr password myChapPw" from the config, then shut/no shut the interface, and run "sho ip int brief". You should see the hardware up and the line protocol down on the serial interface, indicating that there is a misconfiguration on the layer-2 protocols you are using on the serial interface.