Wednesday, October 23, 2013

JNCIA Lesson 1: Basic LAN Configuration

Congratulations! Because of your experience with networking in general, and Juniper equipment in particular, you have just landed a network administrator job with a new, small company. You will be responsible for building the network infrastructure for the new company. Your first decision in this role was to purchase two brand new Juniper EX3200 Layer-3 switches to use for both LAN switching and to route traffic to and from your Internet Service Provider.

First, of course, is racking the equipment and connecting the associated cables -- power, management console, etc. That is a breeze, and in less than half an hour, your new Juniper switches are happily humming away. You connect a serial cable from your laptop to the console port (NOT the "Management" port, however -- that's an Ethernet port for management access!), and are greeted by the Juniper's shell prompt:

root@:RE:0%

You type a few Unix commands:

root@:RE:0% ls
.cshrc  .history  .login  .profile
root@:RE:0% uname -a
JUNOS ex3200 11.4R7.5 JUNOS 11.4R7.5 #0: 2013-03-01 10:14:08 UTC  builder@evenath.juniper.net:/volume/build/junos/11.4/release/11.4R7.5/obj-powerpc/bsd/kernels/JUNIPER-EX/kernel powerpc
root@:RE:0%
Notice that it's running a BSD kernel (FreeBSD, to be exact). You want to begin configuring the switch, so you execute the "cli" command to access JunOS:

root@:RE:0% cli
root@ex3200>


The switch comes with a very basic configuration...:

root@ex3200> show configuration
version 11.4R7.5;
system {
    host-name ex3200;
    syslog {
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
}
interfaces {
    ge-0/0/0 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/1 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/2 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/3 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/4 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/5 {
        unit 0 {
            family ethernet-switching;
        }
    ge-0/0/6 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/7 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/8 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/9 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/10 {
        unit 0 {
            family ethernet-switching; 
        }
    }
    ge-0/0/11 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/12 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/13 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/14 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/15 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/16 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/17 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/18 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/19 {
        unit 0 {
            family ethernet-switching;
    }
    ge-0/0/20 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/21 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/22 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/23 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/1/0 {
        unit 0 {
            family ethernet-switching;
        }
    }
    xe-0/1/0 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/1/1 {
        unit 0 {
            family ethernet-switching;
        }
    }
    xe-0/1/1 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/1/2 {
        unit 0 {
            family ethernet-switching;
        }
    }
    xe-0/1/2 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/1/3 {
        unit 0 {
            family ethernet-switching;
        }
    }
}
protocols {
    igmp-snooping {
        vlan all;
    }
    rstp;
    lldp {
        interface all;
    }
    lldp-med {
        interface all;
    }
}
ethernet-switching-options {
    storm-control {
        interface all;
}
vlans {
    default {
        l3-interface vlan.0;
    }
}
poe {
    interface all;
}

root@ex3200>


With all of the indentations, curly braces and semi-colons, the configuration greatly resembles a Unix/Linux configuration file, rather than a typical network vendor's config file, you think to yourself. At first, it makes the config look a little long-winded, but it's pretty easy to read, once you get used to it.

However, the basic, default configuration is missing some very important elements that you will need to add. First, you decide to give the switch a more meaningful name. Since this will be the switch/router for the main office campus, you decide to change the name to main3200. You enter configuration mode and start to make the necessary changes:

root@ex3200> configure
Entering configuration mode

[edit]
root@ex3200# set system host-name main3200

[edit]
root@ex3200#


Notice that that didn't seem to change anything. Juniper doesn't actually make the changes live until you commit them, using -- of all things -- the "commit" command.:

root@ex3200# commit
commit complete

[edit]
root@main3200#


JunOS also includes a nifty feature to allow you to undo a change. When committing a configuration change, JunOS copies the existing configuration to an archive before writing the new config to storage. Consequently, if you make a change that breaks something, you can always roll back to an earlier configuration:

root@main3200# set system host-name oops

[edit]
root@main3200# commit
commit complete

[edit]
root@oops# rollback 1
load complete

[edit]
root@oops# commit
commit complete

[edit]
root@main3200#


There are a number of other cool features available when committing changes: you can tentatively commit the changes...:

root@main3200# set host-name mo3200

[edit system]
root@main3200# commit confirmed
commit confirmed will be automatically rolled back in 10 minutes unless confirmed
commit complete

# commit confirmed will be rolled back in 10 minutes
[edit system]
root@mo3200#
Broadcast Message from root@main3200
(no tty) at 8:37 AKDT...

Commit was not confirmed; automatic rollback complete.


Cool...this could be handy when configuring remote systems, where a botched config could mean downtime while you drive across town [Edit: or fly across the country, which would be the case in my real day job]. Use "commit confirmed", and if the new config doesn't work, it will automatically roll back to the previous version after the time-out period, and you can try again.

To confirm the commit, just type "commit" again:

...
# commit confirmed will be rolled back in 10 minutes
[edit]
root@main3200# commit
commit complete

[edit]
root@main3200#


You can also simply view the changes to the active configuration...:

root@main3200# show | compare rollback 1
[edit system]
- host-name mo3200;
+ host-name main3200;

[edit]
root@main3200#
...or check the config changes without actually making them active...:

root@main3200# commit check
configuration check succeeds

[edit]
root@main3200#


Another point about JunOS that's worth noting: if you look at the "system" portion of the default config...:

root@ex3200> show configuration
version 11.4R7.5;
system {
    host-name ex3200;
    syslog {
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
}


Notice the hierarchy for the host-name setting:

system | host-name

When we changed the hostname, we did so with the "set system host-name main3200" command. However, sometimes, you will be making a number of changes under a hierarchy, and it can be a bit of a chore repeating all of the hierarchy for every change you want to make. Consequently, you can tell JunOS that you want to work within a particular portion of the hierarchy, and to assume that all of the commands you are running are to be taken in the context of that hierarchy. For example, these two commands are identical to the Juniper:

root@ex3200# set system host-name main3200

[edit]
root@ex3200# commit
commit complete

[edit]
root@main3200#


...and...:

root@ex3200# edit system

[edit system]
root@ex3200# set host-name main3200

[edit system]
root@ex3200# commit
commit complete

[edit system]
root@main3200#


Think of it as being similar to the difference between...:

me@myllt:~$ cp /var/www/localhost/htdocs/gallery/images/thumbnails/some.jpg /var/www/localhost/htdocs/gallery/images/thumbnails/some-other.jpg


...and...:

me@myllt~$ cd /var/www/localhost/htdocs/gallery/images/thumbnails/
me@myllt:/var/www/localhost/htdocs/gallery/images/thumbnails$ cp some.jpg some-other.jpg


It's the exact same thing: you are setting the position within the hierarchy in which you want to work. In the trivial examples above, it might not seem like the "edit..." syntax saves much effort, but suppose you want to make a number of changes to an Ethernet port -- say, hard-code speed and duplex and set a description:

root@main3200# edit interfaces ge-0/0/4

[edit interfaces ge-0/0/4]
root@main3200# set description "Legacy Billing Server"

[edit interfaces ge-0/0/4]
root@main3200# set ether-options speed 10m

[edit interfaces ge-0/0/4]
root@main3200# set ether-options link-mode half-duplex

[edit interfaces ge-0/0/4]
root@main3200# set ether-options auto-negotiation no-flow-control

[edit interfaces ge-0/0/4]
root@main3200# commit check
configuration check succeeds

[edit interfaces ge-0/0/4]
root@main3200# commit
commit complete

[edit interfaces ge-0/0/4]
root@main3200#


If you hadn't used the "edit interface ge-0/0/4" command, you would have had to type this instead:

root@main3200# set interfaces ge-0/0/4 description "Legacy Billing Server"

[edit]
root@main3200# set interfaces ge-0/0/4 ether-options speed 10m

[edit]
root@main3200# set interfaces ge-0/0/4 ether-options link-mode half-duplex

[edit]
root@main3200# set interfaces ge-0/0/4 ether-options auto-negotiation no-flow-control

[edit]
root@main3200# commit check
configuration check succeeds

[edit]
root@main3200# commit
commit complete

[edit]
root@main3200#


Let's configure some other basic settings on the switch:

root@main3200# set system domain-name example.com

[edit]
root@main3200# set system root-authentication plain-text-password
New password:
Retype new password:

[edit]
root@main3200# set system services ssh protocol-version v2

[edit]
root@main3200#


With those commands, we have set the domain name, set a root password and enabled SSH access. However, you will probably have a hard time using SSH to connect to the switch, since the switch does not have a management IP address set. Let's address that issue now:

root@main3200# set interfaces vlan unit 0 family inet address 192.168.1.1/24

[edit]
root@main3200# commit
commit confirmed

[edit]
root@main3200#


With that, you should have a minimal working configuration on your switch. Each of the 24 Ethernet ports is enabled and will allow hosts to communicate with each other, and you can SSH to the switch for management access at 192.168.1.1.

That's a good start on a basic, real-world configuration for a LAN switch, but there are a couple of other things to do, too. First, in a one-person shop, it might be fine to simply use the root account for all configuration. However, if there are (or will be) multiple admins managing network devices, then it might be wise to configure separate logins for each user. Let's create three user accounts on the EX-3200 -- an admin account for you, an admin account for your assistant George and a read-only account for your Network Operations Center (NOC):

root@main3200# set system login class noc permissions interface

[edit]
root@main3200# set system login class noc permissions view

[edit]
root@main3200# set system login user nocuser class noc authentication plain-text-password
New password:
Retype new password:

[edit]
root@main3200# set system login user mike class super-user authentication plain-text-password
New password:
Retype new password:

[edit]
root@main3200# set system login user george class super-user authentication plain-text-password
New password:
Retype new password:

[edit]
root@main3200# commit
commit complete

[edit]
root@main3200#


You can also configure TACACS or RADIUS authentication:

root@main3200# set system authentication-order [radius password]

[edit]
root@main3200# set system radius-server 192.168.1.17 secret myradiussecretpw

[edit]
root@main3200# commit
commit confirmed

[edit]
root@main3200#


Next, you want the switch to forward logs to a syslog server, located at 192.168.1.20:

root@main3200# set system syslog host 192.168.1.20 any any

[edit]
root@main3200#


If you wanted, you could also send firewall logs to one server, or send warning (or higher) messages to a third server:

root@main3200# set system syslog host 192.168.1.21 firewall any

[edit]
root@main3200# set system syslog host 192.168.1.20 any warning


[edit]
root@main3200# commit
commit confirmed

[edit]
root@main3200#


Next, we will configure the switch to set its internal clock with a local NTP server...:

root@console9# set system ntp server 192.168.1.23

[edit]
root@main3200# commit
commit confirmed

[edit]
root@main3200#


...and we will configure SNMP on the switch:

root@main3200# set system location building "123 A Street" country-code US postal-code 12345 floor 2 rack 36

[edit]
root@main3200# set snmp community my-ro-community-string authorization read-only

[edit]
root@main3200# set snmp community my-rw-community-string authorization read-write

[edit]
root@main3200# root@console9# set snmp contact miswork@uui-alaska.com

[edit]
root@main3200# set snmp location Anchorage

[edit]
root@main3200# commit
commit complete

[edit]
root@main3200#


That leaves one last, very important step before we are done: we need to create a "rescue configuration." The rescue configuration is a "known good" configuration that provides basic connectivity to the switch, if the current configuration is deleted or lost. To be honest, I'm not sure I see the point of the rescue configuration, since JunOS keeps an archive of past configs (perhaps, just in case you have moved/re-addressed the switch?), but...we'll go ahead and create one from our current configuration. First, notice that there is no rescue configuration yet:

root@main3200> file list /config/

/config/:
.snap/
db /
juniper.conf.1.gz
juniper.conf.2.gz
juniper.conf.3.gz
juniper.conf.gz
ssh_host_dsa_key
ssh_host_dsa_key.pub
ssh_host_ecdsa_key
ssh_host_ecdsa_key.pub
ssh_host_key
ssh_host_key.pub
ssh_host_rsa_key
ssh_host_rsa_key.pub
usage.db
vchassis/

[edit]
root@main3200>


We'll save the current configuration as the rescue configuration, and show the contents of the /config directory again:

root@main3200> request system configuration rescue save

root@main3200> file list /config/

/config/:
.snap/
db /
juniper.conf.1.gz
juniper.conf.2.gz
juniper.conf.3.gz
juniper.conf.gz
rescue.conf.gz
ssh_host_dsa_key
ssh_host_dsa_key.pub
ssh_host_ecdsa_key
ssh_host_ecdsa_key.pub
ssh_host_key
ssh_host_key.pub
ssh_host_rsa_key
ssh_host_rsa_key.pub
usage.db
vchassis/

root@main3200>


Notice the new, bold-face file in the listing [edit: it's not actually bold-face on the switch; I highlighted it so it would stand out]? That's the rescue configuration. If you need to remove the rescue configuration, the "request system configuration rescue delete" command will do it. If you reboot the switch without an active configuration, the switch will boot with the rescue configuration, instead. Once the switch has booted, you can commit to write a new active configuration to the file system.

Wednesday, October 16, 2013

Restoring the Boot Image on a Foundry FES2402 Switch

Don't ask me how I know this...

If, by some strange alignment of the stars and planets, both the primary and secondary flash images on a Foundry FastIron Edge switch happens to get erased, AND the switch happens to be rebooted before you catch the problem, it is possible to reinstall a flash image via TFTP from the boot monitor. Here's how to do it. You know, just in case it were to happen...speaking hypothetically, of course.

When the switch reboots without a flash image, it will enter boot monitor mode. You will know it is in boot monitor mode because the prompt will say...:

BOOT MONITOR>

First, you need to assign an IP address and, if necessary, a default gateway to the switch:

BOOT MONITOR>ip address 10.1.2.3/24
Load bootp-tftp image
ip address = 10.1.2.3, subnet mask = 255.255.255.0
BOOT MONITOR>ip default-gateway 10.1.2.1
ip default gateway = 10.1.2.1
BOOT MONITOR>


Next, verify that the IP address and default gateway have been set correctly:

BOOT MONITOR>show ip
     IP address: 10.1.2.3
    Subnet mask: 255.255.255.0
Default gateway: 10.1.2.1
BOOT MONITOR>


So far, so good. Now verify that you can reach the TFTP server (don't forget to start the TFTP server!):

BOOT MONITOR>ping 10.1.2.2
Received 1 replies from remote host
BOOT MONITOR>


If you can't ping the TFTP server, find out why before proceeding. Otherwise, boot the switch from TFTP:
BOOT MONITOR>boot system tftp 10.1.2.1 FES03500.bin
BOOT MONITOR>


At this point, the switch should be running the new image, but it hasn't stored the image in flash yet. Unfortunately, you can't just write the running image to the flash drive; you have to upload it from the TFTP server again:
FES2402 SWITCH>en
FES2402 SWITCH#config t
FES2402 SWITCH(config)#ip address 10.1.2.3/24
FES2402 SWITCH(config)#exit
FES2402 Switch#copy tftp flash 10.1.2.2 FES03500.bin primary
FES2402 Switch#Flash Memory Write (8192 bytes per dot)........................
............................................................................................................
............................................................................................................
...........................
TFTP to Flash Done.
FES2402 Switch#


You should have a nice, shiny, new image in flash now. But if something didn't go right (like, I don't know, maybe you forgot to set your IP address, or you set it wrong, perhaps?), you don't want to start over from the beginning. So, before you reboot the switch, verify that you have the new image in flash:
FES2402 Switch#show flash
    Compressed Pri Code size = 2193091, Version 03.5.00Tc1 (FES03500.bin)
    Sec Code Flash Empty
    ...


You will get a lot more output than this, but as you can see from the first line of output, Version 03.5.00Tc1 (FES03500.bin) is installed in the primary flash. As long as you see that your image has been stored in the primary flash, you can reboot the switch. If you don't see it, figure out what went wrong and try again.

Do you see the image in flash, now? Good. Let's reboot the switch:
FES2402 Switch#reload
Are you sure? (enter 'y' or 'n'): y
Could not verify if the Running Config data has been changed.
Do you want to continue the reload anyway? (enter 'y' or 'n'): y
Halt and reboot

FE Boot Monitor Version 03.5.00 (BLDR-Rev1a)
Enter 'b' to go to boot monitor ...
BOOT INFO: load monitor from code flash, size = 49021
BOOT INFO: load image from primary copy
BOOT INFO: bootparam at 000228e0, mp_flash_size = 002176e3
BOOT INFO: code decompression completed
BOOT INFO: branch to 00400100
........
Crossbar initialization Done
Parsing Config Data ...
INFO: empty config data in the primary area, try to read from backup
INFO: empty config data in the backup area also

  SW: Version 03.5.00Tc1 Copyright (c) 1996-2006 Foundry Networks, Inc.
      Compiled on Mar 01 2006 at 10:06:41 labeled as FES03500
      (2193091 bytes) from Primary FES03500.bin
      Boot Monitor: Version 03.5.00Tc4
...
FES2402 Switch>en
No password has been assigned yet...
FES2402 Switch#


WOOHOO! You dodged the bullet this time! Reload your configs on the switch, and next time, be more careful!!!

Thursday, October 10, 2013

Lesson 15 -- HSRP

You finally get caught up on all of your work and schedule a well-deserved week of PTO. The big day comes, and you relish in the chance to sleep in a little bit. You finally get up, then load up your motorcycle with gear (hey, it's my story, and with a week off, I'd go someplace cool on my motorcycle, lol) and race out of town to go camping in the mountains. You have just picked out a nice camping spot in the woods near a calming, babbling stream -- very Zen, you decide -- when you suddenly hear the most dreadful, blood-curdling, adrenaline-inducing noise known to man. The horrible wail sends chills up your spine, and you cringe with fear, wondering what is about to happen. You take a deep breath, summon all of your courage and...

...answer your cell phone.

It's the manager of the branch office. "I'm so sorry to bother you on the first day of your vacation, but we've just lost the network to the main office."

"No problem," you lie, sighing. You walk her through a few initial troubleshooting steps -- pings, traceroutes and such -- and confirm that the branch office router is off-line. You ask her to go to the wire closet and look for a power light on the front of the router. It takes a few tries to help her find the correct device, but soon enough, she has located it, and as you suspected, it is powered off. She cycles the power switch to no avail, so you have her check the circuit breakers in the wall. Sure enough, the breaker has tripped, so you have her reset it, and she tells you that she now sees "blinky lights" on the front of the router. By the time she has returned to her desk, the router has booted and she verifies that she can now reach the main office network again.

"Thanks again for the help," she tells you, as she hangs up, while you silently vow to camp tomorrow night in a remote valley farther back in the mountains where there is no cell service.

The rest of the week passes with no more interruptions, and you return to the office the following Monday morning with a plan to buy a second router and configure HSRP so that a failure of a router will cause the network to route all traffic through a standby router, reducing the number of after-hours calls you receive and providing better uptime to your users.

But mostly to reduce the number of after-hours calls you receive... :)

When the new router arrives, you copy the config from the existing router, then change the hostname on the new router and change the IP address of the serial line back to the main office. Once you've verified that the serial line to the main office is working, you configure HSRP on both routers. On the old router, you delete the IP address on the Fa0/0 port, then add these lines:

lab3640rtra(config)#interface FastEthernet0/0
lab3640rtra(config-if)#ip address 192.168.2.2 255.255.255.0
lab3640rtra(config-if)#standby 42 ip 192.168.2.1
lab3640rtra(config-if)#standby 42 priority 64
lab3640rtra(config-if)#standby 42 name Branch_Office_HSRP_Group


...and on the new router:

lab3640rtrb(config)#interface FastEthernet0/0
lab3640rtrb(config-if)#ip address 192.168.2.3 255.255.255.0
lab3640rtrb(config-if)#standby 42 ip 192.168.2.1
lab3640rtrb(config-if)#standby 42 priority 128
lab3640rtrb(config-if)#standby 42 name Branch_Office_HSRP_Group


A couple of things are important in this configuration. First, the old router (3640a) used to have the IP address 192.168.2.1./24 assigned to fa0/0, but I have now assigned that IP address to the standby group ("standby 42 ip 192.168.2.1"). Next, both the old router and the new router have new IP addresses in the same subnet as the IP address for the standby group (192.168.2.2, 192.168.2.3 and 192.168.2.1, respectively). The two routers will be negotiating responsibility for forwarding LAN traffic to the WAN port over their new IP addresses, and will be listening for traffic from the LAN on 192.168.2.1 (which the LAN hosts will use as the default gateway address). If the active router goes off-line for some reason, the standby router will notice that it is no longer receiving keep-alive messages, and will assume the role of the default gateway. Finally, notice that all of the lines that activate HSRP begin, "standby 42..." The number 42 is the standby group number, and can be anything that you desire (well, within the bounds of 0 and 255, anyway). I picked 42 because it is "a completely ordinary number, a number not just divisible by two but also six and seven. In fact it's the sort of number that you could without any fear introduce to your parents." Or something like that, anyway, lol.

<...shoots the rabbit, gets off of the bunny trail and gets back on topic...>

After setting up HSRP, you tap out a few more commands on the keyboard to make sure the routers really are providing fail-over protection:

lab3640artr#sho standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp Prio P State    Active addr     Standby addr    Group addr     
Fa0/0       42  64     Standby  192.168.2.3     local           192.168.2.1    
lab3640artr#


...and...:

lab3640brtr#sho standby brief
                     P indicates configured to preempt.
                     |
Interface   Grp Prio P State    Active          Standby         Virtual IP     
Fa0/0       42  128    Active   local           192.168.2.2     192.168.2.1    
lab3640brtr#


Yep, looks good! You hook up a laptop to an unused switchport, then start pinging the main office router. Since 3640b is the active and 3640a is the standby, you decide to pull the Ethernet cable to 3640b to verify that the failover occurs as expected, and to see how many pings you drop during the switch:

64 bytes from 192.168.2.1: icmp_req=8 ttl=255 time=1.35ms
64 bytes from 192.168.2.1: icmp_req=9 ttl=255 time=1.38ms
64 bytes from 192.168.2.1: icmp_req=18 ttl=255 time=1.17ms
64 bytes from 192.168.2.1: icmp_req=19 ttl=255 time=1.14ms


Hmmm...about 9 seconds to fail over. Users will probably notice the drop, if they are paying attention, but the network will be back up before they can dial your cell phone. You check the HSRP status on the two routers again:

lab3640artr#sho standby brief                      P indicates configured to preempt.                      | Interface   Grp Prio P State    Active addr     Standby addr    Group addr      Fa0/0       42  64     Active   local           unknown         192.168.2.1     lab3640artr#

...and...:

lab3640brtr#sho standby brief                      P indicates configured to preempt.                      | Interface   Grp Prio P State    Active          Standby         Virtual IP      Fa0/0       42  128    Init     unknown         unknown         192.168.2.1     lab3640brtr#

Looks good. The output shows that the routers have switched roles, due to the disconnected network cable on the 3640b. Unless the WAN link on the active router is flapping, that should provide some redundancy between the main office and branch office networks for the users in the branch office. You pat yourself on the back and call it a day.

Tuesday, October 8, 2013

Lesson 14 -- Advanced WAN Configurations, Part 1

In Lesson 5 you set up a very simple WAN between your main office 2651 router and the remote site's 3640 router. The configuration for this link was pretty minimalist:

interface Serial0/1
description T1 to lab3640rtr
bandwidth 1544
ip address 192.168.3.5 255.255.255.252


This configuration used HDLC -- the default on Cisco routers -- to establish the WAN link to the 3640 router, set the bandwidth (used by routing protocols that consider bandwidth in their metrics) to 1.544Mbps, and establishes a /30 (two host) subnet between the routers, one IP address for each endpoint. However, you decide that you would like to implement PPP rather than HDLC, since PPP can be configured to use PAP or CHAP to authenticate the routers on each side of the link, and because if you manage to scrounge a couple of new T1 cards (and the funds to buy a second leased line) you can use Multilink PPP to give you a 3Mbps channel between sites. You telnet to the remote router, issue a "reload in 30" command to return the 3640 to its original configuration if you botch the changes (it's saved my backside more than once), then get started on configuring PPP:

lab3640rtr#conf t
lab3640rtr(config)#int serial 0/1
lab3640rtr(config-if)#encapsulation ppp


Your telnet session locks up, but you expected that. Now, configure the local side:

lab2651rtr#conf t
lab2651rtr(config-if)#encapsulation ppp
lab2651rtr(config-if)#exit
lab2651rtr(config)#exit
lab2651rtr#sho int ser0/1
Serial0/1 is up, line protocol is up


You return to your telnet session to the 3640 router, and hit the "Enter" key a couple of times to make sure it's responding again, and it is. Woohoo! You've enabled PPP successfully. You type "reload cancel" and "write mem" on both routers to save the current configurations, then set up PPP authentication using CHAP. To do this, you'll need to configure a user name on each router that matches the host name of the other router and assign the same password to both user names. That is, on the lab3640rtr router, you will create a user "lab2651rtr" and on the lab2651rtr, you will create a user "lab3640rtr":

lab2651rtr#conf t
lab2651rtr(config)#username lab3640rtr password myChapPw


...and on the remote router:

lab3640rtr#conf t
lab3640rtr(config)#username lab2651rtr password myChapPw


Next, in the Serial 0/1 interface configurations, you will tell the routers to authenticate using CHAP (substitute the word "pap" for "chap" in the config if you would rather use PAP):

lab3640rtr(config)#int ser0/1
lab3640rtr(config-if)#ppp authentication chap


...and on the local router:

lab2651rtr(config)#int serial 0/1
lab2651rtr(config-if)#ppp authentication chap


...and that's it!

Note 1: No joke about using the "reload in 30" before making config changes to the management interfaces on a remote router. It can be the difference between being the hero of the day and the idiot who had the misfortune to execute a career-limiting move. True story from the IT trenches: I work in Anchorage, Alaska, but manage network devices in Bethel, Alaska, 500 miles to the west (and there are no roads to Bethel from Anchorage -- it's a one hour flight via Alaska Airlines if you have to go in person). One day, I was working on my LAN router/firewall in Bethel, and fortunately, I heard that little voice inside my head telling me to run the "/sbin/shutdown -r +15 &" command (it's a Linux-based router, but that's the equivalent of Cisco's "reload in 15" command). I did, even though I was sure what I was going to do wouldn't lock me out of the router. I started making changes, and when I restarted the network interfaces, I lost my SSH connection to the router. I tried to create a new SSH session, but no joy. I tried to ping the router...nothing. I even ran "nmap -sP" just to make sure it was really off-line, but I already knew it would be (and it was). About two minutes later, my boss hollers at me from his office, "I think we just lost the Bethel router!" I calmly replied, "Yep, but it should be back in...<...checks watch...>...maybe ten minutes." I explained what happened, and he just said, "Let me know when it's back on-line." It's never good to take a LAN/WAN router off-line in the middle of the business day without arranging a scheduled outage, but, well, (*cough*) excrement occurs sometimes. However, had I not had the foresight ("luck") to tell the router reboot itself in fifteen minutes before I started working on it, I would have been in a much less comfortable position when my boss asked about the router being off-line.

Note 2: Notice how I made changes to the remote router before making changes to the local router every time? That's not by coincidence :) I lost my connection to the remote router when I removed HDLC encapsulation and added PPP encapsulation. If I had made the changes to lab2651rtr first, I would not have been able to reach lab3640rtr to make the changes that would restore the connection. It's obvious in hindsight, but maybe not so obvious the first time you are making changes in the middle of the night so as to not take users off-line during the business day. Yep, been there and done that, too :)

Note 3: Extra-credit: want to make sure that PPP really is doing what you think it's doing? Delete the "username lab3640rtr password myChapPw" from the config, then shut/no shut the interface, and run "sho ip int brief". You should see the hardware up and the line protocol down on the serial interface, indicating that there is a misconfiguration on the layer-2 protocols you are using on the serial interface.

Monday, October 7, 2013

Appendix B -- More Detail on OSPF

OSPF is a very, very deep subject, and my original post on the topic didn't even begin to do justice to the complexity and intricacy of OSPF. As I began preparing for the CCNA exam, I realized there were some fundamentals of OSPF that I didn't quite grasp, and so -- just as I did with Spanning Tree -- I powered up my Cisco lab and set to work trying to fill in some of the gaps in my knowledge.

Classful vs. Classless Routing: The first thing I wanted to understand was why I was getting the following error on my Cisco routers when I would run the "redistribute connected" command:

lab3640rtr#conf t
lab3640rtr(config)#router ospf 42
lab3640rtr(config-router)#redistribute connected
% Only classful networks will be redistributed
lab3640rtr(config-router)#


Ummm...what? I would understand that error on RIPv1, since it's a classful protocol, but why am I getting it in OSPF?!?! I tried adding the "ip classless" command in global config mode since maybe the router wasn't operating in classless mode, but that made no difference.

Apparently, this isn't just a nuisance message either, because my lab2651rtr router (the one with 192.168.1.0/24 divided into multiple /27 and /28 subnets), isn't propagating those 192.168.1.x routes to the other routers:

lab3640rtr#sho ip route
<...snip...>
Gateway of last resort is not set

     100.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
O IA    100.64.1.0/24 [110/128] via 192.168.3.5, 00:09:29, Serial0/1
O E2    100.64.1.22/32 [110/20] via 192.168.3.5, 00:09:29, Serial0/1
     10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
O IA    10.254.254.0/30 [110/129] via 192.168.3.5, 00:09:29, Serial0/1
O E2    10.100.64.2/32 [110/20] via 192.168.3.5, 00:09:29, Serial0/1
O E2    10.100.64.1/32 [110/20] via 192.168.3.5, 00:09:29, Serial0/1
     192.168.3.0/30 is subnetted, 1 subnets
C       192.168.3.4 is directly connected, Serial0/1
lab3640rtr#


After poking around with OSPF configurations, and with the help of the friendly "?" character on the CLI ;) I managed to find an additional parameter on the "redistribute connected" command:

lab3640rtr(config-router)#no redistribute connected
lab3640rtr(config-router)#redistribute connected subnets
lab3640rtr(config-router)#


Hey, what do you know! No error message! I applied the command on my routers, then checked my routing tables:

lab3640rtr#sho ip route
<...snip...>
     100.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
O E2    100.64.1.1/32 [110/20] via 192.168.3.5, 00:02:43, Serial0/1
O IA    100.64.1.0/24 [110/128] via 192.168.3.5, 00:02:43, Serial0/1
O E2    100.64.1.22/32 [110/20] via 192.168.3.5, 00:02:43, Serial0/1
     10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
O IA    10.254.254.0/30 [110/129] via 192.168.3.5, 00:02:43, Serial0/1
O E2    10.100.64.2/32 [110/20] via 192.168.3.5, 00:02:43, Serial0/1
O E2    10.100.64.1/32 [110/20] via 192.168.3.5, 00:02:43, Serial0/1
     192.168.1.0/24 is variably subnetted, 7 subnets, 2 masks
O E2    192.168.1.96/28 [110/20] via 192.168.3.5, 00:02:43, Serial0/1
O E2    192.168.1.112/28 [110/20] via 192.168.3.5, 00:02:43, Serial0/1
O E2    192.168.1.64/28 [110/20] via 192.168.3.5, 00:02:43, Serial0/1
O E2    192.168.1.80/28 [110/20] via 192.168.3.5, 00:02:43, Serial0/1
O E2    192.168.1.32/27 [110/20] via 192.168.3.5, 00:02:43, Serial0/1
O E2    192.168.1.0/27 [110/20] via 192.168.3.5, 00:02:43, Serial0/1
O E2    192.168.1.128/27 [110/20] via 192.168.3.5, 00:02:43, Serial0/1

     192.168.3.0/30 is subnetted, 1 subnets
C       192.168.3.4 is directly connected, Serial0/1
lab3640rtr#


w00t! There are my subnets!

As an aside, there is also a "redistribute static subnets" as well, which I assume serves the same purpose.

Router ID's: One other concept I had difficulty with is how a router running OSPF automatically assigns a router ID. Conceptually, it is simple -- if the administrator doesn't statically assign a router ID, then the router ID is chosen from the IP addresses configured on the router. Unfortunately, between the multiple websites I visited, the numerous CCNA prep books I read, and the various practice tests I used to study, I got thoroughly confused as to which IP address the router will use as its router ID. I could always search for the most popular answer among my numerous resources, but...it's more fun (and it will stick with me better) if I actually test it on live equipment. To that end, I took my trusty 2651 router and one of the 3640 routers, configured OSPF, and left the router-id statement out of the config. Then, I mucked about with the various interfaces until I understood how the routers assign a router ID.

I started by removing all of the IP addresses from the lab3640rtr router, except for the serial interface connecting to the lab2651rtr (192.168.3.6/30), then ran the "sho ip ospf int" command:

lab3640rtr#sho run | inc address
no ip address
no ip address
ip address 192.168.3.6 255.255.255.252
lab3640rtr#sho ip ospf int
Serial0/1 is up, line protocol is up
  Internet Address 192.168.3.6/30, Area 3
  Process ID 42, Router ID 192.168.3.6, Network Type POINT_TO_POINT, Cost: 64
<...snip...>


No surprise, there. The router ID is the one IP address available on the router, 192.168.3.6. What happens if I put an IP address back on Fa0/0, but leave the interface shut down? Well, there's only one way to find out:

lab3640rtr#conf t
lab3640rtr(config)#int fa0/0
lab3640rtr(config-if)#ip address 192.168.2.1 255.255.255.0
lab3640rtr(config-if)#shut
lab3640rtr(config-if)#exit
lab3640rtr(config)#exit
lab3640rtr#sho ip ospf intSerial0/1 is up, line protocol is up
  Internet Address 192.168.3.6/30, Area 3
  Process ID 42, Router ID 192.168.3.6, Network Type POINT_TO_POINT, Cost: 64
<...snip...>


It's still using the address of the serial interface; not entirely unexpected. What if we enable Fa0/0?

lab3640rtr#conf t
lab3640rtr(config)#int fa0/0
lab3640rtr(config-if)#no shut
lab3640rtr(config-if)#exit
lab3640rtr(config)#exit
lab3640rtr#sho ip ospf int
Serial0/1 is up, line protocol is up
  Internet Address 192.168.3.6/30, Area 3
  Process ID 42, Router ID 192.168.3.6, Network Type POINT_TO_POINT, Cost: 64
<...snip...>


Still using the 192.168.3.6 address for the router ID. Is that just because Fa0/0 is not up (there's no device connected to it)? Let's connect an Ethernet cable to the 2924 switch and see what happens:

lab3640rtr#sho ip int brief | inc FastEthernet0/0
FastEthernet0/0            192.168.2.1     YES manual up                    down
lab3640rtr#sho ip int brief | inc FastEthernet0/0
00:25:13: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
lab3640rtr#sho ip int brief | inc FastEthernet0/0
FastEthernet0/0            192.168.2.1     YES manual up                    up
lab3640rtr#sho ip ospf int
Serial0/1 is up, line protocol is up
  Internet Address 192.168.3.6/30, Area 3
  Process ID 42, Router ID 192.168.3.6, Network Type POINT_TO_POINT, Cost: 64


Apparently not; 192.168.3.6 is still the router ID. Maybe it's because 192.168.2.0/24 is not in the OSPF configuration?

lab3640rtr#sho ip ospf int | inc ID
  Process ID 42, Router ID 192.168.3.6, Network Type BROADCAST, Cost: 1
  Designated Router (ID) 192.168.3.6, Interface address 192.168.2.1
  Process ID 42, Router ID 192.168.3.6, Network Type POINT_TO_POINT, Cost: 64
lab3640rtr#


Maybe the router is picking the highest IP address for the router ID?

lab3640rtr(config)#int fa0/0
lab3640rtr(config-if)#no ip address
lab3640rtr(config-if)#ip address 192.168.168.192 255.255.255.0
lab3640rtr(config-if)#router ospf 42
lab3640rtr(config-router)#no network 192.168.2.0 0.0.0.255 area 3
lab3640rtr(config-router)#network 192.168.168.192 0.0.0.255 area 3
lab3640rtr(config-router)#exit
lab3640rtr(config)#exit
lab3640rtr#sho ip ospf int | inc ID
  Process ID 42, Router ID 192.168.3.6, Network Type BROADCAST, Cost: 1
  Process ID 42, Router ID 192.168.3.6, Network Type POINT_TO_POINT, Cost: 64
lab3640rtr#


Hmmm...I really thought the router ID would have changed at some point now. Let's try restarting the OSPF process, and see what happens:

lab3640rtr(config)#no router ospf 42
lab3640rtr(config)#router ospf 42
lab3640rtr(config-router)# log-adjacency-changes
lab3640rtr(config-router)# redistribute connected subnets
lab3640rtr(config-router)# network 192.168.3.4 0.0.0.3 area 3
lab3640rtr(config-router)# network 192.168.168.0 0.0.0.255 area 3
lab3640rtr(config-router)#exit
lab3640rtr(config)#exit
lab3640rtr#sho ip ospf int | inc ID
  Process ID 42, Router ID 192.168.168.192, Network Type BROADCAST, Cost: 1
  Designated Router (ID) 192.168.168.192, Interface address 192.168.168.192
  Process ID 42, Router ID 192.168.168.192, Network Type POINT_TO_POINT, Cost: 64
lab3640rtr#


And we have a winner!!! Restarting OSPF with a higher numbered interface did the trick. What if we configure a loopback interface? Will OSPF use a loopback interface for a router ID over a FastEthernet or Serial interface? We'll configure a low-numbered loopback0 and see if OSPF grabs it or sticks with the high-numbered Fa0/0:

lab3640rtr(config)#int loopback0
lab3640rtr(config-if)#ip address 192.168.2.1 255.255.255.255
lab3640rtr(config-if)#no router ospf 42
lab3640rtr(config)#router ospf 42
lab3640rtr(config-router)# log-adjacency-changes
lab3640rtr(config-router)# redistribute connected subnets
lab3640rtr(config-router)# network 192.168.3.4 0.0.0.3 area 3
lab3640rtr(config-router)# network 192.168.168.0 0.0.0.255 area 3
lab3640rtr(config-router)# network 192.168.2.1 0.0.0.0 area 3
lab3640rtr(config-router)#exit
lab3640rtr(config)#exit
lab3640rtr#sho ip ospf int | inc ID
  Process ID 42, Router ID 192.168.2.1, Network Type LOOPBACK, Cost: 1
  Process ID 42, Router ID 192.168.2.1, Network Type BROADCAST, Cost: 1
  Designated Router (ID) 192.168.2.1, Interface address 192.168.168.192
  Process ID 42, Router ID 192.168.2.1, Network Type POINT_TO_POINT, Cost: 64
lab3640rtr#


Weird...I didn't expect that. I can see why a router might prefer a loopback interface for an OSPF router ID (a loopback interface never goes down, for one thing). Let's just double-check the preference for the loopback by swapping the networks for the loopback and Fa0/0:

lab3640rtr(config)#int loopback0
lab3640rtr(config-if)#no ip address
lab3640rtr(config-if)#int fa0/0
lab3640rtr(config-if)#no ip address
lab3640rtr(config-if)#ip address 192.168.2.1 255.255.255.0
lab3640rtr(config-if)#int loopback0
lab3640rtr(config-if)#ip address 192.168.168.192 255.255.255.255
lab3640rtr(config-if)#no router ospf 42
lab3640rtr(config)#router ospf 42
lab3640rtr(config-router)#log-adjacency-changes
lab3640rtr(config-router)#redistribute connected subnets
lab3640rtr(config-router)#network 192.168.3.4 0.0.0.3 area 3
lab3640rtr(config-router)#network 192.168.168.192 0.0.0.0 area 3
lab3640rtr(config-router)#network 192.168.2.0 0.0.0.255 area 3
lab3640rtr(config-router)#exit
lab3640rtr(config)#exit
lab3640rtr#sho ip ospf int | inc ID
  Process ID 42, Router ID 192.168.168.192, Network Type BROADCAST, Cost: 1
  Designated Router (ID) 192.168.168.192, Interface address 192.168.2.1
  Process ID 42, Router ID 192.168.168.192, Network Type LOOPBACK, Cost: 1
  Process ID 42, Router ID 192.168.168.192, Network Type POINT_TO_POINT, Cost: 64
lab3640rtr#


Yep, it still preferred to use the loopback. Okay, that makes sense. But that raises a question: I didn't get Fa0/0's IP address to become the router ID until I restarted the OSPF process. Did Fa0/0's IP address become the router ID because its IP address was greater than Serial0/0's address, or because it was a FastEthernet interface? Let's find out:

lab3640rtr(config)#no int loopback0
lab3640rtr(config)#no router ospf 42
lab3640rtr(config)#router ospf 42
lab3640rtr(config-router)# log-adjacency-changes
lab3640rtr(config-router)# redistribute connected subnets
lab3640rtr(config-router)# network 192.168.2.0 0.0.0.255 area 3
lab3640rtr(config-router)# network 192.168.3.4 0.0.0.3 area 3
lab3640rtr(config-router)#exit
lab3640rtr(config)#exit
lab3640rtr#sho ip ospf int | inc ID
  Process ID 42, Router ID 192.168.3.6, Network Type POINT_TO_POINT, Cost: 64
  Process ID 42, Router ID 192.168.3.6, Network Type BROADCAST, Cost: 1
  Designated Router (ID) 192.168.3.6, Interface address 192.168.2.1
lab3640rtr#


There you have it: if a loopback interface is present, OSPF will use it as the router ID, and if not, OSPF will use the highest IP address configured on the system. HOWEVER, OSPF will NOT change the router ID until the OSPF process is restarted, which makes sense if you think about it.

And now...it's late. I'm going to bed ;)

Sunday, October 6, 2013

Appendix A -- More on Spanning Tree

Spanning Tree was enabled by default on the 2924 switches I have in my lab, but sometimes the network admin needs to tweak the settings a little. For example, the network admin may want to force a particular switch to be the root bridge -- wouldn't it make sense to have the most powerful switch in your network be the root bridge rather than a small, inexpensive, low-powered access switch in a wire closet in the hut next door to your main building on your campus?

Unfortunately, I have become quite frustrated with the documentation on Spanning Tree in the several CCNA test prep books I have purchased over the years (I started working on a CCNA over eight years ago, but circumstances changed, and I never took the exam). I don't know about you, but for me, I don't retain much by simply reading technical documentation. Since I can't afford to NOT know the correct answers when I take the CCNA exam this coming week, I decided the best way to get the straight scoop was to configure, tweak, and observe Spanning Tree on my switches.

Root Bridge Election: Let's start with manipulating the root bridge election process. This was an area where I got really frustrated with the documentation, so let's just play with it on the switches and see what happens. Here goes:

lab2924a#sho span

Spanning tree 1 is executing the IEEE compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, address 0003.e3e4.f887
  Configured hello time 2, max age 20, forward delay 15
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set, changes 1
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15
  Timers: hello 0, topology change 0, notification 0

Interface Fa0/1 (port 13) in Spanning tree 1 is FORWARDING
   Port path cost 19, Port priority 128
   Designated root has priority 32768, address 0003.e3e4.f887
   Designated bridge has priority 32768, address 0003.e3e4.f887
   Designated port is 13, path cost 0
   Timers: message age 0, forward delay 0, hold 0
   BPDU: sent 1416, received 8

Interface Fa0/2 (port 14) in Spanning tree 1 is FORWARDING
   Port path cost 19, Port priority 128
   Designated root has priority 32768, address 0003.e3e4.f887
   Designated bridge has priority 32768, address 0003.e3e4.f887
   Designated port is 14, path cost 0
   Timers: message age 0, forward delay 0, hold 0
   BPDU: sent 1383, received 2


...and lab2924b:

lab2924b#sho span

Spanning tree 1 is executing the IEEE compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, address 00d0.58dd.d186
  Configured hello time 2, max age 20, forward delay 15
  Current root has priority 32768, address 0003.e3e4.f887
  Root port is 13, cost of root path is 19
  Topology change flag not set, detected flag not set, changes 3
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15
  Timers: hello 0, topology change 0, notification 0

Interface Fa0/1 (port 13) in Spanning tree 1 is FORWARDING
   Port path cost 19, Port priority 128
   Designated root has priority 32768, address 0003.e3e4.f887
   Designated bridge has priority 32768, address 0003.e3e4.f887
   Designated port is 13, path cost 0
   Timers: message age 3, forward delay 0, hold 0
   BPDU: sent 39, received 3092

Interface Fa0/2 (port 14) in Spanning tree 1 is BLOCKING
   Port path cost 19, Port priority 128
   Designated root has priority 32768, address 0003.e3e4.f887
   Designated bridge has priority 32768, address 0003.e3e4.f887
   Designated port is 14, path cost 0
   Timers: message age 6, forward delay 0, hold 0
   BPDU: sent 1, received 2916


The rest of the ports on the two switches are just "leaf nodes" -- that is, they connect to routers and laptops, rather to switches, and therefore, they are incapable of creating loops in the layer-2 network. From the output above, we can see that lab2924a is the root bridge, that fa0/1 and fa0/2 on lab2924a are designated ports, that fa0/1 on lab2924b is a root port and that fa0/2 on lab2924b is a blocking port. What happens if I pull the Ethernet cable on fa0/1 on one of the switches? Let's find out! Here's what I see on lab2924a:

lab2924a#sho span

Spanning tree 1 is executing the IEEE compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, address 0003.e3e4.f887
  Configured hello time 2, max age 20, forward delay 15
  We are the root of the spanning tree
  Topology change flag set, detected flag set, changes 2
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15
  Timers: hello 0, topology change 30, notification 0

Interface Fa0/1 (port 13) in Spanning tree 1 is down
   Port path cost 19, Port priority 128
   Designated root has priority 32768, address 0003.e3e4.f887
   Designated bridge has priority 32768, address 0003.e3e4.f887
   Designated port is 13, path cost 0
   Timers: message age 0, forward delay 0, hold 0
   BPDU: sent 1734, received 8

Interface Fa0/2 (port 14) in Spanning tree 1 is FORWARDING
   Port path cost 19, Port priority 128
   Designated root has priority 32768, address 0003.e3e4.f887
   Designated bridge has priority 32768, address 0003.e3e4.f887
   Designated port is 14, path cost 0
   Timers: message age 0, forward delay 0, hold 0
   BPDU: sent 1705, received 4


For almost a full minute, however, I couldn't see anything from lab2924b (since I was directly connected to 2924a), which underscores a big problem with STP -- the network will recover from failure of a redundant link, but users will definitely notice the outage while spanning tree reconverges. Eventually, however, my telnet session recovered, and here is what I saw on lab2924b:

lab2924b#sho span

Spanning tree 1 is executing the IEEE compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, address 00d0.58dd.d186
  Configured hello time 2, max age 20, forward delay 15
  Current root has priority 32768, address 0003.e3e4.f887
  Root port is 14, cost of root path is 19
  Topology change flag set, detected flag not set, changes 4
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15
  Timers: hello 0, topology change 0, notification 0

Interface Fa0/1 (port 13) in Spanning tree 1 is down
   Port path cost 19, Port priority 128
   Designated root has priority 32768, address 0003.e3e4.f887
   Designated bridge has priority 32768, address 00d0.58dd.d186
   Designated port is 13, path cost 19
   Timers: message age 0, forward delay 0, hold 0
   BPDU: sent 39, received 3474

Interface Fa0/2 (port 14) in Spanning tree 1 is FORWARDING
   Port path cost 19, Port priority 128
   Designated root has priority 32768, address 0003.e3e4.f887
   Designated bridge has priority 32768, address 0003.e3e4.f887
   Designated port is 14, path cost 0
   Timers: message age 1, forward delay 0, hold 0
   BPDU: sent 3, received 3340


Now we see that fa0/1 is down on both switches, and fa0/2 is forwarding on both switches once again. No surprises, right?

From the output above, we can see that both switches have a priority of 32768. If both switches have the same priority, then the choice of root bridge is decided by MAC address. lab2924a has MAC 0003.e3e4.f887 and lab2924b has MAC 00d0.58dd.d186. 0x0003 < 0x00d0 so apparently the low MAC address wins the election process when the priority is the same. But what if I wanted lab2924b to be the root bridge? Would I set the STP priority on lab2924b higher or lower than lab2924a? Let's try it and find out:

lab2924b#conf t
lab2924b(config)#spanning-tree priority 4096
lab2924b(config)#exit
lab2924b#sho span

Spanning tree 1 is executing the IEEE compatible Spanning Tree protocol
  Bridge Identifier has priority 4096, address 00d0.58dd.d186
  Configured hello time 2, max age 20, forward delay 15
  We are the root of the spanning tree
  Topology change flag set, detected flag set, changes 6
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15
  Timers: hello 1, topology change 24, notification 0

Interface Fa0/1 (port 13) in Spanning tree 1 is FORWARDING
   Port path cost 19, Port priority 128
   Designated root has priority 4096, address 00d0.58dd.d186
   Designated bridge has priority 4096, address 00d0.58dd.d186
   Designated port is 13, path cost 0
   Timers: message age 0, forward delay 0, hold 0
   BPDU: sent 48, received 3744

Interface Fa0/2 (port 14) in Spanning tree 1 is LISTENING
   Port path cost 19, Port priority 128
   Designated root has priority 4096, address 00d0.58dd.d186
   Designated bridge has priority 4096, address 00d0.58dd.d186
   Designated port is 14, path cost 0
   Timers: message age 0, forward delay 3, hold 0
   BPDU: sent 10, received 4070


Wow...that was fast! Since STP takes almost a minute to move the ports to the forwarding state, I didn't expect lab2924b to assume its new role as root bridge quite so quickly, but it did. You can see that fa0/2 is still in the listening state as I ran the "sho span" command, but it already knows that it's the root bridge. Edit: I later reset lab2924b's priority back to 32768, and it seemed to take a few seconds to update its status, so it's not quite instantaneous -- say about 10 or 15 seconds, maybe?

How about lab2924a?

lab2924a#sho span

Spanning tree 1 is executing the IEEE compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, address 0003.e3e4.f887
  Configured hello time 2, max age 20, forward delay 15
  Current root has priority 4096, address 00d0.58dd.d186
  Root port is 13, cost of root path is 19
  Topology change flag set, detected flag not set, changes 4
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15
  Timers: hello 0, topology change 0, notification 0

Interface Fa0/1 (port 13) in Spanning tree 1 is FORWARDING
   Port path cost 19, Port priority 128
   Designated root has priority 4096, address 00d0.58dd.d186
   Designated bridge has priority 4096, address 00d0.58dd.d186
   Designated port is 13, path cost 0
   Timers: message age 2, forward delay 0, hold 0
   BPDU: sent 1869, received 36

Interface Fa0/2 (port 14) in Spanning tree 1 is BLOCKING
   Port path cost 19, Port priority 128
   Designated root has priority 4096, address 00d0.58dd.d186
   Designated bridge has priority 4096, address 00d0.58dd.d186
   Designated port is 14, path cost 0
   Timers: message age 10, forward delay 0, hold 0
   BPDU: sent 2087, received 30


Yep, it has relinquished its role as root bridge already, and has fa0/2 in a blocking state. So apparently, a low priority and/or a low MAC address makes a switch a root bridge, and the lowest numbered redundant interface on a switch becomes the root port. Easy :)

Here's something even cooler. In our scenario, we have two switches with two redundant links. However, we are only using fa0/1 to carry traffic between the two switches. In effect, while fa0/2 is providing redundancy, it is essentially wasted bandwidth until a failure occurs. Wouldn't it be great if we could distribute the load between these two ports without setting up LAG or Etherchannel? As it turns out, we can.

You see, Spanning Tree runs a separate instance for each VLAN, which is often described as "Per-VLAN Spanning Tree" or "PVST." It turns out that we can tweak the priority of each VLAN on the FastEthernet port on each switch, making fa0/1 the designated/root port for some VLANs and fa0/2 the designated/root port for other VLANs. To keep it simple, I set this up with odd numbered VLANs on fa0/1 and even numbered VLANs on fa0/2. In a real environment, it would probably be better to do some traffic analysis and find out which VLANs typically have equal amounts of traffic and try to balance load intelligently. At any rate, here's how you do it:

lab2924b(config)#int fa0/1
lab2924b(config-if)#spanning-tree vlan 1 port-priority 64
lab2924b(config-if)#spanning-tree vlan 3 port-priority 64
lab2924b(config-if)#spanning-tree vlan 5 port-priority 64
lab2924b(config-if)#spanning-tree vlan 7 port-priority 64
lab2924b(config-if)#int fa0/2
lab2924b(config-if)#spanning-tree vlan 2 port-priority 64
lab2924b(config-if)#spanning-tree vlan 4 port-priority 64
lab2924b(config-if)#spanning-tree vlan 6 port-priority 64
lab2924b(config-if)#exit
lab2924b(config)#exit
lab2924b#sho spanning-tree vlan 2

Spanning tree 2 is executing the IEEE compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, address 00d0.58dd.d180
  Configured hello time 2, max age 20, forward delay 15
  Current root has priority 32768, address 0003.e3e4.f880
  Root port is 14, cost of root path is 19
  Topology change flag set, detected flag not set, changes 9
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15
  Timers: hello 0, topology change 0, notification 0

Interface Fa0/1 (port 13) in Spanning tree 2 is BLOCKING
   Port path cost 19, Port priority 128
   Designated root has priority 32768, address 0003.e3e4.f880
   Designated bridge has priority 32768, address 0003.e3e4.f880
   Designated port is 13, path cost 0
   Timers: message age 1, forward delay 0, hold 0
   BPDU: sent 8, received 3274

Interface Fa0/2 (port 14) in Spanning tree 2 is FORWARDING
   Port path cost 19, Port priority 64
   Designated root has priority 32768, address 0003.e3e4.f880
   Designated bridge has priority 32768, address 0003.e3e4.f880
   Designated port is 14, path cost 0
   Timers: message age 2, forward delay 0, hold 0
   BPDU: sent 5, received 3376

lab2924b#sho span vlan 1

Spanning tree 1 is executing the IEEE compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, address 00d0.58dd.d186
  Configured hello time 2, max age 20, forward delay 15
  Current root has priority 32768, address 0003.e3e4.f887
  Root port is 13, cost of root path is 19
  Topology change flag not set, detected flag not set, changes 7
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15
  Timers: hello 0, topology change 0, notification 0

Interface Fa0/1 (port 13) in Spanning tree 1 is FORWARDING
   Port path cost 19, Port priority 64
   Designated root has priority 32768, address 0003.e3e4.f887
   Designated bridge has priority 32768, address 0003.e3e4.f887
   Designated port is 13, path cost 0
   Timers: message age 2, forward delay 0, hold 0
   BPDU: sent 347, received 6622

Interface Fa0/2 (port 14) in Spanning tree 1 is BLOCKING
   Port path cost 19, Port priority 128
   Designated root has priority 32768, address 0003.e3e4.f887
   Designated bridge has priority 32768, address 0003.e3e4.f887
   Designated port is 14, path cost 0
   Timers: message age 4, forward delay 0, hold 0
   BPDU: sent 309, received 6948


Important! I didn't show it in the config above, but I repeated the same configuration on lab2924a -- this doesn't seem to work if the priority is changed on only one switch!

Notice how fa0/1 is forwarding on VLAN 1, but is blocking on VLAN 2, and vice versa for fa0/2? You are now utilizing both 100M Ethernet ports on your switch while still avoiding loops!

Saturday, October 5, 2013

Lesson 13 -- MAC Filtering and Port Security

In the wake of the virus incident, you do a little searching on-line and in your Cisco books, and decide perhaps it is time to implement another layer of security on your network. Cisco switches allow the network admin to configure ports to allow no more than a specified number of MAC addresses before taking some action, which on newer switches can be anything from simply dropping offending packets to logging the offence to shutting down the affected switch port entirely. Unfortunately, your 2924 switch only allows a subset of those actions, namely shutting down the port or simply sending an SNMP trap for the security violation. Still peeved that someone connected a personal laptop to your network -- in violation of example.com's corporate compliance policy -- you send out an e-mail to all employees reminding them that example.com does not yet have a BYOD policy, and that company policies currently prohibit connecting personal devices to the corporate network. Then, deciding to risk possibly becoming a bit of a BOfH, you implement port security on the 2924 switches, limiting switch ports to one MAC address per access (untagged VLAN) port, and automatically shutting down ports when a security violation is encountered.

lab2924a# conf t
lab2924a(config)# int fa0/7
lab2924a(config-if)# port security
lab2924a(config-if)# port security max-mac-count 1
lab2924a(config-if)# port security action shutdown
lab2924a(config)# int fa0/8
lab2924a(config-if)# port security
lab2924a(config-if)# port security max-mac-count 1
lab2924a(config-if)# port security action shutdown
lab2924a(config)# int fa0/9
lab2924a(config-if)# port security
lab2924a(config-if)# port security max-mac-count 1
lab2924a(config-if)# port security action shutdown
lab2924a(config)# int fa0/10
lab2924a(config-if)# port security
lab2924a(config-if)# port security max-mac-count 1
lab2924a(config-if)# port security action shutdown
<...snip...you get the idea...>


...and then repeat the config on lab2924b. To make sure everything is working as expected, you go to the server room and disconnect the cable from your port on the switch to the patch panel, then connect your laptop directly into the port. After getting a DHCP address from the server, you disconnect your laptop and connect a second laptop to the same port, and notice the link light briefly turns orange as spanning-tree reconverges, then goes out. You connect your laptop to another unused port, renegotiate an IP address with the DHCP server, then log in to the switch. You notice that the switchport to which the second laptop is still connected shows that it is "administratively down":

lab2924a# sho int fa0/9
FastEthernet0/9 is administratively down, line protocol is down
<...snip...>


You double-check the running configuration in RAM, to make sure that you plugged the second laptop into the correct port, and it shows the port is configured up -- looks like port security is working properly! To reset the port, you run "shut" on fa0/9, turn off port security, then run "no shut" and watch as the port comes up again (after a minute or so delay while spanning tree reconverges again). Since you have a spare laptop plugged into fa0/9, and that's your network port, you disconnect the second laptop, replace the cable in the patch panel, then return to your office before re-enabling port security on fa0/9.

Note: By this point, I'm starting to think that it might have been wise to spend a little more on a more modern switch for my CCNA practice. Once again, the commands I need to practice for the current CCNA exam are not supported on the switch I purchased (sigh...). On a modern switch, the procedure (in theory) goes like this:

switch(config)# int fa0/9
switch(config-if)# switchport port-security
switch(config-if)# switchport port-security mac-address 00:15:c5:08:5d:92


...to statically assign a MAC address to the port security feature,...:

switch(config)# int fa0/9
switch(config-if)# switchport port-security
switch(config-if)# switchport port-security mac-address sticky


...to have the switch automatically use the first MAC address it sees on the port, or...:

switch(config)# int fa0/9
switch(config-if)# switchport port-security
switch(config-if)# switchport port-security maximum 5


...to automatically use the first five MAC addresses the switch learns from a given port. Next, you can select from three possible actions when the switch detects a violation. I'll show examples of a port that simply discards offending traffic (fa0/9), that discards and logs offending traffic (fa0/10), and that shuts down a port entirely (fa0/11):

switch(config)# int fa0/9
switch(config-if)# switchport port-security
switch(config-if)# switchport port-security mac-address sticky
switch(config-if)# switchport port-security violations protect
switch(config)# int fa0/10
switch(config-if)# switchport port-security
switch(config-if)# switchport port-security mac-address sticky
switch(config-if)# switchport port-security violations restrict
switch(config)# int fa0/11
switch(config-if)# switchport port-security
switch(config-if)# switchport port-security mac-address sticky
switch(config-if)# switchport port-security violations shutdown


Also, on newer switches, you supposedly only have to shut/no shut the interface to clear an automatic "shutdown" on a port when a violation occurs. On my 2924, a shut/no shut didn't release the port.

Maybe with all the vast wealth that I should receive from prospective employers after earning my CCNA (yeah, right...) I'll spring for a newer switch before starting in on the CCNP ;)

Lesson 12 -- Access Control Lists

A few days after setting up the LAG groups between the two 2924 switches, you start to receive a number of phone calls from users in the main office, who are again complaining of poor network performance. You telnet to the switches, and sure enough, they seem to be responding more sluggishly than normal. You begin checking network interfaces, and see that utilization is abnormally high on a number of ports. On a lark, you start sniffing network traffic on the Ethernet port of your laptop using tcpdump, and see a large number of connection attempts on ports 80, 135-139, and 3128:

me@myllt:~$ sudo tcpdump -envi eth0 -s0 port 80 or port 135 or port 136 or port 137 or port 138 or port 139 or port 3128
[sudo] password for me:
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:14:23.538876 00:05:9b:be:cf:60 > 00:15:c5:08:5d:92, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 44, id 14733, offset 0, flags [none], proto TCP (6), length 40)
    192.168.1.114.63816 > 192.168.1.66.80: Flags [.], cksum 0x2827 (correct), ack 2159576407, win 1024, length 0 14:14:23.538907 00:15:c5:08:5d:92 > 00:05:9b:be:cf:60, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.1.66.80 > 192.168.1.114.63816: Flags [R], cksum 0x2c33 (correct), seq 2159576407, win 0, length 0 14:14:23.562432 00:05:9b:be:cf:60 > 00:15:c5:08:5d:92, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 58, id 53140, offset 0, flags [none], proto TCP (6), length 40)
<...snip...>
me@myllt:~$


You know that port 80 is the well-known port for HTTP and that ports 135-139 are well-known ports for Microsoft Windows file sharing (SMB/CIFS). However, it is port 3128 that really gets your attention. A number of worms and trojans have attempted to use port 3128 as a back door to compromised machines. You suspect someone may have a desktop or laptop with a virus, as a result of the tcpdump output, so you create an access control list on your router to block all traffic from the source IP address while you start investigating the problem:

lab2651rtr# conf t
lab2651rtr(config)# access-list 1 deny 192.168.1.114 0.0.0.0
lab2651rtr(config)# access-list 1 permit any


That created an appropriate filter, but it did not tell the router to actually start filtering any traffic. Before the access list can take effect, we have to apply the filter to the network interfaces. Unfortunately, here is where we have made things a little difficult. Because we are using 802.1Q VLAN trunking, we have to apply the ACL ("Access Control List") to each sub-interface -- we can't just apply it to FA0/0 on the router (I tried...):

lab2651rtr(config-if)#int fa0/0.1
lab2651rtr(config-subif)#ip access-group 1 in
lab2651rtr(config-subif)#int fa0/0.2
lab2651rtr(config-subif)#ip access-group 1 in
lab2651rtr(config-subif)#int fa0/0.3
lab2651rtr(config-subif)#ip access-group 1 in
lab2651rtr(config-subif)#int fa0/0.4
lab2651rtr(config-subif)#ip access-group 1 in
lab2651rtr(config-subif)#int fa0/0.5
lab2651rtr(config-subif)#ip access-group 1 in
lab2651rtr(config-subif)#int fa0/0.6
lab2651rtr(config-subif)#ip access-group 1 in
lab2651rtr(config-subif)#int fa0/0.7
lab2651rtr(config-subif)#ip access-group 1 in


Running tcpdump again, you see that you have now blocked all incoming traffic from this host (sort of -- you have blocked all traffic coming through the router, but this host will still be scanning and potentially infecting the rest of the hosts on that same subnet on the switch, since the router is only blocking traffic trying to hit other subnets). As you turn your attention to the switch to try to find the source of the problem so you can disable the switch port, however, you start to see traffic flooding your network again, this time, just hitting port 3128 but coming from a myriad of IP addresses:

<...snip...>     192.168.1.119.55092 > 192.168.1.66.3128: Flags [FPU], cksum 0x8dab (correct), seq 532322015, win 1024, urg 0, length 0
14:35:51.969318 00:15:c5:08:5d:92 > 00:05:9b:be:cf:60, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.1.66.3128 > 192.168.1.119.55092: Flags [R.], cksum 0x91bf (correct), seq 0, ack 532322016, win 0, length 0
14:35:52.775074 00:05:9b:be:cf:60 > 00:15:c5:08:5d:92, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 50, id 46336, offset 0, flags [none], proto TCP (6), length 40)
    192.168.1.120.40785 > 192.168.1.66.3128: Flags [FPU], cksum 0xc243 (correct), seq 2778797122, win 1024, urg 0, length 0
14:35:52.775146 00:15:c5:08:5d:92 > 00:05:9b:be:cf:60, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
<...snip...>


You could just turn off the port on the router, but that would deny service to everyone in the same subnet as the affected machine. You could also create another standard access list that blocks all of the IP addresses that the traffic is now coming from, but again, it looks like that would be every host in the source subnet. You would like to create an access list that is as minimally invasive as possible, while still achieving the desired goals, so you decide to create an extended access list, which will allow you to filter by port (protocol) rather than simply source address (it would also allow you to filter by destination, but that's not a factor in this case):

lab2651rtr# conf t
lab2651rtr(config)# access-list 101 deny tcp any any eq 3128
lab2651rtr(config)# access-list 101 deny udp any any eq 3128
lab2651rtr(config)# access-list 101 permit tcp any any
lab2651rtr(config)# access-list 101 permit udp any any
lab2651rtr(config)# int fa0/0.1
lab2651rtr(config-subif)# no ip access-group 1 in
lab2651rtr(config-subif)# ip access-group 101 out
lab2651rtr(config-subif)# int fa0/0.2
lab2651rtr(config-subif)# no ip access-group 1 in
lab2651rtr(config-subif)# ip access-group 101 out
lab2651rtr(config-subif)# int fa0/0.3
lab2651rtr(config-subif)# no ip access-group 1 in
lab2651rtr(config-subif)# ip access-group 101 out
lab2651rtr(config-subif)# int fa0/0.4
lab2651rtr(config-subif)# no ip access-group 1 in
lab2651rtr(config-subif)# ip access-group 101 out
lab2651rtr(config-subif)# int fa0/0.5
lab2651rtr(config-subif)# no ip access-group 1 in
lab2651rtr(config-subif)# ip access-group 101 out
lab2651rtr(config-subif)# int fa0/0.6
lab2651rtr(config-subif)# no ip access-group 1 in
lab2651rtr(config-subif)# ip access-group 101 out
lab2651rtr(config-subif)# int fa0/0.7
lab2651rtr(config-subif)# no ip access-group 1 in
lab2651rtr(config-subif)# ip access-group 101 out
lab2651rtr(config-subif)# exit
lab2651rtr(config)# exit


Now, to make sure the ACL is working:

lab2651rtr# sho ip access-lists
Extended IP access list 101
    deny tcp any any eq 3128 (27 matches)
    deny udp any any eq 3128
    permit tcp any any
    permit udp any any (15 matches)
lab2651rtr#


You verify with tcpdump that the problem has been temporarily minimized, and then spend a while analyzing the data from tcpdump to try to identify the source of the network traffic. You notice that all of the source IP addresses are from a single subnet, 192.168.1.112/28, which is in use by Sales (VLAN 6), so you look at MAC addresses on the first switch to see who is on-line:

lab2924a# sho mac vlan 6
<...snip...>
Destination Address Address Type VLAN Destination Port
------------------- ------------ ---- --------------------
0005.9bbe.cf60      Dynamic         6 FastEthernet0/3
0024.e8cb.f9e9      Dynamic         6 FastEthernet0/1
lab2924a#


From this, you know that the compromised host is on the second switch, since Fa0/3 is the port to the file server, and Fa0/1 is one of the ports in LAG group to the second switch. You telnet to the second switch and repeat the same command:
< lab2924b#sho mac vlan 6
<...snip...>
Destination Address Address Type VLAN Destination Port
------------------- ------------ ---- --------------------
0003.e3e4.f881      Dynamic         6 FastEthernet0/2
0005.9bbe.cf60      Dynamic         6 FastEthernet0/2
0024.e8cb.f9e9      Dynamic         6 FastEthernet0/14
lab2924b#


Fa0/2 is the second port in the LAG group from the first switch, but the third line contains the information you needed. Fa0/14 is an access port to the Sales group, and therefore, that's almost certainly where the offending traffic is coming from. You shut down the port on the switch, then remove the ACL from VLAN 3 (IT) on the router to see if you are still seeing the port scans:

lab2924b(config)# int fa0/14
lab2924b(config-if)# shut


...and on the router:

lab2651rtr# conf t
lab2651rtr(config)#int fa0/0.3
lab2651rtr(config-subif)#no ip access-group 101 out
lab2651rtr(config-subif)#


Satisfied that you have stopped the flood of malicious traffic, you head to the Sales department to try to find out whose computer was compromised.

Note: Be aware that, while you can create a crude firewall with the ACL features on Cisco routers, you should try to filter as little as possible. Cisco routers, or for that matter, most comparable vendors' routers (with the exception of ImageStream routers which are actually full-fledged PC's with multiple Ethernet interfaces, running a Linux kernel and using iptables for firewalling) simply don't have enough horsepower to do complex firewalling. As an extreme example, I once tried to use a Cisco 806 SOHO router (admittedly, a very low-end box with minimal processing power) as a DSL router/firewall. It worked well as a router, but once I started applying ACL's to the 806, I noticed a definite slow-down on throughput. A cheap, $35 Linksys DSL router/firewall had more than enough CPU to route, NAT and firewall, but the 806 simply couldn't keep up with the ACLs required to NAT and firewall the DSL connection.