Cisco Advanced Routing Lab: IP Service Level Agreements

So you've taken all of the pieces that we've discussed so far -- Multicast, QoS, etc. -- and built a solid network for your customers. How do you prove to your customers that they are getting the quality that they are paying for? Arguably more important, how do you keep tabs on what's happening on your network so that you can fix any problems before your customer's begin complaining about them? As I'm sure you've guessed, there is a solution that is built in to most reasonably modern versions of Cisco IOS, and that solution is called "ip sla" (service level agreements). To mock up this lab, I had to upgrade my virtual routers slightly by adding IOS images for Cisco 7200 routers, as the IOS versions I had for the 3640 and 2651 routers I've been modeling don't support the ip sla feature.

Basically, the ip sla feature sends traffic across the network to measure packet loss, latency, jitter, etc. What's nice is that ip sla's allow you to craft the packets (or datagrams) that will be sent to mimic whatever kind of traffic you want to monitor. For example, you can set the TOS markings to monitor priority traffic across the network. You can tell the IP SLA engine to send data inside the G729 codec to measure how voice traffic is affected by the network. You can specify how often to send this traffic across the network, and how many packets to send at a time. It's kind of like the ping tool on steroids ;)

Suppose you have the following network...:


Because we have clients connected to R5, and a streaming server connected through R3, we want to make sure that jitter and latency is within acceptable limits between these two routers. We could build a Smokeping server (and in fact, Smokeping is running on the server labeled "CentOS6"), but Smokeping just sends standard ICMP packets. What if we want to measure latency of UDP traffic with specific QoS markings? Smokeping can't do that. Fortunately, this is exactly the kind of problem that the ip sla feature was designed to address, so we will enable ip sla on R3 and R5:
R3#sho run | begin ip sla
ip sla responder
ip sla 10015
udp-jitter 10.254.254.5 32766 source-ip 10.254.254.3 num-packets 20
tos 184
frequency 300
ip sla schedule 10015 life forever start-time now
ip sla 10025
udp-echo 10.254.254.5 32764 source-ip 10.254.254.3
ip sla schedule 10025 life forever start-time now

...and on R5:
R5#sho run | begin ip sla
ip sla responder
ip sla 10013
udp-jitter 10.254.254.3 32766 source-ip 10.254.254.5 num-packets 20
tos 184
frequency 300
ip sla schedule 10013 life forever start-time now
ip sla 10023
udp-echo 10.254.254.3 32764 source-ip 10.254.254.5
ip sla schedule 10023 life forever start-time now

The first line, "ip sla responder" tells the router to respond to IP SLA traffic. Next, we create an IP SLA process to send traffic to remote destination. The 5-digit identifier is simply a unique identifier on that router; in my case, the first three digits were pretty much arbitrary, the fourth digit I used to separate the UDP jitter test from the UDP echo test, and the last digit was taken from the router number (i.e., xxxx3 was a query for R3, xxxx5 was a query for R5, etc.). This is perhaps overkill with the examples I've shown, but on R1, for example, I set up similar IP SLA probes for R2-R5. Next, I specified what kind of test I wanted to run ("udp-jitter, udp-echo -- there are many, many more options to choose from), the destination IP address, a random port number to probe on the destination, the source IP address to use in the outbound datagram, and the optional "num-packets" flag to specify how many packets (grrr...not to be pedantic, but UDP -> datagram, Cisco! In technical fields, precise terminology often is important </rant>). In the UDP jitter test, the next line specifies that we want to apply a TOS marking so that this packet receives the same kind of priority handling that VoIP traffic (DSCP EF) would receive. After that, I specified a frequency of one test every 5 minutes for the UDP jitter test (I left the UDP echo test at the default settings). Finally, I specified the schedule for the IP SLA probes, start now and run indefinitely. However, if you wanted to monitor performance over a weekend or for a short period during backups at night, you could specify different values:
R3(config)#ip sla schedule 10023 life ?
  <0-2147483647>  Life seconds (default 3600)
  forever         continue running forever

R3(config)#ip sla schedule 10023 life 18600 ?
  ageout      How long to keep this Entry when inactive
  recurring   Probe to be scheduled automatically every day
  start-time  When to start this entry
  <cr>

R3(config)#ip sla schedule 10023 life 18600 recurring ?
  ageout      How long to keep this Entry when inactive
  start-time  When to start this entry
  <cr>

R3(config)#ip sla schedule 10023 life 18600 recurring start-time ?
  after     Start after a certain amount of time from now
  hh:mm     Start time (hh:mm)
  hh:mm:ss  Start time (hh:mm:ss)
  now       Start now
  pending   Start pending

R3(config)#ip sla schedule 10023 life 18600 recurring start-time 23:00 ?
  <1-31>  Day of the month
  MONTH   Month of the year
  ageout  How long to keep this Entry when inactive
  <cr>

R3(config)#$dule 10023 life 18600 recurring start-time 23:00 November 1 ?
  ageout  How long to keep this Entry when inactive
  <cr>

Cool. So...what happens when you enable IP SLA monitoring on your routers?

R3#sho ip sla statistics
IPSLAs Latest Operation Statistics

IPSLA operation id: 10015
Type of operation: udp-jitter
        Latest RTT: 17 milliseconds
Latest operation start time: *00:20:17.115 AKDT Wed Oct 26 2016
Latest operation return code: OK
RTT Values:
        Number Of RTT: 19               RTT Min/Avg/Max: 1/17/51 milliseconds
Latency one-way time:
        Number of Latency one-way Samples: 0
        Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds
        Destination to Source Latency one way Min/Avg/Max: 0/0/0 milliseconds
Jitter Time:
        Number of SD Jitter Samples: 17
        Number of DS Jitter Samples: 18
        Source to Destination Jitter Min/Avg/Max: 0/11/39 milliseconds
        Destination to Source Jitter Min/Avg/Max: 0/8/22 milliseconds
Packet Loss Values:
        Loss Source to Destination: 1           Loss Destination to Source: 0
        Out Of Sequence: 0      Tail Drop: 0
        Packet Late Arrival: 0  Packet Skipped: 0
Voice Score Values:
        Calculated Planning Impairment Factor (ICPIF): 0
        Mean Opinion Score (MOS): 0
Number of successes: 6
Number of failures: 0
Operation time to live: Forever



IPSLA operation id: 10025
Type of operation: udp-echo
        Latest RTT: 8 milliseconds
Latest operation start time: *00:24:23.535 AKDT Wed Oct 26 2016
Latest operation return code: OK
Number of successes: 58
Number of failures: 0
Operation time to live: Forever


R3#

That is at once a lot of information and not nearly as much information as you might hope to receive. While there is a lot of output showing quite a bit of detail, this is only the results of the last test. If you want any kind of historical perspective, you will need to put in some additional work. For example, I believe that you can have the results of the IP SLA monitoring logged to a syslog server (Splunk, perhaps?), although I haven't tried that. Alternatively, you can use SNMP to pull the results, although finding the right OIDs to query is a bit of a chore.

For now, let's focus on the CLI output. First, the UDP jitter test: the executive summary of the IP SLA monitoring is presented at the very end of the output:
Voice Score Values:
        Calculated Planning Impairment Factor (ICPIF): 0
        Mean Opinion Score (MOS): 0
Number of successes: 6
Number of failures: 0

A quick Google search provides a little insight into this output. Both ICPIF and MOS attempt to provide a metric for evaluating how suitable a network is for Voice over IP telephony, with lower scores being better than higher scores.
ICPIF values numerically less than 20 are generally considered "adequate."

MOS, on the other hand, is rated on a scale from 1 to 5, with 1 being very poor quality and 5 being very good quality. Note: In my output, the given MOS score was zero. Per Cisco, this means that the "...MOS data could not be generated for the given operation."

That should be enough to get you started with IP SLA monitoring on IOS. It's definitely a feature worth checking out if you will be using Cisco routers on any kind of production network.

JNCIA Lesson 1: Basic LAN Configuration

Congratulations! Because of your experience with networking in general, and Juniper equipment in particular, you have just landed a network administrator job with a new, small company. You will be responsible for building the network infrastructure for the new company. Your first decision in this role was to purchase two brand new Juniper EX3200 Layer-3 switches to use for both LAN switching and to route traffic to and from your Internet Service Provider.

First, of course, is racking the equipment and connecting the associated cables -- power, management console, etc. That is a breeze, and in less than half an hour, your new Juniper switches are happily humming away. You connect a serial cable from your laptop to the console port (NOT the "Management" port, however -- that's an Ethernet port for management access!), and are greeted by the Juniper's shell prompt:

root@:RE:0%

You type a few Unix commands:

root@:RE:0% ls
.cshrc  .history  .login  .profile
root@:RE:0% uname -a
JUNOS ex3200 11.4R7.5 JUNOS 11.4R7.5 #0: 2013-03-01 10:14:08 UTC  builder@evenath.juniper.net:/volume/build/junos/11.4/release/11.4R7.5/obj-powerpc/bsd/kernels/JUNIPER-EX/kernel powerpc
root@:RE:0%
Notice that it's running a BSD kernel (FreeBSD, to be exact). You want to begin configuring the switch, so you execute the "cli" command to access JunOS:

root@:RE:0% cli
root@ex3200>


The switch comes with a very basic configuration...:

root@ex3200> show configuration
version 11.4R7.5;
system {
    host-name ex3200;
    syslog {
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
}
interfaces {
    ge-0/0/0 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/1 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/2 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/3 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/4 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/5 {
        unit 0 {
            family ethernet-switching;
        }
    ge-0/0/6 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/7 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/8 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/9 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/10 {
        unit 0 {
            family ethernet-switching; 
        }
    }
    ge-0/0/11 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/12 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/13 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/14 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/15 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/16 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/17 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/18 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/19 {
        unit 0 {
            family ethernet-switching;
    }
    ge-0/0/20 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/21 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/22 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/0/23 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/1/0 {
        unit 0 {
            family ethernet-switching;
        }
    }
    xe-0/1/0 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/1/1 {
        unit 0 {
            family ethernet-switching;
        }
    }
    xe-0/1/1 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/1/2 {
        unit 0 {
            family ethernet-switching;
        }
    }
    xe-0/1/2 {
        unit 0 {
            family ethernet-switching;
        }
    }
    ge-0/1/3 {
        unit 0 {
            family ethernet-switching;
        }
    }
}
protocols {
    igmp-snooping {
        vlan all;
    }
    rstp;
    lldp {
        interface all;
    }
    lldp-med {
        interface all;
    }
}
ethernet-switching-options {
    storm-control {
        interface all;
}
vlans {
    default {
        l3-interface vlan.0;
    }
}
poe {
    interface all;
}

root@ex3200>


With all of the indentations, curly braces and semi-colons, the configuration greatly resembles a Unix/Linux configuration file, rather than a typical network vendor's config file, you think to yourself. At first, it makes the config look a little long-winded, but it's pretty easy to read, once you get used to it.

However, the basic, default configuration is missing some very important elements that you will need to add. First, you decide to give the switch a more meaningful name. Since this will be the switch/router for the main office campus, you decide to change the name to main3200. You enter configuration mode and start to make the necessary changes:

root@ex3200> configure
Entering configuration mode

[edit]
root@ex3200# set system host-name main3200

[edit]
root@ex3200#


Notice that that didn't seem to change anything. Juniper doesn't actually make the changes live until you commit them, using -- of all things -- the "commit" command.:

root@ex3200# commit
commit complete

[edit]
root@main3200#


JunOS also includes a nifty feature to allow you to undo a change. When committing a configuration change, JunOS copies the existing configuration to an archive before writing the new config to storage. Consequently, if you make a change that breaks something, you can always roll back to an earlier configuration:

root@main3200# set system host-name oops

[edit]
root@main3200# commit
commit complete

[edit]
root@oops# rollback 1
load complete

[edit]
root@oops# commit
commit complete

[edit]
root@main3200#


There are a number of other cool features available when committing changes: you can tentatively commit the changes...:

root@main3200# set host-name mo3200

[edit system]
root@main3200# commit confirmed
commit confirmed will be automatically rolled back in 10 minutes unless confirmed
commit complete

# commit confirmed will be rolled back in 10 minutes
[edit system]
root@mo3200#
Broadcast Message from root@main3200
(no tty) at 8:37 AKDT...

Commit was not confirmed; automatic rollback complete.


Cool...this could be handy when configuring remote systems, where a botched config could mean downtime while you drive across town [Edit: or fly across the country, which would be the case in my real day job]. Use "commit confirmed", and if the new config doesn't work, it will automatically roll back to the previous version after the time-out period, and you can try again.

To confirm the commit, just type "commit" again:

...
# commit confirmed will be rolled back in 10 minutes
[edit]
root@main3200# commit
commit complete

[edit]
root@main3200#


You can also simply view the changes to the active configuration...:

root@main3200# show | compare rollback 1
[edit system]
- host-name mo3200;
+ host-name main3200;

[edit]
root@main3200#
...or check the config changes without actually making them active...:

root@main3200# commit check
configuration check succeeds

[edit]
root@main3200#


Another point about JunOS that's worth noting: if you look at the "system" portion of the default config...:

root@ex3200> show configuration
version 11.4R7.5;
system {
    host-name ex3200;
    syslog {
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
}


Notice the hierarchy for the host-name setting:

system | host-name

When we changed the hostname, we did so with the "set system host-name main3200" command. However, sometimes, you will be making a number of changes under a hierarchy, and it can be a bit of a chore repeating all of the hierarchy for every change you want to make. Consequently, you can tell JunOS that you want to work within a particular portion of the hierarchy, and to assume that all of the commands you are running are to be taken in the context of that hierarchy. For example, these two commands are identical to the Juniper:

root@ex3200# set system host-name main3200

[edit]
root@ex3200# commit
commit complete

[edit]
root@main3200#


...and...:

root@ex3200# edit system

[edit system]
root@ex3200# set host-name main3200

[edit system]
root@ex3200# commit
commit complete

[edit system]
root@main3200#


Think of it as being similar to the difference between...:

me@myllt:~$ cp /var/www/localhost/htdocs/gallery/images/thumbnails/some.jpg /var/www/localhost/htdocs/gallery/images/thumbnails/some-other.jpg


...and...:

me@myllt~$ cd /var/www/localhost/htdocs/gallery/images/thumbnails/
me@myllt:/var/www/localhost/htdocs/gallery/images/thumbnails$ cp some.jpg some-other.jpg


It's the exact same thing: you are setting the position within the hierarchy in which you want to work. In the trivial examples above, it might not seem like the "edit..." syntax saves much effort, but suppose you want to make a number of changes to an Ethernet port -- say, hard-code speed and duplex and set a description:

root@main3200# edit interfaces ge-0/0/4

[edit interfaces ge-0/0/4]
root@main3200# set description "Legacy Billing Server"

[edit interfaces ge-0/0/4]
root@main3200# set ether-options speed 10m

[edit interfaces ge-0/0/4]
root@main3200# set ether-options link-mode half-duplex

[edit interfaces ge-0/0/4]
root@main3200# set ether-options auto-negotiation no-flow-control

[edit interfaces ge-0/0/4]
root@main3200# commit check
configuration check succeeds

[edit interfaces ge-0/0/4]
root@main3200# commit
commit complete

[edit interfaces ge-0/0/4]
root@main3200#


If you hadn't used the "edit interface ge-0/0/4" command, you would have had to type this instead:

root@main3200# set interfaces ge-0/0/4 description "Legacy Billing Server"

[edit]
root@main3200# set interfaces ge-0/0/4 ether-options speed 10m

[edit]
root@main3200# set interfaces ge-0/0/4 ether-options link-mode half-duplex

[edit]
root@main3200# set interfaces ge-0/0/4 ether-options auto-negotiation no-flow-control

[edit]
root@main3200# commit check
configuration check succeeds

[edit]
root@main3200# commit
commit complete

[edit]
root@main3200#


Let's configure some other basic settings on the switch:

root@main3200# set system domain-name example.com

[edit]
root@main3200# set system root-authentication plain-text-password
New password:
Retype new password:

[edit]
root@main3200# set system services ssh protocol-version v2

[edit]
root@main3200#


With those commands, we have set the domain name, set a root password and enabled SSH access. However, you will probably have a hard time using SSH to connect to the switch, since the switch does not have a management IP address set. Let's address that issue now:

root@main3200# set interfaces vlan unit 0 family inet address 192.168.1.1/24

[edit]
root@main3200# commit
commit confirmed

[edit]
root@main3200#


With that, you should have a minimal working configuration on your switch. Each of the 24 Ethernet ports is enabled and will allow hosts to communicate with each other, and you can SSH to the switch for management access at 192.168.1.1.

That's a good start on a basic, real-world configuration for a LAN switch, but there are a couple of other things to do, too. First, in a one-person shop, it might be fine to simply use the root account for all configuration. However, if there are (or will be) multiple admins managing network devices, then it might be wise to configure separate logins for each user. Let's create three user accounts on the EX-3200 -- an admin account for you, an admin account for your assistant George and a read-only account for your Network Operations Center (NOC):

root@main3200# set system login class noc permissions interface

[edit]
root@main3200# set system login class noc permissions view

[edit]
root@main3200# set system login user nocuser class noc authentication plain-text-password
New password:
Retype new password:

[edit]
root@main3200# set system login user mike class super-user authentication plain-text-password
New password:
Retype new password:

[edit]
root@main3200# set system login user george class super-user authentication plain-text-password
New password:
Retype new password:

[edit]
root@main3200# commit
commit complete

[edit]
root@main3200#


You can also configure TACACS or RADIUS authentication:

root@main3200# set system authentication-order [radius password]

[edit]
root@main3200# set system radius-server 192.168.1.17 secret myradiussecretpw

[edit]
root@main3200# commit
commit confirmed

[edit]
root@main3200#


Next, you want the switch to forward logs to a syslog server, located at 192.168.1.20:

root@main3200# set system syslog host 192.168.1.20 any any

[edit]
root@main3200#


If you wanted, you could also send firewall logs to one server, or send warning (or higher) messages to a third server:

root@main3200# set system syslog host 192.168.1.21 firewall any

[edit]
root@main3200# set system syslog host 192.168.1.20 any warning


[edit]
root@main3200# commit
commit confirmed

[edit]
root@main3200#


Next, we will configure the switch to set its internal clock with a local NTP server...:

root@console9# set system ntp server 192.168.1.23

[edit]
root@main3200# commit
commit confirmed

[edit]
root@main3200#


...and we will configure SNMP on the switch:

root@main3200# set system location building "123 A Street" country-code US postal-code 12345 floor 2 rack 36

[edit]
root@main3200# set snmp community my-ro-community-string authorization read-only

[edit]
root@main3200# set snmp community my-rw-community-string authorization read-write

[edit]
root@main3200# root@console9# set snmp contact miswork@uui-alaska.com

[edit]
root@main3200# set snmp location Anchorage

[edit]
root@main3200# commit
commit complete

[edit]
root@main3200#


That leaves one last, very important step before we are done: we need to create a "rescue configuration." The rescue configuration is a "known good" configuration that provides basic connectivity to the switch, if the current configuration is deleted or lost. To be honest, I'm not sure I see the point of the rescue configuration, since JunOS keeps an archive of past configs (perhaps, just in case you have moved/re-addressed the switch?), but...we'll go ahead and create one from our current configuration. First, notice that there is no rescue configuration yet:

root@main3200> file list /config/

/config/:
.snap/
db /
juniper.conf.1.gz
juniper.conf.2.gz
juniper.conf.3.gz
juniper.conf.gz
ssh_host_dsa_key
ssh_host_dsa_key.pub
ssh_host_ecdsa_key
ssh_host_ecdsa_key.pub
ssh_host_key
ssh_host_key.pub
ssh_host_rsa_key
ssh_host_rsa_key.pub
usage.db
vchassis/

[edit]
root@main3200>


We'll save the current configuration as the rescue configuration, and show the contents of the /config directory again:

root@main3200> request system configuration rescue save

root@main3200> file list /config/

/config/:
.snap/
db /
juniper.conf.1.gz
juniper.conf.2.gz
juniper.conf.3.gz
juniper.conf.gz
rescue.conf.gz
ssh_host_dsa_key
ssh_host_dsa_key.pub
ssh_host_ecdsa_key
ssh_host_ecdsa_key.pub
ssh_host_key
ssh_host_key.pub
ssh_host_rsa_key
ssh_host_rsa_key.pub
usage.db
vchassis/

root@main3200>


Notice the new, bold-face file in the listing [edit: it's not actually bold-face on the switch; I highlighted it so it would stand out]? That's the rescue configuration. If you need to remove the rescue configuration, the "request system configuration rescue delete" command will do it. If you reboot the switch without an active configuration, the switch will boot with the rescue configuration, instead. Once the switch has booted, you can commit to write a new active configuration to the file system.