Cisco Advanced Routing Lab: IP Service Level Agreements

So you've taken all of the pieces that we've discussed so far -- Multicast, QoS, etc. -- and built a solid network for your customers. How do you prove to your customers that they are getting the quality that they are paying for? Arguably more important, how do you keep tabs on what's happening on your network so that you can fix any problems before your customer's begin complaining about them? As I'm sure you've guessed, there is a solution that is built in to most reasonably modern versions of Cisco IOS, and that solution is called "ip sla" (service level agreements). To mock up this lab, I had to upgrade my virtual routers slightly by adding IOS images for Cisco 7200 routers, as the IOS versions I had for the 3640 and 2651 routers I've been modeling don't support the ip sla feature.

Basically, the ip sla feature sends traffic across the network to measure packet loss, latency, jitter, etc. What's nice is that ip sla's allow you to craft the packets (or datagrams) that will be sent to mimic whatever kind of traffic you want to monitor. For example, you can set the TOS markings to monitor priority traffic across the network. You can tell the IP SLA engine to send data inside the G729 codec to measure how voice traffic is affected by the network. You can specify how often to send this traffic across the network, and how many packets to send at a time. It's kind of like the ping tool on steroids ;)

Suppose you have the following network...:


Because we have clients connected to R5, and a streaming server connected through R3, we want to make sure that jitter and latency is within acceptable limits between these two routers. We could build a Smokeping server (and in fact, Smokeping is running on the server labeled "CentOS6"), but Smokeping just sends standard ICMP packets. What if we want to measure latency of UDP traffic with specific QoS markings? Smokeping can't do that. Fortunately, this is exactly the kind of problem that the ip sla feature was designed to address, so we will enable ip sla on R3 and R5:
R3#sho run | begin ip sla
ip sla responder
ip sla 10015
udp-jitter 10.254.254.5 32766 source-ip 10.254.254.3 num-packets 20
tos 184
frequency 300
ip sla schedule 10015 life forever start-time now
ip sla 10025
udp-echo 10.254.254.5 32764 source-ip 10.254.254.3
ip sla schedule 10025 life forever start-time now

...and on R5:
R5#sho run | begin ip sla
ip sla responder
ip sla 10013
udp-jitter 10.254.254.3 32766 source-ip 10.254.254.5 num-packets 20
tos 184
frequency 300
ip sla schedule 10013 life forever start-time now
ip sla 10023
udp-echo 10.254.254.3 32764 source-ip 10.254.254.5
ip sla schedule 10023 life forever start-time now

The first line, "ip sla responder" tells the router to respond to IP SLA traffic. Next, we create an IP SLA process to send traffic to remote destination. The 5-digit identifier is simply a unique identifier on that router; in my case, the first three digits were pretty much arbitrary, the fourth digit I used to separate the UDP jitter test from the UDP echo test, and the last digit was taken from the router number (i.e., xxxx3 was a query for R3, xxxx5 was a query for R5, etc.). This is perhaps overkill with the examples I've shown, but on R1, for example, I set up similar IP SLA probes for R2-R5. Next, I specified what kind of test I wanted to run ("udp-jitter, udp-echo -- there are many, many more options to choose from), the destination IP address, a random port number to probe on the destination, the source IP address to use in the outbound datagram, and the optional "num-packets" flag to specify how many packets (grrr...not to be pedantic, but UDP -> datagram, Cisco! In technical fields, precise terminology often is important </rant>). In the UDP jitter test, the next line specifies that we want to apply a TOS marking so that this packet receives the same kind of priority handling that VoIP traffic (DSCP EF) would receive. After that, I specified a frequency of one test every 5 minutes for the UDP jitter test (I left the UDP echo test at the default settings). Finally, I specified the schedule for the IP SLA probes, start now and run indefinitely. However, if you wanted to monitor performance over a weekend or for a short period during backups at night, you could specify different values:
R3(config)#ip sla schedule 10023 life ?
  <0-2147483647>  Life seconds (default 3600)
  forever         continue running forever

R3(config)#ip sla schedule 10023 life 18600 ?
  ageout      How long to keep this Entry when inactive
  recurring   Probe to be scheduled automatically every day
  start-time  When to start this entry
  <cr>

R3(config)#ip sla schedule 10023 life 18600 recurring ?
  ageout      How long to keep this Entry when inactive
  start-time  When to start this entry
  <cr>

R3(config)#ip sla schedule 10023 life 18600 recurring start-time ?
  after     Start after a certain amount of time from now
  hh:mm     Start time (hh:mm)
  hh:mm:ss  Start time (hh:mm:ss)
  now       Start now
  pending   Start pending

R3(config)#ip sla schedule 10023 life 18600 recurring start-time 23:00 ?
  <1-31>  Day of the month
  MONTH   Month of the year
  ageout  How long to keep this Entry when inactive
  <cr>

R3(config)#$dule 10023 life 18600 recurring start-time 23:00 November 1 ?
  ageout  How long to keep this Entry when inactive
  <cr>

Cool. So...what happens when you enable IP SLA monitoring on your routers?

R3#sho ip sla statistics
IPSLAs Latest Operation Statistics

IPSLA operation id: 10015
Type of operation: udp-jitter
        Latest RTT: 17 milliseconds
Latest operation start time: *00:20:17.115 AKDT Wed Oct 26 2016
Latest operation return code: OK
RTT Values:
        Number Of RTT: 19               RTT Min/Avg/Max: 1/17/51 milliseconds
Latency one-way time:
        Number of Latency one-way Samples: 0
        Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds
        Destination to Source Latency one way Min/Avg/Max: 0/0/0 milliseconds
Jitter Time:
        Number of SD Jitter Samples: 17
        Number of DS Jitter Samples: 18
        Source to Destination Jitter Min/Avg/Max: 0/11/39 milliseconds
        Destination to Source Jitter Min/Avg/Max: 0/8/22 milliseconds
Packet Loss Values:
        Loss Source to Destination: 1           Loss Destination to Source: 0
        Out Of Sequence: 0      Tail Drop: 0
        Packet Late Arrival: 0  Packet Skipped: 0
Voice Score Values:
        Calculated Planning Impairment Factor (ICPIF): 0
        Mean Opinion Score (MOS): 0
Number of successes: 6
Number of failures: 0
Operation time to live: Forever



IPSLA operation id: 10025
Type of operation: udp-echo
        Latest RTT: 8 milliseconds
Latest operation start time: *00:24:23.535 AKDT Wed Oct 26 2016
Latest operation return code: OK
Number of successes: 58
Number of failures: 0
Operation time to live: Forever


R3#

That is at once a lot of information and not nearly as much information as you might hope to receive. While there is a lot of output showing quite a bit of detail, this is only the results of the last test. If you want any kind of historical perspective, you will need to put in some additional work. For example, I believe that you can have the results of the IP SLA monitoring logged to a syslog server (Splunk, perhaps?), although I haven't tried that. Alternatively, you can use SNMP to pull the results, although finding the right OIDs to query is a bit of a chore.

For now, let's focus on the CLI output. First, the UDP jitter test: the executive summary of the IP SLA monitoring is presented at the very end of the output:
Voice Score Values:
        Calculated Planning Impairment Factor (ICPIF): 0
        Mean Opinion Score (MOS): 0
Number of successes: 6
Number of failures: 0

A quick Google search provides a little insight into this output. Both ICPIF and MOS attempt to provide a metric for evaluating how suitable a network is for Voice over IP telephony, with lower scores being better than higher scores.
ICPIF values numerically less than 20 are generally considered "adequate."

MOS, on the other hand, is rated on a scale from 1 to 5, with 1 being very poor quality and 5 being very good quality. Note: In my output, the given MOS score was zero. Per Cisco, this means that the "...MOS data could not be generated for the given operation."

That should be enough to get you started with IP SLA monitoring on IOS. It's definitely a feature worth checking out if you will be using Cisco routers on any kind of production network.

Cisco Intro to QoS and CoS, Part 4 -- Shape Average vs. Bandwidth in Policy Maps

If you've poked around at all with the policy-map statement, you've probably noticed that there are several options for configuring how much bandwidth to allocate to different classes within the QoS config. Let's look at two of the options, shape average and bandwidth.

We'll use the following network...:


...and the following configuration:
ip access-list extended PRIORITY-IN
 permit ip 192.168.1.0 0.0.0.255 any
 permit ip any 192.168.1.0 0.0.0.255
 deny ip any any
ip access-list extended SCAVENGER-IN
 permit ip 192.168.2.0 0.0.0.255 any
 permit ip any 192.168.2.0 0.0.0.255
 deny ip any any
!
class-map match-any PRIORITY
 match dscp af41
class-map match-any SCAVENGER
 match dscp cs1
!
class-map match-any PRIORITY-IN
 match access-group name PRIORITY-IN
class-map match-any SCAVENGER-IN
 match access-group name SCAVENGER-IN
!
policy-map REMARK_ALL
 class PRIORITY-IN
  set dscp af41
 class SCAVENGER-IN
  set dscp cs1
policy-map EGRESS-SHAPER
 class PRIORITY
  shape average percent 70
 class SCAVENGER
  shape average percent 5
policy-map EGRESS-BANDWIDTH
 class PRIORITY
  bandwidth percent 70
 class SCAVENGER
  bandwidth percent 5
!

For my first test, I set up the policy-map, "EGRESS-SHAPER," on interface eth3/0 on both routers, and set up the "REMARK_ALL" service policy on the ingress interfaces (fa0/0 on both routers, and fa1/0 on R1):
interface FastEthernet0/0
 service-policy input REMARK_ALL
!
interface FastEthernet1/0
 service-policy input REMARK_ALL
!
interface Ethernet3/0
 service-policy output EGRESS-SHAPER
!

Then, I set up iperf as a service on host "Knoppix" and ran iperf as a client on both Knoppix Clones, using the flags "-i 1 -t 120," finding that Knoppix Clone 1 could send data at a little over 5Mbps, and Knoppix Clone 2 sent data at about 500Kbps. I then re-ran the test using the service policy "EGRESS-BANDWIDTH," where I saw a transfer speed of just over 4Mbps for Knoppix Clone 1 and about 1 1/3Mbps for Knoppix Clone 2. That surprised me; I didn't really understand why my bandwidth decreased in class PRIORITY (Knoppix Clone 1) and increased in class SCAVENGER (Knoppix Clone 2). To understand what was happening, I did some more tests, using both TCP and UDP transfers ("-u -b 10M" for the flags to set a UDP test at up to 10Mbps), and with both hosts in contention for bandwidth as well as with the hosts transferring data one at a time (i.e., run iperf on Knoppix Host 1, and then running it on Knoppix Clone 2 after the test completed on Knoppix Clone 1). Here are the results from all of the tests:

Congested:
  TCP:
    Using EGRESS-SHAPER:
      192.168.1.x: 5.15Mbps
      192.168.2.x: 0.48Mbps

    Using EGRESS-BANDWIDTH:
      192.168.1.x: 4.16Mbps
      192.168.2.x: 1.36Mbps


  UDP:
    Using EGRESS-SHAPER:
      192.168.1.x: 3.07Mbps
      192.168.2.x: 0.16Mbps

    Using EGRESS-BANDWIDTH:
      192.168.1.x: 2.8Mbps
      192.168.2.x: 1.26Mbps


Uncongested:
  TCP:
    Using EGRESS-SHAPER:
      192.168.1.x: 5.76Mbps
      192.168.2.x: 0.50Mbps

    Using EGRESS-BANDWIDTH:
      192.168.1.x: 5.52Mbps
      192.168.2.x: 5.51Mbps

  UDP:
    Using EGRESS-SHAPER:
      192.168.1.x: 5.82Mbps
      192.168.2.x: 0.47Mbps

    Using EGRESS-BANDWIDTH:
      192.168.1.x: 6.03Mbps
      192.168.2.x: 6.14Mbps

What's happening here is that "shape average" tells the router to strictly allocate the specified amount of bandwidth to the various traffic classes. Even if more bandwidth is available through the interface, the allocated bandwidth is all that the traffic class will receive. When using the "bandwidth" statement, however, the router will allow the traffic classes to use all of the available bandwidth. Since class PRIORITY takes, well, priority over class SCAVENGER, when there is traffic in both traffic classes, class SCAVENGER will only receive the allocated bandwidth. However, when there is little or no priority traffic on the interface, the router will allow class SCAVENGER to "borrow" bandwidth from the other traffic classes in the service policy (in this case, class PRIORITY). Consequently, when there was no contention for bandwidth, class SCAVENGER can get the full data rate on the interface*.

*Q: Wait a minute! An Ethernet port runs at 10Mbps, but these results are showing about 6Mbps. What happened to the other 4Mbps?
A: Keep in mind that an Ethernet port runs at half duplex, meaning that the same copper wire is used for both transmit and receive. Therefore, the 10Mbps bandwidth available on the Ethernet port has to be shared between transmit and receive.**

**Q: Okay, so that explains why it's not 10Mbps, but in that case, shouldn't the data rate be lower, like 5Mbps?
A: If both sides were transmitting data at line rate, then yes. However, in this case the flow of data is mostly one-sided, especially with the UDP traffic, where there is no ACK being sent back to the two Knoppix Clones.***

***Q: In that case, should the data rate be higher?
A: The devil is in the details, and there are a lot of details to consider ;) "Customer" data isn't all that is flowing between R1 and R2. R1 and R2 are also sending their own data back and forth, for example, OSPF updates, CDP updates, etc. Consequently, while the flow of data between R1 and R2 is mostly one-sided, it's not completely one sided. If you REALLY want a good understanding of what's happening, replace R2 with the Knoppix host and run wireshark to capture the packets during a test. I'll leave that as an exercise for the reader ;)

Cisco Intro to QoS and CoS, Part 3 -- Classifying Traffic with NBAR

In our QoS labs so far, we've had to find some way to identify traffic, usually by creating an ACL to match packets against some criteria, such as source/destination port, source/destination IP address, or protocol (TCP/UDP/ICMP), etc. However, Cisco offers another way to match traffic: NBAR, or "Network Based Application Recognition. I created the following network in GNS3 to start playing with NBAR:

To enable NBAR on a router, you first need to turn on CEF, and then you need to enable NBAR on any interface that will have a service policy to mark and classify traffic. In this network, I am using an NM-16ESW switchport module in slot 0, so while Fa0/0 is technically the ingress port, I would enable NBAR on my VLAN interface (int VLAN10). If you are using a plain FastEthernet port rather than a switch module, then you would enable NBAR on the Fa port. In either case, here is how to do it:
R4(config)#ip cef
R4(config)#int vlan 10
R4(config-if)#ip nbar protocol-discovery

After enabling NBAR, we'll need to design our QoS schema. Let's start by identifying the types of traffic on our network, and by deciding what traffic will take priority over other traffic. I came up with the following (admittedly hokey) schema, ordered by priority:

1. EIGRP
2. HTTP (simulating voice/video with a streaming mp4 file)
3. Telnet
4. SNMP
5. SSH
6. ICMP

I then mapped this traffic to the following traffic classes:

Traffic	DSCP Value
EIGRP	CS6
HTTP	EF
Telnet	AF41
SNMP	CS3
SSH	AF21
ICMP	CS2

Cool! Let's start configuring the class-maps on R4, R5 and R6:
R4(config)#class-map match-all EIGRP
R4(config-cmap)#match protocol eigrp
R4(config-cmap)#class-map match-all HTTP
R4(config-cmap)# match protocol http
R4(config-cmap)#class-map match-all SSH
R4(config-cmap)# match protocol ssh
R4(config-cmap)#class-map match-all TELNET
R4(config-cmap)# match protocol telnet
R4(config-cmap)#class-map match-all SNMP
R4(config-cmap)# match protocol snmp
R4(config-cmap)#class-map match-all ICMP
R4(config-cmap)#match protocol icmp

...and now, the policy-map to use these classes:
R4(config-cmap)#policy-map REMARK_ALL
R4(config-pmap)#description policy-map to place traffic into the appropriate traffic class
R4(config-pmap)# class EIGRP
R4(config-pmap-c)# set ip dscp cs6
R4(config-pmap-c)# class HTTP
R4(config-pmap-c)# set ip dscp ef
R4(config-pmap-c)# class SSH
R4(config-pmap-c)# set ip dscp af21
R4(config-pmap-c)# class TELNET
R4(config-pmap-c)# set ip dscp af41
R4(config-pmap-c)# class SNMP
R4(config-pmap-c)# set ip dscp cs3
R4(config-pmap-c)# class ICMP
R4(config-pmap-c)# set ip dscp cs2

Now, just enable the policy map in your ingress interface (VLAN10 in this lab), and you are done:
R4(config-if)#int vlan10
R4(config-if)# service-policy input REMARK_ALL

Pretty easy, huh? To utilize NBAR to identify traffic from various protocols, all you have to do is use the "match protocol <protocol>" statement inside a class-map and NBAR will identify the traffic for you! Keep in mind, however, that NBAR is deep-packet inspection -- it isn't just a simple match for port number and IP address -- and therefore, this functionality comes at a cost of CPU cycles (unless you are using a more recent model switch/router that off-loads the packet inspection to a separate processor). Consequently, if a simple ACL will meet your needs, it MAY be less resource intensive to use the ACL, as we've done in earlier labs. However, if you have a higher-powered router, NBAR certainly makes identifying and classifying traffic much easier on the network admin.

Note:
This is just a really simple example of NBAR, and doesn't even begin to scratch the surface of what NBAR can do for you. For example, when matching the HTTP protocol, you can create multiple classes based upon the host in the HTTP request and you can even use a regex to pattern-match within the URL. You can also extend NBAR with external files to add support for protocols that are not already built in.

For the sake of completeness, here is the rest of the QoS configuration on R4: R4(config)#class-map match-any PRIORITY
R4(config-cmap)# match ip dscp ef
R4(config-cmap)#class-map match-any CONTROL
R4(config-cmap)# match ip dscp cs6
R4(config-cmap)#class-map match-any CRITICAL
R4(config-cmap)# match ip dscp af41
R4(config-cmap)# match ip dscp cs3
R4(config-cmap)#class-map match-any ROUTINE
R4(config-cmap)# match ip dscp cs2
R4(config-cmap)# match ip dscp af21
R4(config-cmap)#class-map match-any SCAVENGER
R4(config-cmap)# match ip dscp CS1
R4(config)#policy-map EDGE_CHILD
R4(config-pmap)# description Core links
R4(config-pmap)# class PRIORITY
R4(config-pmap-c)# priority percent 25
R4(config-pmap-c)# class CONTROL
R4(config-pmap-c)# bandwidth percent 10
R4(config-pmap-c)# class CRITICAL
R4(config-pmap-c)# bandwidth percent 15
R4(config-pmap-c)# random-detect dscp-based
R4(config-pmap-c)# class ROUTINE
R4(config-pmap-c)# bandwidth percent 10
R4(config-pmap-c)# random-detect dscp-based
R4(config-pmap-c)# class SCAVENGER
R4(config-pmap-c)# bandwidth percent 1
R4(config-pmap-c)# random-detect dscp-based
R4(config-pmap-c)# class class-default
R4(config-pmap-c)# fair-queue
R4(config-pmap-c)# random-detect dscp-based
R4(config-pmap-c)#!
R4(config-pmap-c)#policy-map EDGE2CORE
R4(config-pmap)# description Parent policy for Edge-to-Core links
R4(config-pmap)# class class-default
R4(config-pmap-c)# shape average 10000000
R4(config-pmap-c)# service-policy EDGE_CHILD
R4(config-pmap-c)#!
R4(config-if)#int fa1/0
R4(config-if)# service-policy output EDGE2CORE
R4(config-if)#!
R4(config-if)#int fa3/0
R4(config-if)# service-policy output EDGE2CORE
R4(config-if)#!