Advanced Cisco Networking: Policy-Based Routing (PBR)

Suppose you have a multi-homed network where you want to direct certain traffic out one interface, but other traffic out another. For example, maybe you want your VoIP traffic to use a moderately low bandwidth circuit, but with extremely strict QoS policies to provide low latency and jitter, while your bulk data traffic takes a higher bandwidth circuit with no QoS protection. Or, perhaps you have a small-bandwidth circuit for management traffic (one network I managed had an "overhead" T1 on an OC-3 microwave shot and we used the overhead T1 for out-of-band management). In any case, Policy-Based Routing (PBR) is a way for you to designate specific routes for certain traffic, based upon any of a number of characteristics -- basically, if you can match it with an access-list, you can use it to make PBR decisions.

Once again, we'll start with a network diagram:



I've stacked the deck pretty heavily in favor of the route R1-R3-R5 in this network: this route has Gig-E interfaces, while R1-R2-R4-R5 is only using FastEthernet interfaces, and there are fewer hops via R1-R3-R5 than R1-R2-R4-R5. As you can see in the screenshot below, this network design does, in fact, favor using R1-R3-R5 as the preferred route between the two hosts connected to R1 and the CentOS server connected to R5:



Now, let's set up policy-based routing so that system management traffic (Telnet, SSH and SNMP), as well as any traffic from the Sysmon CentOS server are routed through the lower-bandwidth -- but lower latency -- route across R2 and R4:

R1:
R1(config)#ip access-list extended matchSYSMON
R1(config-ext-nacl)#permit tcp any any eq 22
R1(config-ext-nacl)#permit tcp any any eq 23
R1(config-ext-nacl)#permit tcp any any eq 161
R1(config-ext-nacl)#permit ip host 192.168.1.4 any
R1(config-ext-nacl)#deny ip any any
R1(config-ext-nacl)#route-map SYSMON permit 10
R1(config-route-map)#match ip address matchSYSMON
R1(config-route-map)#set ip next-hop 10.1.2.2
R1(config-route-map)#int fa0/0
R1(config-if)#ip policy route-map SYSMON
R1(config-if)#exit

Now, let's try the traceroutes again:



Looks like it did before. However, from Sysmon, we see that we are taking a different route, just as expected:



Since the Knoppix host is simply using the default route, OSPF is using the higher-bandwidth, lower hop-count route. However, the router has identified the traffic originating on the Sysmon server as matching the routing policy that we added to R1, and therefore is steering this traffic through R2 and R4, just as we intended.

If you'll recall, our design goal in this scenario was to ensure that management traffic had low-latency queueing across the network. Suppose our service provider on the R1-R2-R4-R5 path had agreed to honor our QoS markings, but the provider on the R1-R3-R5 path re-marked everything with a lower priority. We can use the route-map we have created for the routing policy to also adjust our QoS markings for traffic going through R2 and R4:

R1(config)#route-map SYSMON permit 10
R1(config-route-map)#match ip address matchSYSMON
R1(config-route-map)#set ip next-hop 10.1.2.2
R1(config-route-map)#set ip precedence flash
R1(config-route-map)#exit
R1(config)#do sho run | section route-map SYSMON permit 10
route-map SYSMON permit 10
match ip address matchSYSMON
set ip precedence flash
set ip next-hop 10.1.2.2
R1(config)#

Cool! Suppose we wanted to do some traffic engineering across an MPLS network:

R1(config)#route-map SYSMON permit 10
R1(config-route-map)#match ip address matchSYSMON
R1(config-route-map)#set ip ?
...
  vrf         VRF name
R1(config-route-map)#

That's really cool! As you can see, policy-based routing is a very powerful tool, allowing you to do a lot of traffic manipulation to optimize your network and traffic flows.

At this point, those of you who are paying attention ;) will be thinking to yourself, "That's great, but what happens if we lose the next-hop router specified in our routing policy?" That is a great question, and with the configuration shown here, your traffic will be dropped on the floor. That's hardly optimal, but as I'm sure you've suspected, there is a solution to this problem...which we'll cover in a later lesson.

Cisco Advanced Routing Lab: IP Service Level Agreements

So you've taken all of the pieces that we've discussed so far -- Multicast, QoS, etc. -- and built a solid network for your customers. How do you prove to your customers that they are getting the quality that they are paying for? Arguably more important, how do you keep tabs on what's happening on your network so that you can fix any problems before your customer's begin complaining about them? As I'm sure you've guessed, there is a solution that is built in to most reasonably modern versions of Cisco IOS, and that solution is called "ip sla" (service level agreements). To mock up this lab, I had to upgrade my virtual routers slightly by adding IOS images for Cisco 7200 routers, as the IOS versions I had for the 3640 and 2651 routers I've been modeling don't support the ip sla feature.

Basically, the ip sla feature sends traffic across the network to measure packet loss, latency, jitter, etc. What's nice is that ip sla's allow you to craft the packets (or datagrams) that will be sent to mimic whatever kind of traffic you want to monitor. For example, you can set the TOS markings to monitor priority traffic across the network. You can tell the IP SLA engine to send data inside the G729 codec to measure how voice traffic is affected by the network. You can specify how often to send this traffic across the network, and how many packets to send at a time. It's kind of like the ping tool on steroids ;)

Suppose you have the following network...:


Because we have clients connected to R5, and a streaming server connected through R3, we want to make sure that jitter and latency is within acceptable limits between these two routers. We could build a Smokeping server (and in fact, Smokeping is running on the server labeled "CentOS6"), but Smokeping just sends standard ICMP packets. What if we want to measure latency of UDP traffic with specific QoS markings? Smokeping can't do that. Fortunately, this is exactly the kind of problem that the ip sla feature was designed to address, so we will enable ip sla on R3 and R5:
R3#sho run | begin ip sla
ip sla responder
ip sla 10015
udp-jitter 10.254.254.5 32766 source-ip 10.254.254.3 num-packets 20
tos 184
frequency 300
ip sla schedule 10015 life forever start-time now
ip sla 10025
udp-echo 10.254.254.5 32764 source-ip 10.254.254.3
ip sla schedule 10025 life forever start-time now

...and on R5:
R5#sho run | begin ip sla
ip sla responder
ip sla 10013
udp-jitter 10.254.254.3 32766 source-ip 10.254.254.5 num-packets 20
tos 184
frequency 300
ip sla schedule 10013 life forever start-time now
ip sla 10023
udp-echo 10.254.254.3 32764 source-ip 10.254.254.5
ip sla schedule 10023 life forever start-time now

The first line, "ip sla responder" tells the router to respond to IP SLA traffic. Next, we create an IP SLA process to send traffic to remote destination. The 5-digit identifier is simply a unique identifier on that router; in my case, the first three digits were pretty much arbitrary, the fourth digit I used to separate the UDP jitter test from the UDP echo test, and the last digit was taken from the router number (i.e., xxxx3 was a query for R3, xxxx5 was a query for R5, etc.). This is perhaps overkill with the examples I've shown, but on R1, for example, I set up similar IP SLA probes for R2-R5. Next, I specified what kind of test I wanted to run ("udp-jitter, udp-echo -- there are many, many more options to choose from), the destination IP address, a random port number to probe on the destination, the source IP address to use in the outbound datagram, and the optional "num-packets" flag to specify how many packets (grrr...not to be pedantic, but UDP -> datagram, Cisco! In technical fields, precise terminology often is important </rant>). In the UDP jitter test, the next line specifies that we want to apply a TOS marking so that this packet receives the same kind of priority handling that VoIP traffic (DSCP EF) would receive. After that, I specified a frequency of one test every 5 minutes for the UDP jitter test (I left the UDP echo test at the default settings). Finally, I specified the schedule for the IP SLA probes, start now and run indefinitely. However, if you wanted to monitor performance over a weekend or for a short period during backups at night, you could specify different values:
R3(config)#ip sla schedule 10023 life ?
  <0-2147483647>  Life seconds (default 3600)
  forever         continue running forever

R3(config)#ip sla schedule 10023 life 18600 ?
  ageout      How long to keep this Entry when inactive
  recurring   Probe to be scheduled automatically every day
  start-time  When to start this entry
  <cr>

R3(config)#ip sla schedule 10023 life 18600 recurring ?
  ageout      How long to keep this Entry when inactive
  start-time  When to start this entry
  <cr>

R3(config)#ip sla schedule 10023 life 18600 recurring start-time ?
  after     Start after a certain amount of time from now
  hh:mm     Start time (hh:mm)
  hh:mm:ss  Start time (hh:mm:ss)
  now       Start now
  pending   Start pending

R3(config)#ip sla schedule 10023 life 18600 recurring start-time 23:00 ?
  <1-31>  Day of the month
  MONTH   Month of the year
  ageout  How long to keep this Entry when inactive
  <cr>

R3(config)#$dule 10023 life 18600 recurring start-time 23:00 November 1 ?
  ageout  How long to keep this Entry when inactive
  <cr>

Cool. So...what happens when you enable IP SLA monitoring on your routers?

R3#sho ip sla statistics
IPSLAs Latest Operation Statistics

IPSLA operation id: 10015
Type of operation: udp-jitter
        Latest RTT: 17 milliseconds
Latest operation start time: *00:20:17.115 AKDT Wed Oct 26 2016
Latest operation return code: OK
RTT Values:
        Number Of RTT: 19               RTT Min/Avg/Max: 1/17/51 milliseconds
Latency one-way time:
        Number of Latency one-way Samples: 0
        Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds
        Destination to Source Latency one way Min/Avg/Max: 0/0/0 milliseconds
Jitter Time:
        Number of SD Jitter Samples: 17
        Number of DS Jitter Samples: 18
        Source to Destination Jitter Min/Avg/Max: 0/11/39 milliseconds
        Destination to Source Jitter Min/Avg/Max: 0/8/22 milliseconds
Packet Loss Values:
        Loss Source to Destination: 1           Loss Destination to Source: 0
        Out Of Sequence: 0      Tail Drop: 0
        Packet Late Arrival: 0  Packet Skipped: 0
Voice Score Values:
        Calculated Planning Impairment Factor (ICPIF): 0
        Mean Opinion Score (MOS): 0
Number of successes: 6
Number of failures: 0
Operation time to live: Forever



IPSLA operation id: 10025
Type of operation: udp-echo
        Latest RTT: 8 milliseconds
Latest operation start time: *00:24:23.535 AKDT Wed Oct 26 2016
Latest operation return code: OK
Number of successes: 58
Number of failures: 0
Operation time to live: Forever


R3#

That is at once a lot of information and not nearly as much information as you might hope to receive. While there is a lot of output showing quite a bit of detail, this is only the results of the last test. If you want any kind of historical perspective, you will need to put in some additional work. For example, I believe that you can have the results of the IP SLA monitoring logged to a syslog server (Splunk, perhaps?), although I haven't tried that. Alternatively, you can use SNMP to pull the results, although finding the right OIDs to query is a bit of a chore.

For now, let's focus on the CLI output. First, the UDP jitter test: the executive summary of the IP SLA monitoring is presented at the very end of the output:
Voice Score Values:
        Calculated Planning Impairment Factor (ICPIF): 0
        Mean Opinion Score (MOS): 0
Number of successes: 6
Number of failures: 0

A quick Google search provides a little insight into this output. Both ICPIF and MOS attempt to provide a metric for evaluating how suitable a network is for Voice over IP telephony, with lower scores being better than higher scores.
ICPIF values numerically less than 20 are generally considered "adequate."

MOS, on the other hand, is rated on a scale from 1 to 5, with 1 being very poor quality and 5 being very good quality. Note: In my output, the given MOS score was zero. Per Cisco, this means that the "...MOS data could not be generated for the given operation."

That should be enough to get you started with IP SLA monitoring on IOS. It's definitely a feature worth checking out if you will be using Cisco routers on any kind of production network.

Cisco Intro to QoS and CoS, Part 4 -- Shape Average vs. Bandwidth in Policy Maps

If you've poked around at all with the policy-map statement, you've probably noticed that there are several options for configuring how much bandwidth to allocate to different classes within the QoS config. Let's look at two of the options, shape average and bandwidth.

We'll use the following network...:


...and the following configuration:
ip access-list extended PRIORITY-IN
 permit ip 192.168.1.0 0.0.0.255 any
 permit ip any 192.168.1.0 0.0.0.255
 deny ip any any
ip access-list extended SCAVENGER-IN
 permit ip 192.168.2.0 0.0.0.255 any
 permit ip any 192.168.2.0 0.0.0.255
 deny ip any any
!
class-map match-any PRIORITY
 match dscp af41
class-map match-any SCAVENGER
 match dscp cs1
!
class-map match-any PRIORITY-IN
 match access-group name PRIORITY-IN
class-map match-any SCAVENGER-IN
 match access-group name SCAVENGER-IN
!
policy-map REMARK_ALL
 class PRIORITY-IN
  set dscp af41
 class SCAVENGER-IN
  set dscp cs1
policy-map EGRESS-SHAPER
 class PRIORITY
  shape average percent 70
 class SCAVENGER
  shape average percent 5
policy-map EGRESS-BANDWIDTH
 class PRIORITY
  bandwidth percent 70
 class SCAVENGER
  bandwidth percent 5
!

For my first test, I set up the policy-map, "EGRESS-SHAPER," on interface eth3/0 on both routers, and set up the "REMARK_ALL" service policy on the ingress interfaces (fa0/0 on both routers, and fa1/0 on R1):
interface FastEthernet0/0
 service-policy input REMARK_ALL
!
interface FastEthernet1/0
 service-policy input REMARK_ALL
!
interface Ethernet3/0
 service-policy output EGRESS-SHAPER
!

Then, I set up iperf as a service on host "Knoppix" and ran iperf as a client on both Knoppix Clones, using the flags "-i 1 -t 120," finding that Knoppix Clone 1 could send data at a little over 5Mbps, and Knoppix Clone 2 sent data at about 500Kbps. I then re-ran the test using the service policy "EGRESS-BANDWIDTH," where I saw a transfer speed of just over 4Mbps for Knoppix Clone 1 and about 1 1/3Mbps for Knoppix Clone 2. That surprised me; I didn't really understand why my bandwidth decreased in class PRIORITY (Knoppix Clone 1) and increased in class SCAVENGER (Knoppix Clone 2). To understand what was happening, I did some more tests, using both TCP and UDP transfers ("-u -b 10M" for the flags to set a UDP test at up to 10Mbps), and with both hosts in contention for bandwidth as well as with the hosts transferring data one at a time (i.e., run iperf on Knoppix Host 1, and then running it on Knoppix Clone 2 after the test completed on Knoppix Clone 1). Here are the results from all of the tests:

Congested:
  TCP:
    Using EGRESS-SHAPER:
      192.168.1.x: 5.15Mbps
      192.168.2.x: 0.48Mbps

    Using EGRESS-BANDWIDTH:
      192.168.1.x: 4.16Mbps
      192.168.2.x: 1.36Mbps


  UDP:
    Using EGRESS-SHAPER:
      192.168.1.x: 3.07Mbps
      192.168.2.x: 0.16Mbps

    Using EGRESS-BANDWIDTH:
      192.168.1.x: 2.8Mbps
      192.168.2.x: 1.26Mbps


Uncongested:
  TCP:
    Using EGRESS-SHAPER:
      192.168.1.x: 5.76Mbps
      192.168.2.x: 0.50Mbps

    Using EGRESS-BANDWIDTH:
      192.168.1.x: 5.52Mbps
      192.168.2.x: 5.51Mbps

  UDP:
    Using EGRESS-SHAPER:
      192.168.1.x: 5.82Mbps
      192.168.2.x: 0.47Mbps

    Using EGRESS-BANDWIDTH:
      192.168.1.x: 6.03Mbps
      192.168.2.x: 6.14Mbps

What's happening here is that "shape average" tells the router to strictly allocate the specified amount of bandwidth to the various traffic classes. Even if more bandwidth is available through the interface, the allocated bandwidth is all that the traffic class will receive. When using the "bandwidth" statement, however, the router will allow the traffic classes to use all of the available bandwidth. Since class PRIORITY takes, well, priority over class SCAVENGER, when there is traffic in both traffic classes, class SCAVENGER will only receive the allocated bandwidth. However, when there is little or no priority traffic on the interface, the router will allow class SCAVENGER to "borrow" bandwidth from the other traffic classes in the service policy (in this case, class PRIORITY). Consequently, when there was no contention for bandwidth, class SCAVENGER can get the full data rate on the interface*.

*Q: Wait a minute! An Ethernet port runs at 10Mbps, but these results are showing about 6Mbps. What happened to the other 4Mbps?
A: Keep in mind that an Ethernet port runs at half duplex, meaning that the same copper wire is used for both transmit and receive. Therefore, the 10Mbps bandwidth available on the Ethernet port has to be shared between transmit and receive.**

**Q: Okay, so that explains why it's not 10Mbps, but in that case, shouldn't the data rate be lower, like 5Mbps?
A: If both sides were transmitting data at line rate, then yes. However, in this case the flow of data is mostly one-sided, especially with the UDP traffic, where there is no ACK being sent back to the two Knoppix Clones.***

***Q: In that case, should the data rate be higher?
A: The devil is in the details, and there are a lot of details to consider ;) "Customer" data isn't all that is flowing between R1 and R2. R1 and R2 are also sending their own data back and forth, for example, OSPF updates, CDP updates, etc. Consequently, while the flow of data between R1 and R2 is mostly one-sided, it's not completely one sided. If you REALLY want a good understanding of what's happening, replace R2 with the Knoppix host and run wireshark to capture the packets during a test. I'll leave that as an exercise for the reader ;)

Cisco Intro to QoS and CoS, Part 3 -- Classifying Traffic with NBAR

In our QoS labs so far, we've had to find some way to identify traffic, usually by creating an ACL to match packets against some criteria, such as source/destination port, source/destination IP address, or protocol (TCP/UDP/ICMP), etc. However, Cisco offers another way to match traffic: NBAR, or "Network Based Application Recognition. I created the following network in GNS3 to start playing with NBAR:

To enable NBAR on a router, you first need to turn on CEF, and then you need to enable NBAR on any interface that will have a service policy to mark and classify traffic. In this network, I am using an NM-16ESW switchport module in slot 0, so while Fa0/0 is technically the ingress port, I would enable NBAR on my VLAN interface (int VLAN10). If you are using a plain FastEthernet port rather than a switch module, then you would enable NBAR on the Fa port. In either case, here is how to do it:
R4(config)#ip cef
R4(config)#int vlan 10
R4(config-if)#ip nbar protocol-discovery

After enabling NBAR, we'll need to design our QoS schema. Let's start by identifying the types of traffic on our network, and by deciding what traffic will take priority over other traffic. I came up with the following (admittedly hokey) schema, ordered by priority:

1. EIGRP
2. HTTP (simulating voice/video with a streaming mp4 file)
3. Telnet
4. SNMP
5. SSH
6. ICMP

I then mapped this traffic to the following traffic classes:

Traffic	DSCP Value
EIGRP	CS6
HTTP	EF
Telnet	AF41
SNMP	CS3
SSH	AF21
ICMP	CS2

Cool! Let's start configuring the class-maps on R4, R5 and R6:
R4(config)#class-map match-all EIGRP
R4(config-cmap)#match protocol eigrp
R4(config-cmap)#class-map match-all HTTP
R4(config-cmap)# match protocol http
R4(config-cmap)#class-map match-all SSH
R4(config-cmap)# match protocol ssh
R4(config-cmap)#class-map match-all TELNET
R4(config-cmap)# match protocol telnet
R4(config-cmap)#class-map match-all SNMP
R4(config-cmap)# match protocol snmp
R4(config-cmap)#class-map match-all ICMP
R4(config-cmap)#match protocol icmp

...and now, the policy-map to use these classes:
R4(config-cmap)#policy-map REMARK_ALL
R4(config-pmap)#description policy-map to place traffic into the appropriate traffic class
R4(config-pmap)# class EIGRP
R4(config-pmap-c)# set ip dscp cs6
R4(config-pmap-c)# class HTTP
R4(config-pmap-c)# set ip dscp ef
R4(config-pmap-c)# class SSH
R4(config-pmap-c)# set ip dscp af21
R4(config-pmap-c)# class TELNET
R4(config-pmap-c)# set ip dscp af41
R4(config-pmap-c)# class SNMP
R4(config-pmap-c)# set ip dscp cs3
R4(config-pmap-c)# class ICMP
R4(config-pmap-c)# set ip dscp cs2

Now, just enable the policy map in your ingress interface (VLAN10 in this lab), and you are done:
R4(config-if)#int vlan10
R4(config-if)# service-policy input REMARK_ALL

Pretty easy, huh? To utilize NBAR to identify traffic from various protocols, all you have to do is use the "match protocol <protocol>" statement inside a class-map and NBAR will identify the traffic for you! Keep in mind, however, that NBAR is deep-packet inspection -- it isn't just a simple match for port number and IP address -- and therefore, this functionality comes at a cost of CPU cycles (unless you are using a more recent model switch/router that off-loads the packet inspection to a separate processor). Consequently, if a simple ACL will meet your needs, it MAY be less resource intensive to use the ACL, as we've done in earlier labs. However, if you have a higher-powered router, NBAR certainly makes identifying and classifying traffic much easier on the network admin.

Note:
This is just a really simple example of NBAR, and doesn't even begin to scratch the surface of what NBAR can do for you. For example, when matching the HTTP protocol, you can create multiple classes based upon the host in the HTTP request and you can even use a regex to pattern-match within the URL. You can also extend NBAR with external files to add support for protocols that are not already built in.

For the sake of completeness, here is the rest of the QoS configuration on R4: R4(config)#class-map match-any PRIORITY
R4(config-cmap)# match ip dscp ef
R4(config-cmap)#class-map match-any CONTROL
R4(config-cmap)# match ip dscp cs6
R4(config-cmap)#class-map match-any CRITICAL
R4(config-cmap)# match ip dscp af41
R4(config-cmap)# match ip dscp cs3
R4(config-cmap)#class-map match-any ROUTINE
R4(config-cmap)# match ip dscp cs2
R4(config-cmap)# match ip dscp af21
R4(config-cmap)#class-map match-any SCAVENGER
R4(config-cmap)# match ip dscp CS1
R4(config)#policy-map EDGE_CHILD
R4(config-pmap)# description Core links
R4(config-pmap)# class PRIORITY
R4(config-pmap-c)# priority percent 25
R4(config-pmap-c)# class CONTROL
R4(config-pmap-c)# bandwidth percent 10
R4(config-pmap-c)# class CRITICAL
R4(config-pmap-c)# bandwidth percent 15
R4(config-pmap-c)# random-detect dscp-based
R4(config-pmap-c)# class ROUTINE
R4(config-pmap-c)# bandwidth percent 10
R4(config-pmap-c)# random-detect dscp-based
R4(config-pmap-c)# class SCAVENGER
R4(config-pmap-c)# bandwidth percent 1
R4(config-pmap-c)# random-detect dscp-based
R4(config-pmap-c)# class class-default
R4(config-pmap-c)# fair-queue
R4(config-pmap-c)# random-detect dscp-based
R4(config-pmap-c)#!
R4(config-pmap-c)#policy-map EDGE2CORE
R4(config-pmap)# description Parent policy for Edge-to-Core links
R4(config-pmap)# class class-default
R4(config-pmap-c)# shape average 10000000
R4(config-pmap-c)# service-policy EDGE_CHILD
R4(config-pmap-c)#!
R4(config-if)#int fa1/0
R4(config-if)# service-policy output EDGE2CORE
R4(config-if)#!
R4(config-if)#int fa3/0
R4(config-if)# service-policy output EDGE2CORE
R4(config-if)#!


Cisco Intro to QoS and CoS, Part 2 -- A Deeper Dive into Marking Traffic

In our last lab, we set up a simple network, and showed how applying a QoS policy to prioritize certain traffic over other traffic could help high-priority, time-sensitive applications work properly, even during times of network congestion. But what did all of the access lists, class maps and policy maps really do? In this lab, we'll take a closer look at what the class maps and policy maps are really doing at the packet layer.

First, we'll start with a new network diagram:


As we stated last time, the first step in designing an effective QoS policy is identifying the traffic on your network, and deciding how you want to prioritize it. On this network, we will sort network traffic into the following classes:

1. Core Control: OSPF, in this case (which is automatically identified and marked by the router itself);
2. VoIP: We will be simulating voice traffic by streaming an MP4 video file from the host CentOS6 to hosts on the "voice" VLAN (VLAN 100);
3. Call Signaling: We will also be simulating call signaling traffic with SSH from the "voice" VLAN (VLAN 100);
4. Routine: Bulk data traffic from the "LAN" VLAN (VLAN 10);
5. default: Anything not otherwise classified.

Here are the access lists that we will use to identify the traffic:
ip access-list extended CALLSIGNALING
 permit tcp 192.168.100.0 0.0.0.255 any eq 22
 permit tcp any eq 22 192.168.100.0 0.0.0.255
 deny ip any any
ip access-list extended ROUTINE
 permit ip 192.168.1.0 0.0.0.255 any
 permit ip any 192.168.1.0 0.0.0.255
 deny ip any any
ip access-list extended VOIP
 permit tcp 192.168.100.0 0.0.0.255 any eq www
 permit tcp any eq www 192.168.100.0 0.0.0.255
 deny ip any any
!

Like I said earlier, the router automatically identifies and marks routing protocol traffic, so we don't need to create a separate ACL for that.

After creating the ACL's to match the network traffic, we will create the class maps:
class-map match-any NETWORK_CONTROL
 match ip dscp cs6
class-map match-any VOIP
 match ip dscp ef
 match access-group name VOIP
class-map match-any CALLSIGNALING
 match ip dscp cs3
 match ip dscp af31
 match access-group name CALLSIGNALING
class-map match-any ROUTINE
 match ip dscp cs2
 match access-group name ROUTINE
!

Let's discuss the class map in a little more detail. First, notice that the first line of each class map contains the phrase "match-any." Take, for example, the class map for CALLSIGNALING. We have a line that states, "match ip dscp cs3," and then we have two other "match..." statements following that line. Since we created the class map using the "match-any" statement, then we are essentially using a logical OR to match either the DSCP marking CS3, the DSCP marking AF31 or the "CALLSIGNALING" access list. If we had used the phrase, "match-all," then we would be performing a logical AND against all of the statements (which would never match, as the DSCP CS3 and DSCP AF31 markings are mutually exclusive...but it can be useful if you are matching on other criteria than DSCP markings).

We will explain why we are matching against either DSCP markings or an access list shortly, but for now, let's proceed to the policy-map:
policy-map EGRESS
 class VOIP
  priority percent 20
 class CALLSIGNALING
  bandwidth percent 5
 class ROUTINE
  bandwidth percent 50
  random-detect dscp-based
 class NETWORK_CONTROL
  bandwidth percent 5
 class class-default
  bandwidth percent 1
  random-detect dscp-based
!
policy-map EGRESS-BW
 class class-default
  shape average 10000000
  service-policy EGRESS
!
policy-map INGRESS
 class VOIP
  set dscp ef
 class CALLSIGNALING
  set dscp af31
 class ROUTINE
  set dscp cs2
 class class-default
  set dscp cs1

We have created three policy-maps in this example: EGRESS, EGRESS-BW, and INGRESS (creative names, no?).

EGRESS-BW is pretty simple: essentially, we are shaping all of the traffic in this policy to 10Mbps, then calling the "EGRESS" policy. This is a pretty common strategy, as it allows you to create a parent policy (EGRESS-BW, in this case) to define the shaper/policer, and then a child policy to break out the bandwidth to the individual traffic classes. This allows for a very flexible approach, since you can create multiple parent policies that call the same child policy, allowing you to quickly make changes to your traffic shaper.

The EGRESS policy matches traffic against the class-maps that we previously defined, then divvies that bandwidth up into the individual traffic classes by percentage. There are two details worth a little extra discussion in the EGRESS policy. First, notice that class VOIP uses "priority percent..." whereas all of the other classes use "bandwidth percent..." This places VOIP traffic on the priority (low-latency) queue. The other classes buffer incoming traffic, placing it on a queue until the router gets a chance to transmit it on the interface; class VOIP, on the other hand is transmitted IMMEDIATELY. This is because VoIP traffic is very sensitive to jitter, and queueing up traffic before transmitting it causes latency. However, be aware that if traffic in class VOIP cannot be transmitted immediately, it will be DISCARDED! Therefore, you don't want to use the "priority" statement unless it is for a traffic class that can tolerate drops, but cannot tolerate jitter. Second, notice how class "ROUTINE" and class "class-default" (a built-in traffic class on Cisco routers) use the statement, "random-detect dscp-based?" This statement tells the QoS policy to enable the "random early detection" scheduler, which will randomly discard packets as the traffic queues become congested. The thresholds at which RED begins to discard traffic are configurable, but in this example, we are using the default settings, as this is beyond the scope of this tutorial. At first, randomly discarding traffic might sound like a bad idea, but the idea behind this strategy is that by randomly discarding the occasional packet before the network becomes fully congested, the service sending the traffic that is being dropped will begin to slow down the rate at which it is transmitted, thus delaying the onset of congestion. However, we do not want to enable this feature in class VOIP or class CALLSIGNALING, as this is priority traffic that you really don't want to drop.

Finally, the INGRESS policy is where the traffic inbound to the router from the network clients (Knoppix_Clone_1 and Knoppix_Clone_2) is identified and marked. The only action taken by the INGRESS policy is applying the DSCP markings to the traffic, based upon the access control lists. In truth, if an incoming packet already had a DSCP marking, the INGRESS policy would happily classify traffic based upon that, unless other configuration items were present in the router config, but for now, let's just assume that all incoming traffic has no DSCP markings, and therefore the only lines in the class-maps that will match are the "match access-group name..." statements.

So...this is great, in theory, but does it actually work? To find out, I ran tcpdump on the CentOS6 host to capture all incoming network traffic. If the router configuration is good, then...:

1. Any traffic coming from 192.168.1.x should be marked with a DSCP value of CS2;
2. Any traffic coming from 192.168.100.x outbound to any host on port 22 should be marked with a DSCP value of AF31;
3. Any traffic coming from 192.168.100.x outbound to any host on port 80 should be marked with a DSCP value of EF;
4. and Any traffic originating on the router itself that is identified as OSPF should be marked with a DSCP value of CS6¹.

After running several tests (streaming the MP4 file on CentOS6 to the Knoppix Clone on the VoIP VLAN, SSH'ing to CentOS6 from the Knoppix host on the VoIP VLAN, and connecting to the HTTP process on CentOS 6 from the Knoppix host on the LAN VLAN), I copied the capture file from CentOS6 to my local desktop, and opened up the file with Wireshark. Let's look at traffic originating on 192.168.1.x:


Yep, "Class Selector 2" is the value of the DSCP marking for traffic originating on 192.168.1.2, so it looks like our QoS policies are correctly identifying traffic on the LAN VLAN, and correctly marking it with a CS2 DSCP marking. Next let's look for traffic originating on the Voice VLAN, outbound for CentOS6 on port 22 (SSH):


Perfect! This screen capture shows that traffic originating on 192.168.100.2, sent to 100.64.1.2 on port 22 was correctly identified and marked with a DSCP value of AF31. Next, let's see if traffic from 192.168.100.2 to 100.64.1.2 on port 80 is properly marked as "EF:"


Yep, this traffic has a DSCP marking of "EF," just as expected. Finally, let's verify that the OSPF multicast traffic from the router is properly marked as CS6:


That looks good, too.

Now that we have verified that traffic is being classified and marked appropriately, let's look at a very useful command for troubleshooting QoS and congestion on a Cisco router. Suppose that one of your users calls to report problems with their voice-over-IP telephones. They've already had the telephone tech look at the phone and the Call Manager, but because the phone tech found nothing wrong, they suspect it is a network problem. How can you tell if your QoS policy is dropping packets, due to excessive traffic in one of your queues?

If I had received a call such as this, I would start by looking at the QoS statistics for the outbound interface on the router -- in this case, FastEthernet1/0:
R1#sho policy-map int fa1/0  
FastEthernet1/0

  Service-policy output: EGRESS-BW

    Class-map: class-default (match-any)
      4298 packets, 316655 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any
      Traffic Shaping
           Target/Average   Byte   Sustain   Excess    Interval  Increment
             Rate           Limit  bits/int  bits/int  (ms)      (bytes)  
         10000000/10000000  62500  250000    250000    25        31250    

        Adapt  Queue     Packets   Bytes     Packets   Bytes     Shaping
        Active Depth                         Delayed   Delayed   Active
        -      0         4298      316655    0         0         no

      Service-policy : EGRESS

        Class-map: VOIP (match-any)
          1556 packets, 103168 bytes
          5 minute offered rate 0 bps, drop rate 0 bps
          Match: ip dscp ef
            1556 packets, 103168 bytes
            5 minute rate 0 bps
          Match: access-group name VOIP
            0 packets, 0 bytes
            5 minute rate 0 bps
          Queueing
            Strict Priority
            Output Queue: Conversation 264
            Bandwidth 20 (%)
            Bandwidth 2000 (kbps) Burst 50000 (Bytes)
            (pkts matched/bytes matched) 0/0
            (total drops/bytes drops) 0/0

        Class-map: CALLSIGNALING (match-any)
          1035 packets, 79610 bytes
          5 minute offered rate 0 bps, drop rate 0 bps
          Match: ip dscp cs3
            0 packets, 0 bytes
            5 minute rate 0 bps
          Match: ip dscp af31
            1035 packets, 79610 bytes
            5 minute rate 0 bps
          Match: access-group name CALLSIGNALING
            0 packets, 0 bytes
            5 minute rate 0 bps
          Queueing
            Output Queue: Conversation 265
            Bandwidth 5 (%)
            Bandwidth 500 (kbps)
            (pkts matched/bytes matched) 0/0
        (depth/total drops/no-buffer drops) 0/0/0
             exponential weight: 9
             mean queue depth: 0

   dscp    Transmitted      Random drop      Tail drop    Minimum Maximum  Mark
           pkts/bytes       pkts/bytes       pkts/bytes    thresh  thresh  prob
   af11       0/0               0/0              0/0           32      40  1/10
   af12       0/0               0/0              0/0           28      40  1/10
   af13       0/0               0/0              0/0           24      40  1/10
   af21       0/0               0/0              0/0           32      40  1/10
   af22       0/0               0/0              0/0           28      40  1/10
   af23       0/0               0/0              0/0           24      40  1/10
   af31    1035/79610           0/0              0/0           32      40  1/10
   af32       0/0               0/0              0/0           28      40  1/10
   af33       0/0               0/0              0/0           24      40  1/10
   af41       0/0               0/0              0/0           32      40  1/10
   af42       0/0               0/0              0/0           28      40  1/10
   af43       0/0               0/0              0/0           24      40  1/10
    cs1       0/0               0/0              0/0           22      40  1/10
    cs2       0/0               0/0              0/0           24      40  1/10
    cs3       0/0               0/0              0/0           26      40  1/10
    cs4       0/0               0/0              0/0           28      40  1/10
    cs5       0/0               0/0              0/0           30      40  1/10
    cs6       0/0               0/0              0/0           32      40  1/10
    cs7       0/0               0/0              0/0           34      40  1/10
     ef       0/0               0/0              0/0           36      40  1/10
   rsvp       0/0               0/0              0/0           36      40  1/10
default       0/0               0/0              0/0           20      40  1/10


        Class-map: ROUTINE (match-any)
          284 packets, 23609 bytes
          5 minute offered rate 0 bps, drop rate 0 bps
          Match: ip dscp cs2
            284 packets, 23609 bytes
            5 minute rate 0 bps
          Match: access-group name ROUTINE
            0 packets, 0 bytes
            5 minute rate 0 bps
          Queueing
            Output Queue: Conversation 266
            Bandwidth 50 (%)
            Bandwidth 5000 (kbps)
            (pkts matched/bytes matched) 0/0
        (depth/total drops/no-buffer drops) 0/0/0
             exponential weight: 9
             mean queue depth: 0

   dscp    Transmitted      Random drop      Tail drop    Minimum Maximum  Mark
           pkts/bytes       pkts/bytes       pkts/bytes    thresh  thresh  prob
   af11       0/0               0/0              0/0           32      40  1/10
   af12       0/0               0/0              0/0           28      40  1/10
   af13       0/0               0/0              0/0           24      40  1/10
   af21       0/0               0/0              0/0           32      40  1/10
   af22       0/0               0/0              0/0           28      40  1/10
   af23       0/0               0/0              0/0           24      40  1/10
   af31       0/0               0/0              0/0           32      40  1/10
   af32       0/0               0/0              0/0           28      40  1/10
   af33       0/0               0/0              0/0           24      40  1/10
   af41       0/0               0/0              0/0           32      40  1/10
   af42       0/0               0/0              0/0           28      40  1/10
   af43       0/0               0/0              0/0           24      40  1/10
    cs1       0/0               0/0              0/0           22      40  1/10
    cs2     284/23609           0/0              0/0           24      40  1/10
    cs3       0/0               0/0              0/0           26      40  1/10
    cs4       0/0               0/0              0/0           28      40  1/10
    cs5       0/0               0/0              0/0           30      40  1/10
    cs6       0/0               0/0              0/0           32      40  1/10
    cs7       0/0               0/0              0/0           34      40  1/10
     ef       0/0               0/0              0/0           36      40  1/10
   rsvp       0/0               0/0              0/0           36      40  1/10
default       0/0               0/0              0/0           20      40  1/10


        Class-map: class-default (match-any)
          1423 packets, 110268 bytes
          5 minute offered rate 0 bps, drop rate 0 bps
          Match: any
          Queueing
            Output Queue: Conversation 267
            Bandwidth 1 (%)
            Bandwidth 100 (kbps) Max Threshold 64 (packets)
            (pkts matched/bytes matched) 0/0
        (depth/total drops/no-buffer drops) 0/0/0
R1#

cough...cough...That's a lot of output...what does it mean?

First, we need a little more information from the user. Were they having problems answering the phone, transferring the call to another number, or hanging up once the call was complete? If so, you would need to look into the call signaling queue. Was the call garbled, or was the audio cutting out during the call? In that case, you would need to look at the VOIP queue. Let's assume that the caller complained of call signaling symptoms. We would narrow our focus to the statistics relating to class CALLSIGNALING. I won't re-copy all of the output from class CALLSIGNALING, since it's so long, but scroll back up and look at the output starting with "Class-map CALLSIGNALING (match-any)." The first thing I would check is how many drops you've had in this queue. See the line that says, "(depth/total drops/no-buffer drops) 0/0/0?" That tells you that you have allocated adequate bandwidth for class CALLSIGNALING. You do not have any packets in the buffer waiting to be transmitted, and you have not dropped any packets since the last time the interface counters were cleared or since the service policy was instated (whichever was later). Also, look at this part of the output:
   dscp    Transmitted      Random drop      Tail drop    Minimum Maximum  Mark
           pkts/bytes       pkts/bytes       pkts/bytes    thresh  thresh  prob
<...snip...>
   af31    1035/79610           0/0              0/0           32      40  1/10
<...snip...>

"Random Drops" are where your random early detection scheduler begins dropping packets before the network becomes fully congested. "Tail Drops" are where your queue has already buffered all the packets it can hold, and new incoming packets are being discarded, since there is no room left on the queue to store them. In this case, both tail drops and random drops are zero, meaning that the QoS policy has not dropped any traffic with the AF31 marking (since that was all that we were marking with our INGRESS policy; typically, you would want to look at both CS3 and AF31 in a call signaling queue).

Since there are no dropped packets in the call signaling queue, you would need to look elsewhere for the problem (for example, look for dropped packets on the router on the other side of the circuit, or make sure that the router is recognizing traffic from the phone as belonging to class VOIP or class CALLSIGNALING, as appropriate). However, suppose you saw output like this:
        Class-map: CALLSIGNALING (match-any)
          2901880 packets, 192238996 bytes
          30 second offered rate 4000 bps, drop rate 0 bps
          Match: ip dscp cs3 (24) af31 (26)
            2901880 packets, 192238996 bytes
            30 second rate 4000 bps
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/12155/0
          (pkts output/bytes output) 2889729/190911691
          bandwidth 320 kbps

In this case, you have dropped 12,155 packets(!) since the last time the counters were cleared (which was about four days ago, although you can't tell that from the output above). This means that you have not allocated adequate bandwidth to class CALLSIGNALING, and either 1) you need to allocate more bandwidth to handle the call volume, or 2) you have traffic on this circuit that is not being marked appropriately. On this router, I strongly suspect that there is traffic on the circuit that is being marked incorrectly, as we have already allocated 320Kbps to class CALLSIGNALING, which is a LOT of bandwidth for call signaling traffic.

Footnotes:
1:Yes, it is silly to run OSPF on this router, since there are no other routers with which it can share routes. The only reason I configured OSPF in this lab is to show that the router does, indeed, automatically mark OSPF traffic with the CS6 DSCP marking.

Cisco Intro to QoS and CoS, Part 1 -- Identifying, Classifying and Marking Traffic

One problem found in any network of reasonable size is how to handle congestion. When there is more bandwidth available than clients require, there is (of course!) no problem. However, some types of network traffic -- such as streaming media -- are particularly bandwidth intensive, and often, very sensitive to variations in delay time. Consider, for example, HTTP traffic: an end user will most likely not notice the variation in transit times ("jitter") of the various packets that make up a requested download. On the other hand, a Voice over IP call is very sensitive to jitter. If the packets making up a Voice over IP call or videoconference don't arrive within a reasonably consistent timeframe, the call quality will suffer, and your end users will not be happy. Even worse, what would happen if your dynamic routing protocols cannot send updates and keepalive traffic in a timely fashion? Clearly, there needs to be a way to prioritize certain types of traffic to ensure that your clients receive sufficient bandwidth for their needs.

Consider the following network:


On the left-hand side of the network drawing, we have three Linux hosts, clients of the server on the right-hand side of the drawing. These three hosts will be in competition with each other for network resources, and because we are running OSPF (or routing protocol of your choice -- and yes, a dynamic routing protocol is overkill on a network this simple, but stay with me...) on R1 and R2, they will be competing with the network clients for bandwidth as well. Making this problem even more apparent, I have connected all of the PC's to the network via FastEthernet interfaces, but only connected R1 and R2 with a single T1 line. Due to the extreme mismatch between LAN and WAN capabilities, any one of the network clients can easily saturate the "WAN" link between R1 and R2. Don't believe me? Let's do some tests...

On the host "CentOS6," I have configured the "echo" service, and on the host "CentOS6_Clone," I have installed a network diagnostic tool called "tcpspray." With no other traffic on the network, let's use tcpspray to see how long it takes to send a flurry of data to the "CentOS6" server:

[root@centos6_clone ~]# tcpspray -e -n 1000 192.168.2.2
Received 1024000 bytes in 25.670507 seconds (30.955 kbytes/s)
Transmitted 1024000 bytes in 25.146152 seconds (39.768 kbytes/s)
[root@centos6_clone ~]#

Also on the host "CentOS6," I am running a web server, and one of the web pages hosted on the server is an image gallery that loads a 5x5 array of JPEG images (get your mind out of the gutter! They're pictures of motorcycles that inspired me while I was building my Cafe Racer project!). How would the tcpspray bandwidth be affected if one of the Knoppix clients was downloading the images to populate the web page at the same time? Let's find out...:

[root@centos6_clone ~]# tcpspray -e -n 1000 192.168.2.2
Received 1024000 bytes in 118.753065 seconds (8.421 kbytes/s)
Transmitted 1024000 bytes in 104.348504 seconds (9.583 kbytes/s)
[root@centos6_clone ~]#

Wow...it took almost five times as long to receive the replies, due to the congestion on the network! Fortunately, quality of service and/or class of service policies can allow us to prioritize certain kinds of traffic, as our network needs dictate.

Keep in mind that nothing is free, however. If we prioritize some kinds of traffic, that necessarily comes at the expense of other traffic. Therefore, you have to have a good idea what kinds of traffic cross your network, and how they should be balanced against each other...which is a good place for us to begin with our policy. Let's define the traffic that we want prioritized.

As I mentioned earlier, the parameters I used for tcpspray specified that we would use the "echo" service for our traffic. Let's include SSH as priority traffic so we can manage our routers (you are using SSH for network management, right?). As I also mentioned earlier, our network control traffic is also high priority. Too keep the discussion simple for now, we'll assume everything else is "best effort."

Now that we have a good idea how we want to prioritize our traffic, we need a way to have the router filter our priority traffic. If you think that sounds like a job for an Access Control List, you're exactly right:

ip access-list extended CORE_GOLD
 permit tcp any eq echo any
 permit tcp any any eq echo
!
ip access-list extended CUST_EF
 permit tcp any eq 22 any
 permit tcp any any eq 22
!

After identifying the priority traffic with an ACL, we then create "class maps" to map that traffic into different classes (kind of sounds like a tautology, sorry!):

class-map match-any CORE_CONTROL
 description Network control OSPF/BGP/etc
 match ip precedence 6
 match ip precedence 7
class-map match-any CUST-EF
 match access-group name CUST_EF
class-map match-any CORE_GOLD
 description Low-latency/priority queue
 match access-group name CORE_GOLD
!

We have now created three classes of priority traffic:

1. CORE_CONTROL, for our network control traffic,
2. CUST-EF, for high priority traffic ("expedited forwarding"), and
3. CORE_GOLD, for low-latency traffic.

Our next step is to use these maps to define how the different classes of traffic will be handled:

policy-map CUST-IN
 class CUST-EF
  set ip dscp ef
 class class-default
  set dscp cs3
policy-map CUST-Generic
 class CORE_GOLD
  priority percent 50
 class CORE_CONTROL
  bandwidth percent 10
 class class-default
  bandwidth percent 1
!

For our "CUST-EF" traffic, we have said that we want to place a special marking, "dscp ef," on the packets that match this traffic. This gives our "CUST-EF" traffic high priority on the network. For "CORE_GOLD" traffic, we have said that we want to give that traffic 50 per cent of the available bandwidth. For "CORE_CONTROL" traffic, we have said that we want to give such traffic 10 per cent of the available bandwidth (network control traffic is typically small-volume, bursty traffic, so it is not necessary to reserve a large amount of bandwidth for this traffic -- we just need to ensure that the traffic isn't starved for bandwidth during times of congestion). Notice that we also have another de facto class of traffic in the policy maps: class-default. This is for traffic that didn't match any of our access control lists. In the "CUST-IN" policy map, we mark it with "dscp cs3" -- a low priority traffic queue, and in the "CUST-Generic" policy map, we don't reserve any bandwidth for this class.

At this point, we have established the framework for our QoS/CoS configuration, but we haven't actually implemented it yet. Just as an access-control list used to filter certain kinds of traffic has to be applied to an interface before it is useful, so does a policy map:

interface Serial1/0
service-policy output CUST-Generic
!

We also will apply the "CUST-IN" policy map to the ingress interface, VLAN 10 on R1 and Fa0/0 on R2:

R1(config)#int vlan 10
R1(config-if)#service-policy input CUST-IN
R1(config-if)#

R2(config)#int fa0/0
R2(config-if)#service-policy input CUST-IN
R2(config-if)#

Now, let's run our tcpspray test again, while the Knoppix client reloads the web page with the 25 JPEG images:

[root@centos6_clone ~]# tcpspray -e -n 1000 192.168.2.2
Received 1024000 bytes in 78.483684 seconds (12.742 kbytes/s)
Transmitted 1024000 bytes in 69.900585 seconds (14.306 kbytes/s)
[root@centos6_clone ~]#

These results are still slower than our initial baseline test, with no congestion on the network. However, it is significantly better than the test with no QoS/CoS and a congested network.