Cisco Intro to QoS and CoS, Part 4 -- Shape Average vs. Bandwidth in Policy Maps

If you've poked around at all with the policy-map statement, you've probably noticed that there are several options for configuring how much bandwidth to allocate to different classes within the QoS config. Let's look at two of the options, shape average and bandwidth.

We'll use the following network...:


...and the following configuration:
ip access-list extended PRIORITY-IN
 permit ip 192.168.1.0 0.0.0.255 any
 permit ip any 192.168.1.0 0.0.0.255
 deny ip any any
ip access-list extended SCAVENGER-IN
 permit ip 192.168.2.0 0.0.0.255 any
 permit ip any 192.168.2.0 0.0.0.255
 deny ip any any
!
class-map match-any PRIORITY
 match dscp af41
class-map match-any SCAVENGER
 match dscp cs1
!
class-map match-any PRIORITY-IN
 match access-group name PRIORITY-IN
class-map match-any SCAVENGER-IN
 match access-group name SCAVENGER-IN
!
policy-map REMARK_ALL
 class PRIORITY-IN
  set dscp af41
 class SCAVENGER-IN
  set dscp cs1
policy-map EGRESS-SHAPER
 class PRIORITY
  shape average percent 70
 class SCAVENGER
  shape average percent 5
policy-map EGRESS-BANDWIDTH
 class PRIORITY
  bandwidth percent 70
 class SCAVENGER
  bandwidth percent 5
!

For my first test, I set up the policy-map, "EGRESS-SHAPER," on interface eth3/0 on both routers, and set up the "REMARK_ALL" service policy on the ingress interfaces (fa0/0 on both routers, and fa1/0 on R1):
interface FastEthernet0/0
 service-policy input REMARK_ALL
!
interface FastEthernet1/0
 service-policy input REMARK_ALL
!
interface Ethernet3/0
 service-policy output EGRESS-SHAPER
!

Then, I set up iperf as a service on host "Knoppix" and ran iperf as a client on both Knoppix Clones, using the flags "-i 1 -t 120," finding that Knoppix Clone 1 could send data at a little over 5Mbps, and Knoppix Clone 2 sent data at about 500Kbps. I then re-ran the test using the service policy "EGRESS-BANDWIDTH," where I saw a transfer speed of just over 4Mbps for Knoppix Clone 1 and about 1 1/3Mbps for Knoppix Clone 2. That surprised me; I didn't really understand why my bandwidth decreased in class PRIORITY (Knoppix Clone 1) and increased in class SCAVENGER (Knoppix Clone 2). To understand what was happening, I did some more tests, using both TCP and UDP transfers ("-u -b 10M" for the flags to set a UDP test at up to 10Mbps), and with both hosts in contention for bandwidth as well as with the hosts transferring data one at a time (i.e., run iperf on Knoppix Host 1, and then running it on Knoppix Clone 2 after the test completed on Knoppix Clone 1). Here are the results from all of the tests:

Congested:
  TCP:
    Using EGRESS-SHAPER:
      192.168.1.x: 5.15Mbps
      192.168.2.x: 0.48Mbps

    Using EGRESS-BANDWIDTH:
      192.168.1.x: 4.16Mbps
      192.168.2.x: 1.36Mbps


  UDP:
    Using EGRESS-SHAPER:
      192.168.1.x: 3.07Mbps
      192.168.2.x: 0.16Mbps

    Using EGRESS-BANDWIDTH:
      192.168.1.x: 2.8Mbps
      192.168.2.x: 1.26Mbps


Uncongested:
  TCP:
    Using EGRESS-SHAPER:
      192.168.1.x: 5.76Mbps
      192.168.2.x: 0.50Mbps

    Using EGRESS-BANDWIDTH:
      192.168.1.x: 5.52Mbps
      192.168.2.x: 5.51Mbps

  UDP:
    Using EGRESS-SHAPER:
      192.168.1.x: 5.82Mbps
      192.168.2.x: 0.47Mbps

    Using EGRESS-BANDWIDTH:
      192.168.1.x: 6.03Mbps
      192.168.2.x: 6.14Mbps

What's happening here is that "shape average" tells the router to strictly allocate the specified amount of bandwidth to the various traffic classes. Even if more bandwidth is available through the interface, the allocated bandwidth is all that the traffic class will receive. When using the "bandwidth" statement, however, the router will allow the traffic classes to use all of the available bandwidth. Since class PRIORITY takes, well, priority over class SCAVENGER, when there is traffic in both traffic classes, class SCAVENGER will only receive the allocated bandwidth. However, when there is little or no priority traffic on the interface, the router will allow class SCAVENGER to "borrow" bandwidth from the other traffic classes in the service policy (in this case, class PRIORITY). Consequently, when there was no contention for bandwidth, class SCAVENGER can get the full data rate on the interface*.

*Q: Wait a minute! An Ethernet port runs at 10Mbps, but these results are showing about 6Mbps. What happened to the other 4Mbps?
A: Keep in mind that an Ethernet port runs at half duplex, meaning that the same copper wire is used for both transmit and receive. Therefore, the 10Mbps bandwidth available on the Ethernet port has to be shared between transmit and receive.**

**Q: Okay, so that explains why it's not 10Mbps, but in that case, shouldn't the data rate be lower, like 5Mbps?
A: If both sides were transmitting data at line rate, then yes. However, in this case the flow of data is mostly one-sided, especially with the UDP traffic, where there is no ACK being sent back to the two Knoppix Clones.***

***Q: In that case, should the data rate be higher?
A: The devil is in the details, and there are a lot of details to consider ;) "Customer" data isn't all that is flowing between R1 and R2. R1 and R2 are also sending their own data back and forth, for example, OSPF updates, CDP updates, etc. Consequently, while the flow of data between R1 and R2 is mostly one-sided, it's not completely one sided. If you REALLY want a good understanding of what's happening, replace R2 with the Knoppix host and run wireshark to capture the packets during a test. I'll leave that as an exercise for the reader ;)

Cisco Intro to QoS and CoS, Part 3 -- Classifying Traffic with NBAR

In our QoS labs so far, we've had to find some way to identify traffic, usually by creating an ACL to match packets against some criteria, such as source/destination port, source/destination IP address, or protocol (TCP/UDP/ICMP), etc. However, Cisco offers another way to match traffic: NBAR, or "Network Based Application Recognition. I created the following network in GNS3 to start playing with NBAR:

To enable NBAR on a router, you first need to turn on CEF, and then you need to enable NBAR on any interface that will have a service policy to mark and classify traffic. In this network, I am using an NM-16ESW switchport module in slot 0, so while Fa0/0 is technically the ingress port, I would enable NBAR on my VLAN interface (int VLAN10). If you are using a plain FastEthernet port rather than a switch module, then you would enable NBAR on the Fa port. In either case, here is how to do it:
R4(config)#ip cef
R4(config)#int vlan 10
R4(config-if)#ip nbar protocol-discovery

After enabling NBAR, we'll need to design our QoS schema. Let's start by identifying the types of traffic on our network, and by deciding what traffic will take priority over other traffic. I came up with the following (admittedly hokey) schema, ordered by priority:

1. EIGRP
2. HTTP (simulating voice/video with a streaming mp4 file)
3. Telnet
4. SNMP
5. SSH
6. ICMP

I then mapped this traffic to the following traffic classes:

Traffic	DSCP Value
EIGRP	CS6
HTTP	EF
Telnet	AF41
SNMP	CS3
SSH	AF21
ICMP	CS2

Cool! Let's start configuring the class-maps on R4, R5 and R6:
R4(config)#class-map match-all EIGRP
R4(config-cmap)#match protocol eigrp
R4(config-cmap)#class-map match-all HTTP
R4(config-cmap)# match protocol http
R4(config-cmap)#class-map match-all SSH
R4(config-cmap)# match protocol ssh
R4(config-cmap)#class-map match-all TELNET
R4(config-cmap)# match protocol telnet
R4(config-cmap)#class-map match-all SNMP
R4(config-cmap)# match protocol snmp
R4(config-cmap)#class-map match-all ICMP
R4(config-cmap)#match protocol icmp

...and now, the policy-map to use these classes:
R4(config-cmap)#policy-map REMARK_ALL
R4(config-pmap)#description policy-map to place traffic into the appropriate traffic class
R4(config-pmap)# class EIGRP
R4(config-pmap-c)# set ip dscp cs6
R4(config-pmap-c)# class HTTP
R4(config-pmap-c)# set ip dscp ef
R4(config-pmap-c)# class SSH
R4(config-pmap-c)# set ip dscp af21
R4(config-pmap-c)# class TELNET
R4(config-pmap-c)# set ip dscp af41
R4(config-pmap-c)# class SNMP
R4(config-pmap-c)# set ip dscp cs3
R4(config-pmap-c)# class ICMP
R4(config-pmap-c)# set ip dscp cs2

Now, just enable the policy map in your ingress interface (VLAN10 in this lab), and you are done:
R4(config-if)#int vlan10
R4(config-if)# service-policy input REMARK_ALL

Pretty easy, huh? To utilize NBAR to identify traffic from various protocols, all you have to do is use the "match protocol <protocol>" statement inside a class-map and NBAR will identify the traffic for you! Keep in mind, however, that NBAR is deep-packet inspection -- it isn't just a simple match for port number and IP address -- and therefore, this functionality comes at a cost of CPU cycles (unless you are using a more recent model switch/router that off-loads the packet inspection to a separate processor). Consequently, if a simple ACL will meet your needs, it MAY be less resource intensive to use the ACL, as we've done in earlier labs. However, if you have a higher-powered router, NBAR certainly makes identifying and classifying traffic much easier on the network admin.

Note:
This is just a really simple example of NBAR, and doesn't even begin to scratch the surface of what NBAR can do for you. For example, when matching the HTTP protocol, you can create multiple classes based upon the host in the HTTP request and you can even use a regex to pattern-match within the URL. You can also extend NBAR with external files to add support for protocols that are not already built in.

For the sake of completeness, here is the rest of the QoS configuration on R4: R4(config)#class-map match-any PRIORITY
R4(config-cmap)# match ip dscp ef
R4(config-cmap)#class-map match-any CONTROL
R4(config-cmap)# match ip dscp cs6
R4(config-cmap)#class-map match-any CRITICAL
R4(config-cmap)# match ip dscp af41
R4(config-cmap)# match ip dscp cs3
R4(config-cmap)#class-map match-any ROUTINE
R4(config-cmap)# match ip dscp cs2
R4(config-cmap)# match ip dscp af21
R4(config-cmap)#class-map match-any SCAVENGER
R4(config-cmap)# match ip dscp CS1
R4(config)#policy-map EDGE_CHILD
R4(config-pmap)# description Core links
R4(config-pmap)# class PRIORITY
R4(config-pmap-c)# priority percent 25
R4(config-pmap-c)# class CONTROL
R4(config-pmap-c)# bandwidth percent 10
R4(config-pmap-c)# class CRITICAL
R4(config-pmap-c)# bandwidth percent 15
R4(config-pmap-c)# random-detect dscp-based
R4(config-pmap-c)# class ROUTINE
R4(config-pmap-c)# bandwidth percent 10
R4(config-pmap-c)# random-detect dscp-based
R4(config-pmap-c)# class SCAVENGER
R4(config-pmap-c)# bandwidth percent 1
R4(config-pmap-c)# random-detect dscp-based
R4(config-pmap-c)# class class-default
R4(config-pmap-c)# fair-queue
R4(config-pmap-c)# random-detect dscp-based
R4(config-pmap-c)#!
R4(config-pmap-c)#policy-map EDGE2CORE
R4(config-pmap)# description Parent policy for Edge-to-Core links
R4(config-pmap)# class class-default
R4(config-pmap-c)# shape average 10000000
R4(config-pmap-c)# service-policy EDGE_CHILD
R4(config-pmap-c)#!
R4(config-if)#int fa1/0
R4(config-if)# service-policy output EDGE2CORE
R4(config-if)#!
R4(config-if)#int fa3/0
R4(config-if)# service-policy output EDGE2CORE
R4(config-if)#!


Cisco Intro to QoS and CoS, Part 1 -- Identifying, Classifying and Marking Traffic

One problem found in any network of reasonable size is how to handle congestion. When there is more bandwidth available than clients require, there is (of course!) no problem. However, some types of network traffic -- such as streaming media -- are particularly bandwidth intensive, and often, very sensitive to variations in delay time. Consider, for example, HTTP traffic: an end user will most likely not notice the variation in transit times ("jitter") of the various packets that make up a requested download. On the other hand, a Voice over IP call is very sensitive to jitter. If the packets making up a Voice over IP call or videoconference don't arrive within a reasonably consistent timeframe, the call quality will suffer, and your end users will not be happy. Even worse, what would happen if your dynamic routing protocols cannot send updates and keepalive traffic in a timely fashion? Clearly, there needs to be a way to prioritize certain types of traffic to ensure that your clients receive sufficient bandwidth for their needs.

Consider the following network:


On the left-hand side of the network drawing, we have three Linux hosts, clients of the server on the right-hand side of the drawing. These three hosts will be in competition with each other for network resources, and because we are running OSPF (or routing protocol of your choice -- and yes, a dynamic routing protocol is overkill on a network this simple, but stay with me...) on R1 and R2, they will be competing with the network clients for bandwidth as well. Making this problem even more apparent, I have connected all of the PC's to the network via FastEthernet interfaces, but only connected R1 and R2 with a single T1 line. Due to the extreme mismatch between LAN and WAN capabilities, any one of the network clients can easily saturate the "WAN" link between R1 and R2. Don't believe me? Let's do some tests...

On the host "CentOS6," I have configured the "echo" service, and on the host "CentOS6_Clone," I have installed a network diagnostic tool called "tcpspray." With no other traffic on the network, let's use tcpspray to see how long it takes to send a flurry of data to the "CentOS6" server:

[root@centos6_clone ~]# tcpspray -e -n 1000 192.168.2.2
Received 1024000 bytes in 25.670507 seconds (30.955 kbytes/s)
Transmitted 1024000 bytes in 25.146152 seconds (39.768 kbytes/s)
[root@centos6_clone ~]#

Also on the host "CentOS6," I am running a web server, and one of the web pages hosted on the server is an image gallery that loads a 5x5 array of JPEG images (get your mind out of the gutter! They're pictures of motorcycles that inspired me while I was building my Cafe Racer project!). How would the tcpspray bandwidth be affected if one of the Knoppix clients was downloading the images to populate the web page at the same time? Let's find out...:

[root@centos6_clone ~]# tcpspray -e -n 1000 192.168.2.2
Received 1024000 bytes in 118.753065 seconds (8.421 kbytes/s)
Transmitted 1024000 bytes in 104.348504 seconds (9.583 kbytes/s)
[root@centos6_clone ~]#

Wow...it took almost five times as long to receive the replies, due to the congestion on the network! Fortunately, quality of service and/or class of service policies can allow us to prioritize certain kinds of traffic, as our network needs dictate.

Keep in mind that nothing is free, however. If we prioritize some kinds of traffic, that necessarily comes at the expense of other traffic. Therefore, you have to have a good idea what kinds of traffic cross your network, and how they should be balanced against each other...which is a good place for us to begin with our policy. Let's define the traffic that we want prioritized.

As I mentioned earlier, the parameters I used for tcpspray specified that we would use the "echo" service for our traffic. Let's include SSH as priority traffic so we can manage our routers (you are using SSH for network management, right?). As I also mentioned earlier, our network control traffic is also high priority. Too keep the discussion simple for now, we'll assume everything else is "best effort."

Now that we have a good idea how we want to prioritize our traffic, we need a way to have the router filter our priority traffic. If you think that sounds like a job for an Access Control List, you're exactly right:

ip access-list extended CORE_GOLD
 permit tcp any eq echo any
 permit tcp any any eq echo
!
ip access-list extended CUST_EF
 permit tcp any eq 22 any
 permit tcp any any eq 22
!

After identifying the priority traffic with an ACL, we then create "class maps" to map that traffic into different classes (kind of sounds like a tautology, sorry!):

class-map match-any CORE_CONTROL
 description Network control OSPF/BGP/etc
 match ip precedence 6
 match ip precedence 7
class-map match-any CUST-EF
 match access-group name CUST_EF
class-map match-any CORE_GOLD
 description Low-latency/priority queue
 match access-group name CORE_GOLD
!

We have now created three classes of priority traffic:

1. CORE_CONTROL, for our network control traffic,
2. CUST-EF, for high priority traffic ("expedited forwarding"), and
3. CORE_GOLD, for low-latency traffic.

Our next step is to use these maps to define how the different classes of traffic will be handled:

policy-map CUST-IN
 class CUST-EF
  set ip dscp ef
 class class-default
  set dscp cs3
policy-map CUST-Generic
 class CORE_GOLD
  priority percent 50
 class CORE_CONTROL
  bandwidth percent 10
 class class-default
  bandwidth percent 1
!

For our "CUST-EF" traffic, we have said that we want to place a special marking, "dscp ef," on the packets that match this traffic. This gives our "CUST-EF" traffic high priority on the network. For "CORE_GOLD" traffic, we have said that we want to give that traffic 50 per cent of the available bandwidth. For "CORE_CONTROL" traffic, we have said that we want to give such traffic 10 per cent of the available bandwidth (network control traffic is typically small-volume, bursty traffic, so it is not necessary to reserve a large amount of bandwidth for this traffic -- we just need to ensure that the traffic isn't starved for bandwidth during times of congestion). Notice that we also have another de facto class of traffic in the policy maps: class-default. This is for traffic that didn't match any of our access control lists. In the "CUST-IN" policy map, we mark it with "dscp cs3" -- a low priority traffic queue, and in the "CUST-Generic" policy map, we don't reserve any bandwidth for this class.

At this point, we have established the framework for our QoS/CoS configuration, but we haven't actually implemented it yet. Just as an access-control list used to filter certain kinds of traffic has to be applied to an interface before it is useful, so does a policy map:

interface Serial1/0
service-policy output CUST-Generic
!

We also will apply the "CUST-IN" policy map to the ingress interface, VLAN 10 on R1 and Fa0/0 on R2:

R1(config)#int vlan 10
R1(config-if)#service-policy input CUST-IN
R1(config-if)#

R2(config)#int fa0/0
R2(config-if)#service-policy input CUST-IN
R2(config-if)#

Now, let's run our tcpspray test again, while the Knoppix client reloads the web page with the 25 JPEG images:

[root@centos6_clone ~]# tcpspray -e -n 1000 192.168.2.2
Received 1024000 bytes in 78.483684 seconds (12.742 kbytes/s)
Transmitted 1024000 bytes in 69.900585 seconds (14.306 kbytes/s)
[root@centos6_clone ~]#

These results are still slower than our initial baseline test, with no congestion on the network. However, it is significantly better than the test with no QoS/CoS and a congested network.