Lesson 10 -- Spanning Tree

As the company has grown, management has hired more employees, and your poor old 2924 switch finally ran out of ports. However, it's worked reasonably well, so you found another one on E-Bay, and installed it in your main office server rack. You connected an Ethernet patch cable from Fa0/1 on the first switch to Fa0/1 on the second switch, then copied the config using the same trick you used on the routers: copy the config to a TFTP server, modify just the lines that were different (like hostname and IP address) and download the new config onto the new hardware. You also hired a well-meaning, but very, very green assistant to help you with the IT work. He's enthusiastic, but...

Anyway, you leave for lunch one day, and when you return, you find your assistant -- we'll call him George for now -- in the server room, laptop connected through a console cable to the switch, and frowning in thought as he scratches his head.

"What's the matter, George?" you ask.

George shrugs, taking a few seconds to compose his thoughts before replying. "While you were at lunch, I received a phone call from the manager of Customer Service, who was asking if there was anything we could do to speed up the Internet for some of the new employees on the new 2924 switch we installed last week. They said access to the file server was kind of sluggish, too. So, I came in here to the server room and connected a second Ethernet cable between the two switches, but it didn't seem to make any difference. I see both ports are up, but I can't figure out why they aren't seeing any improvement in speed*."

You chuckle, then have George run a couple of commands on his laptop:

lab2924b# conf t
lab2924b(config-if)# int fast0/1
lab2924b(config-if)# shut
lab2924b(config-if)# no shut
lab2924b(config-if)# exit
lab2924b(config)# exit
lab2924b# sho span


Spanning tree 1 is executing the IEEE compatible Spanning Tree protocol
Bridge Identifier has priority 32768, address 00d0.58dd.d180
Configured hello time 2, max age 20, forward delay 15
Current root has priority 32768, address 0003.e3e4.f880
Root port is 13, cost of root path is 19
Topology change flag not set, detected flag not set, changes 2
Times: hold 1, topology change 35, notification 2
hello 2, max age 20, forward delay 15
Timers: hello 0, topology change 0, notification 0


Interface Fa0/1 (port 13) in Spanning tree 1 is LISTENING
Port path cost 19, Port priority 128
Designated root has priority 32768, address 0003.e3e4.f880
Designated bridge has priority 32768, address 0003.e3e4.f880
Designated port is 13, path cost 0
Timers: message age 2, forward delay 0, hold 0
BPDU: sent 4, received 185
^C


George repeats the "sho span" a few times with the same results, then...:

lab2924b# sho span
<...snip...>
Interface Fa0/1 (port 13) in Spanning tree 1 is LEARNING
Port path cost 19, Port priority 128
Designated root has priority 32768, address 0003.e3e4.f880
Designated bridge has priority 32768, address 0003.e3e4.f880
Designated port is 13, path cost 0
Timers: message age 2, forward delay 11, hold 0
BPDU: sent 4, received 187
^C
lab2924b# sho span
<...snip...>
Interface Fa0/1 (port 13) in Spanning tree 1 is FORWARDING
Port path cost 19, Port priority 128
Designated root has priority 32768, address 0003.e3e4.f880
Designated bridge has priority 32768, address 0003.e3e4.f880
Designated port is 13, path cost 0
Timers: message age 1, forward delay 0, hold 0
BPDU: sent 5, received 194


George is thoroughly confused now. "What is with all of the 'listening,' 'learning' and 'forwarding' jazz?"

"Patience, grasshopper, and all will be explained. First run a few more commands..."

lab2924b# sho span int fa0/2
Interface Fa0/2 (port 14) in Spanning tree 1 is BLOCKING
Port path cost 19, Port priority 128
Designated root has priority 32768, address 0003.e3e4.f880
Designated bridge has priority 32768, address 0003.e3e4.f880
Designated port is 14, path cost 0
Timers: message age 1, forward delay 0, hold 0
BPDU: sent 1, received 4
^C


"You see, lucky for you, our switches are running a protocol called 'Spanning Tree.' Spanning tree is a way of making sure there are no loops when you connect multiple switches together. Suppose the switches were forwarding packets through both network interfaces where you've got the network cables connected. What would happen the first time a host on the network sent a Layer-2 broadcast message?"

George furrowed his brows. "I don't know, I guess."

"Think it through. Say my laptop on port 9 sends a broadcast. What does the 2924a switch do with that frame?"

"Oh, that's easy,?" George replied, brightening. "The switch will forward it out every port."

"Including port 1?" you ask.

Again George nods. "Yep."

"Which is connected to the 2924b switch," you add.

"Yep," George agrees again.

"And what does the 2924b switch do with the broadcast frame?"

"Since it's a broadcast, it sends it out every port except the one it received the frame on."

"Including port 2, where you connected the second Ethernet cable between the two switches?" you prompt.

The lightbulb of understanding finally fires in George's brain, and you suppress a laugh -- barely -- as his jaw suddenly falls to the floor. "Ohhhh..."

"Mmmm-hmmm."

"So the broadcast frame just continues to circle repeatedly between both switches, then."

"Exactly. It's called a 'broadcast storm' and, especially on older switches like these 2924's, it can bring your network to a screeching halt. It will easily max out the CPU on the switches so that you can't even log in to fix the problem. Sometimes the only way to stop a broadcast storm is to turn off one of the switches, or start pulling Ethernet cables until you see the switch start responding to telnet or pings again. Here is another command that is helpful for troubleshooting problems with Spanning Tree."

lab2924b# sho span brie
<...snip...>
Port                    Designated
Name Port ID Prio Cost Sts Cost Bridge ID Port ID
------- ------- ---- ---- --- ---- -------------- -------
Fa0/1 128.13 128 19 FWD 0 0003.e3e4.f880 128.13
Fa0/2 128.14 128 19 BLK 0 0003.e3e4.f880 128.14
Fa0/3 128.15 128 19 FWD 19 00d0.58dd.d180 128.15
Fa0/4 128.16 128 19 BLK 19 00d0.58dd.d180 128.16
Fa0/5 128.17 128 19 BLK 19 00d0.58dd.d180 128.17
Fa0/6 128.18 128 19 BLK 19 00d0.58dd.d180 128.18
Fa0/20 128.34 128 19 BLK 19 00d0.58dd.d180 128.34
Fa0/21 128.35 128 19 BLK 19 00d0.58dd.d180 128.35
Fa0/22 128.36 128 19 BLK 19 00d0.58dd.d180 128.36
Fa0/23 128.37 128 19 BLK 19 00d0.58dd.d180 128.37
<...snip...>


"This shows you which ports are forwarding, shown as 'FWD' or blocking, shown as 'BLK.' Now let's see what we can do to troubleshoot that bandwidth problem..."

Post-Script: Lest you think that this is an exaggerated tale, yes, this really happened to me. I was working for a service provider at the time, and a network admin working for one of my customers -- also a service provider -- called at the very end of the business day to request that I turn up a second port on one of my switches in a Metro Ethernet ring. The alarm bells went off in my head, but I foolishly ignored them, and, rather than inquiring about his plans for a second port on the same switch, I simply turned the port up and called it a day. When I returned to work the following day, I had a series of messages in my voice mail, wondering if I had turned the port up, if I had configured it properly, etc. I called my peer at the other service provider back, and started troubleshooting the problem with him. I finally figured out that he was trying to configure a redundant port between his switch and my switch, and really had to suppress a laugh when he admitted that he couldn't figure out why his port was going down when he plugged in his network cable. "Ummm...spanning tree?" I prompted as gently as I could. "Oh..." was his reply.