Troubleshooting Excessive Topology Changes Causing Flooding

In an STP-enabled network with dual core switches,the originally stable network becomes unstable. The Console prints the log"topochange:topology is changed" regularly. The network traffic isunstable and intermittent.

Possible causes

1). The core switches encounter STP protocol flapping,causing the STP topology continually changes.

2). Certain switches in the STP network malfunction due to hardware faultsor cabling problems, causing the constant change of interface status

3). Interfaces on certain switches in the STP network experience frequentrole change.

4). The core switches receive TC packets.

Troubleshooting Procedure

Before youstart to troubleshoot, you must obtain this information:

·       An actual topology diagram that details all of theswitches and bridges

·       Their corresponding (interconnecting) port numbers

·       STP configuration details, such as which switch is theroot and backup root, which links have a non-default cost or priority, and thelocation of blocking ports

Generally, troubleshooting involves these steps(depending on the situation, some steps may not be necessary):

Step 1: Check whether the core switches encounter STP protocol flapping.

Step 2: Check whether the switch interfacestatus changes.

Step 3: Check whether the switch roleschange.

Step 4: Check whether the core switchesreceive TC packets.

Step 5: Collect fault information andsubmit cases to Ruijie Service Portal.

Step1: Check whether the core switches continuously encounter STP protocol flapping.

STP flapping usually has the largestimpact on core switches. Therefore, check the core switches first. Perform thefollowing operations on the core gateway:

Run the showspanning-tree command to check whether the switches continuously encounterSTP protocol flapping.

Ruijie#show spanning-tree

StpVersion : MSTP

SysStpStatus : ENABLED

MaxAge : 20

HelloTime : 2

ForwardDelay : 15

BridgeMaxAge : 20

BridgeHelloTime : 2

BridgeForwardDelay : 15

MaxHops: 20

TxHoldCount : 3

PathCostMethod : Long

BPDUGuard : Disabled

BPDUFilter : Disabled

###### mst 0 vlans map : 1, 3-4094  

BridgeAddr : 00d0.f822.3caa

Priority: 32768

TimeSinceTopologyChange : 0d:3h:43m:21s

TopologyChanges :3

DesignatedRoot : 8000.00d0.f822.3caa

RootCost : 0

RootPort : 0

CistRegionRoot : 8000.00d0.f822.3caa

CistPathCost : 0

If the value of TopologyChangescontinuously increases, it indicates that the topology flaps more and morefrequently. When the topology change occurs on the local switch or otherswitches in the network, the MAC address table of the local switch will becleared.

Common causesare as follows: Switch interfaces undergo status change between Up and Down,role change such as from RP to Alternate, and receive TCN packets or TC BPDUpackets from other switches.

Note: Clearing MAC address tables of layer-2 switches has no impact on users'communication, because unknown unicast packets are flooded as broadcast withina VLAN after the clearance of MAC address tables. However, it will burden theCPU because the switches must re-learn the MAC addresses. Clearing MACaddresses of layer-3 switches may interrupt the layer-3 forwarding for 2 to 10seconds (because the layer-3 forwarding function needs to delay ARP learning)and bring massive MAC address learning (because layer-3 switches are often usedas gateways which learn many MAC addresses). Frequent clearing of MAC addresstables will severely interrupt users' communication, especially the streamsforwarded by the layer-3 switches.

2. Run the show log command to check whether the"topochange:topology is changed" log exists. If yes, it indicatesthat STP flaps as the local topology changes. If not, the topology change maybe caused by TC packets.

&Note: The meaning of "topochange:topology is changed" is that thecurrent switch topology changes. To simplify, port roles change, for example,changing from Disabled to DR, or from Alternateto DR. Possible causes for the topology changes are as follows: An interface changesfrom the forwarding state to the discarding state (forwarding->discarding),or from the discarding state to the forwarding state(discarding->forwarding). There are two possibilities for displaying thisinformation: One is that the link status of certain interface changes. Theinformation will be displayed when the interface goes up or down. The other oneis that no interface link status changes but STP Algorithm changes the roles ofcertain interfaces, for example changing from RP to Alternate.

Step2: Check whether the switch interface status changes.

TC should be a rare event in awell-configured network. When a link on a switch port goes up or down,there is eventually a TC, once the STP state of the port is changing to orfrom forwarding. When the port is flapping, this would cause repetitiveTCs and flooding.

1.       Check whether the status of interfaces on the core switches frequentlychange (port goes up or down) and whether the interfaces are directly connectedto PCs or single-link switches. If yes, configure the BPDU filter or Portfaston the switches to rectify the fault for normal status change. Ports with theSTP portfast feature enabled will not cause TCs when going to or fromthe forwarding state. The configuration of portfast on all end-deviceports (such as printers, PCs, and servers) should limit TCs to a low amount andis highly recommended.

2.       Troubleshoot links frequently go up or down with other cause, include bad transceivers or GigabitInterface Converters (GBICs), cabling issues, or hardware failures on the port,the linecard, or the Supervisor engine).

Step3: Check whether the switch roles change.

1. Check whether the switch rolesfrequently change. Run the showspanning-tree interface vidcommand.

The following information is displayedby using the show spanning-treeinterface command onGi0/24 for example:

Switch#show spanning-tree int gi 0/24

PortAdminPortFast : Disabled

PortOperPortFast : Disabled

PortAdminLinkType : auto

PortOperLinkType : point-to-point  <-----p2pindicates full duplex whereas sharedindicates half duplex.

PortBPDUGuard : disable

PortBPDUFilter : disable

###### MST 0 vlans mapped :1, 3-4094

PortState : forwarding             <----Forwarding state of thecurrent interface in Instance 0.

PortPriority : 128

PortDesignatedRoot : 8000.00d0.f822.3caa   <----General root bridge (not a domainroot bridge) learned by this interface in Instance 0.

PortDesignatedCost : 0

PortDesignatedBridge :8000.00d0.f822.3caa   <----Uplink bridge of this interface inInstance 0.

PortDesignatedPort : 8018: The ID of the uplinkinterface of this interface in Instance0. This value consists of 16 bits; ofwhich the first four bits indicate the priority 16 of the uplink interface andthe last 12 bits indicate the index of the uplink interface. They are in thehexadecimal format. Use the showinterface command to display the index of an interface. In the example,this value indicates that the priority of the uplink interface is 128 (decimal)and the index of the interface is 24 (decimal).

PortForwardTransitions : 2

PortAdminPathCost : 200000

PortOperPathCost : 200000

PortRole:designatedPort----à Role of thisinterface in instance0.

###### MST 1 vlans mapped :2

PortState: forwarding: Forwarding state of thisinterface inInstance1.

PortPriority : 128

PortDesignatedRoot: 0001.00d0.f822.3caa         ----àRootbridge of this interface in Instance1.

PortDesignatedCost : 0

PortDesignatedBridge :0001.00d0.f822.3caa      ---à Uplinkroot bridge of this interface in Instance1.

PortDesignatedPort : 8018

PortForwardTransitions : 1

PortAdminPathCost : 200000

PortOperPathCost : 200000

PortRole:designatedPort----àRole of this portin instance1.

2. Ifinterface roles change frequently, it is recommended that you run the debug mstp topology command to enabledebugging and collect related information for analysis and further faultlocating.

  Debugging operations cause risks (Theworst case is to restart the switch for recovery). Perform debugging only afterinforming customers of the risks and get them accepted. It is recommendeddebugging at low-traffic periods (Be more cautious when dealing with coreswitches). If packet capturing is also required for troubleshooting, rememberto collect information by debugging and packet capturing at the same time..

Debug mstptopochange:

Debug informationabout MSTP topology changes. Use the debugging command to trace MSTP topologychanges on the current switch. Traceable topology changes include role andforwarding state changes of interface, MAC address table clearance, instancechanges, and the interfaces that receive TC BPDU packets.

For example, connect a PCdirectly to the Gi0/24 interface of a switch that is enabled with MSTP. Whenthe debug mstp topochange command isexecuted, the following information is displayed:
1:14:58:56 S2924G: %7:mstp topo:GigabitEthernet 0/24  select role DesignatedPort in msti 0!<-----This interface is selected as the DPin Instance0.
1:14:58:56 S2924G:%7:mstp topo:ssp port state notify.set port GigabitEthernet0/24 state 2,mstid 0  <-----Set theforwarding state of the interface in Instance0 to 2, which indicates thelearning state.
1:14:58:56 S2924G:%7:mstp topo:ssp port state notify.set port GigabitEthernet0/24 state 2,mstid 1  <------Theforwarding state of this interface in Instance1 is learning.
1:14:59:11 S2924G:%7:mstp topo:port[GigabitEthernet 0/24] change statediscarding to forwarding  <-----Thisinterface enters the forwarding state.
1:14:59:11 S2924G:%7:mstp topo:ssp port state notify.set port GigabitEthernet0/24 state 3,mstid 0  <-----3indicates the forwarding state.
1:14:59:11 S2924G: %7:mstp topo:port[GigabitEthernet 0/24] change statediscarding to forwarding
1:14:59:11 S2924G: %7:mstp topo:ssp port state notify.set port GigabitEthernet0/24 state 3,mstid 1
1:14:59:12 S2924G: %7:2007-1-12 10:28:40 topochange:topology is changed  <-----Topology changes are displayed here.

Step4: Check whether the core switches receive TC packets.

1. Check whether the network flapping on coreswitches is due to TC BPDU packets received from other switches. If theflapping occurs not on the local switch, the TC BPDU packets received are thefuse. There are two solutions to this case:

Perform interface mirroring on all accessinterfaces of the core switches one by one to check which of them receive TCBPDU packets.

Run the debugmstp topology command to check which interfaces receive TC BPDU packets inthe debugging information.

3.      By port mirroring or the debug mstp topology command, if the TC BPDU packet are confirmedfrom non-core switches, track the involved interfaces on these switchesfurther. Repeat the methods (steps 1 to 4) to locate the source interfaces.Then, enable the BPDU filter on the STP-enabled switches or troubleshoot linkfaults.

Debugging operations cause risks (The worstcase is to restart the switch for recovery). Perform debugging only afterinforming customers of the risks and get them accepted. It is recommendeddebugging at low-traffic periods (Be more cautious when dealing with coreswitches). If packet capturing is also required for troubleshooting, rememberto collect information by debugging and packet capturing at the same time..

Step5: Collect fault information and submit cases toRuijie Service Portal.

If the fault is not rectified through the preceding steps, collect thefollowing information and the analysis information obtained in steps 1 to 4.Submit cases to Ruijie Service Portal for further handling.

Collect thenetwork topology information on all STP-enabled devices.

show runnig-config

show spanning-tree

show spanning-treesummary

show spanning-tress interface xx (Switch interface)

show version

show version slot //Chassis switchslot