How to do troubleshooting for high CPU utilization

The show cpucommand output shows that the CPU utilization increases to 80% - 100%, whichmay cause slow CLI response, high ping loss or latency, and protocol flapping.

Note: If the CPU utilization is much higher than normal(for example, normal state range from 20% to 30%) but does not exceed 80% andservices are not affected, check whether the CPU utilization keeps increasing.It is recommended to leave briefly increasing (for example, for 1s to 2s)unhandled. Instead, keep monitoring it.

Possible Causes

1)       TheCPU receives a large number ofabnormal protocol packets (for example: packets with TTL value 1, ARP DoSattack packets, DHCP packets, IGMP packets,etc.).

2)       Protocol flapping frequently occurs, such as routing protocols, Spanning Tree Protocol(STP), Virtual Router Redundancy Protocol (VRRP), and Bidirectional ForwardingDetection (BFD), and therefore the protocols need to recalculate which consumedCPU resources.

3)       A physical loop occurs. A large number of packets are sent to the CPU of devicesthat do not support the CPU Protect Policy (CPP).

4)       Whena process is running, many CPUresources are consumed and cannot be released. (For example, the SNMP reads theMIB of the device, or the device prints many syslogs, or many users sendrequests for Web authentication simultaneously.)

5)       Afterreceiving some packets, the CPU responds with many protocol packets. (For example, a non-existing address in aVLAN is constantly scanned, causing the switch to send excessive ARP requests.)

1.1.3 Troubleshooting Procedure

Step 1 Check whether CPU processinformation is abnormal.

Step 2 Check whether CPP statistics are abnormal.

Step 3 Check whether IP scanning occurs.

Step 4 Check whether any loop occurs.

Step 5 Check whether protocol flapping occurs.

Step 6 Collect information and contact Ruijie technicalsupport.

Step1: Check whether CPU process information is abnormal.

Display CPU process information to check whether anyspecial process causes an increase in CPU utilization.

Procedure:

Excute the show cpu command forthree times(every 5 seconds) to display CPU utilization.

Check criterion:

Check whether any process occupies many CPU resources for 5 seconds in theshow cpu command output for threeconsecutive times. (The CPU utilization above 20% is usually defined high.)

If you are not clear about the definition of processes with high CPUutilization, contact Ruijie technical support.

Solution:

When the CPU process is running, a large number of CPU resources areconsumed and cannot be released, causing an increase in CPU utilization.

Solutions are listed as follows:

1)       Temporarily disable a function (Ensure this action does not affect the services and youhave confirmed with the customer): For example, ifservices become abnormal due to 100% CPU utilization of the Simple NetworkManagement Protocol (SNMP) process, disable the SNMP agent; if the CPUutilization of the rl_con process ishigh, disable console logging.

2)       Use the Access Control List (ACL) to filterprocess-related packets: For example, associate the ACL with Secure Shell (SSH)and SNMP to rectify high CPU utilization caused by abnormal SSH and SNMPpackets.

3)       If the CPU is high, check whether CPU Protect Policy(CPP) statistics are abnormal (CPP statistics usually change with CPUutilization). If yes, go toStep 2.

4)       If the CPU utilization persists at a high level, go to Step 2.

【Additional】

Common solutions to High CPU Utilization:

Commonsolutions to high CPU utilization are described as follows: (Note: If you arenot clear about the definition of processes with high CPU utilization, contact Ruijie technical support for more information.)

tnet/tnet6: a process for receiving IPv4/IPv6 packets. If theCPU utilization of this process is high, it is possibly because many ARP/NDpackets or IPv4/IPv6 packets are sent to the CPU.

Solution: Check CPPstatistics and configure the optimization policy according to Step 2.

ssp_flow_rx_task: a process forreceiving data packets. If the CPU utilization of this process is high, it ismostly because many data packets are sent to the CPU. Check CPP information toobtain the information about the packets sent to the CPU.

Solution: Check CPPstatistics and configure the optimization policy according to Step 2.

snmpd: SNMP process. If the CPU utilization of this process is high, it isusually because many SNMP packets are sent to the CPU or the SNMP consumes manyCPU resources when obtaining a MIB node of the device.

Solution:

1) Excute no enable service snmp-agent command totemporarily disable the SNMP agent.

//If SNMP isdisabled, the administrator may fail to manage devices. Besides, it may cause802.1X/WEB authentication failure. Therefore, confirm with the customer beforethis operation.

2) Use the ACL to allowaccess of legal SNMP servers only.

Configure the ACL to allowaccess of specified SNMP servers only:

ip access-list standardtrust-snmp

10 permit 172.18.252.00.0.0.255

20 permit 172.18.126.00.0.0.255

Apply the ACL to SNMP:

Ruijie(config)#snmp-servercommunity ruijie rw trust-snmp

rl_con: a process for console logging. If the CPU utilization of this processis high, it is usually because many logs are generated.

Solution:

Run the no logging console command to disable the console logging.

For example：Ruijie(config)#no loggingconsole

Or run the logging rate-limit all 10 exceptemergencies command to limit the output rate of the log.

For example：Ruijie(config)#logging rate-limit all 10 except emergencies

pimd/pim6d: a process for receiving IGMP/PIM multicastpackets. If the CPU utilization of this process is high, it is usually becausemany unknown multicast packets are sent to the CPU.

Solution:
1) Modify the rate limits of the unknown multicast packets which send to CPU.
Ruijie(config)#cpu-protect type unknown-ipmcpps 30

Ruijie(config)#cpu-protecttype unknown-ipmcv6 pps 30

2) Deploy multicast optimization policy , configure an ACL foravoiding multicast source spoofing (to filter illegal multicast packets fromenduser)

Thereis no security policy for multicast by default. Currently, a large amount ofsoftware installed on a PC can send multicast packets, which may causemulticast packets flooding on the entire network and consume multicast resourceentries and CPU resources of the switch. Therefore, deploy the multicastoptimization policy on all multicast-enabled networks.

Usea specific ACL to allow only multicast packets from valid sources to validdestinations to pass through. (Apply the ACL on the Trunk port or SVI.) Whetherin PIM-DM or PIM-SM mode, deploy the ACL on the entire network.

Optimizationapproach (e.g., IPv4):

Ruijie(config)#ipaccess-list extended deny_mc_source

Ruijie(config-ext-nacl)#10permit igmp any any //Allow all IGMP packets to pass through.

Ruijie(config-ext-nacl)#20permit ip 219.229.134.0 0.0.0.255 239.202.0.00.0.255.255

//Allowmulticast packets from legal sources to legal destinations to pass through. (The legal multicast source and destination should be according to the customer's network requirement.)

Ruijie(config-ext-nacl)#30deny ip any 224.0.0.0 15.255.255.255

//Denyall multicast packets.

Ruijie(config-ext-nacl)#40permit ip any any

//Allowall IPv4 packets.

Tointegrate the above ACL with the old ACL applied on Trunk port or SVI, add ACE10-30 to the old ACL.

If youare not clear about ACL integration, contact Ruijie technical support forassistance.

ef_res: A process for neighbor resolution. If the CPU utilization of thisprocess is high, it is usually because IP scanning or frequent ARP aging anddeletion occurs (mostly because STP receives TC packets).

Solution: If IP scanning occurs, enable IP-guard for theswitch.

Step1 Ruijie(config)#nfpp   // Run the nfpp command to enter the NFPPconfiguration mode.

Step2 Ruijie(config-nfpp)#ip-guardenable  // Run the ip-guard enable command to enable anti-scanning globally. It isenabled by default.

Step3 Ruijie#shownfpp ip-guard hosts  //Runthe show nfpp ip-guard hosts commandto display the attacked hosts.

Step 4 Ruijie(config-nfpp)#ip-guardisolate-period  //Run the ip-guard isolate-period command to isolatethe attacked hosts.

NFPP-based hardware isolation is disabledby default. If it is required, confirm with the customer before enabling it inthe case of IP scanning. If PCs are found as attacker will be isolated, thecommunication fails (for 10 minutes by default).

Totroubleshoot the attack source, you need to identify the attack source port(based on the IP address/ARP/MAC address/port mappings or mirror traffic).

Approach to Identifying Attack Source:

1)      Display logs to check whether the syslog contains abnormal packet sources(IP address and MAC address).

2)      Obtain source addresses (IP address and MAC address) of the abnormalpackets by mirroring traffic on suspicious ports. If too many packets gothrough the port, use flow-based mirroring or Wireshark to filter the packets.

3)      Check the ARP table and MAC address table of the switch to identify theport sending the abnormal packets.

If theSTP topology frequently changes, enable tc-guard or tc ignore on the portreceiving TC packets and then identify the source of TC packets to rectify thefault. After the network is recovered, remove the temporary optimizationconfiguration.

Ruijie(config-if)#spanning-tree tc-guard

Ruijie(config-if)#spanning-tree ignore tc  

Step2: Check the statusof protocol packets which sent to CPU

1.Run the show cpu-protect mbcommand for three to five times (every 5 seconds)

Check Standard:

1. Check whether there is any case of high packet drop with constantgrowth. For example, in a fault case, the number of dropped packets with TTLvalue 1 keeps increasing, as shown in the following figure.

·       If many types of protocolpackets are dropped with an increasing rate and the RX rate (pps) is verylarge, it is usually because a loop occurs. Go toStep 4 totroubleshoot the fault.

·       The fact that some types ofprotocol packets are dropped with an increasing rate proves that a large numberof abnormal packets flood the network. Optimize the network according to theoptimization policy in the following solution and check the result.

2. Check whether there is any case of high RX rate but less drop in thecpp statistics.

To cater to networks of different sizes, the default RX rate limit is setto an empirical value . For example, the default rate limit of unknownmulticast packets is 128 pps.

The rate of unknown multicast packets is very small (for example, 3 pps)on a healthy network. If it ranges from 30 pps to 100 pps, the CPU utilizationwill also increase even though no drop occurs.

·       The fact that some types ofprotocol packets with an increasing rate proves that a large number of abnormalpackets flood the network. Optimize the network according to the optimizationpolicy in the following solution and check the result.

·       If a CPU process is abnormalwithout CPP statistics exception (or such statistics cannot be independentlyanalyzed), go to Step 4.3 for troubleshooting.

Notes

Run theshow cpu-protect summary command todisplay the default RX rate limit of a device.

Solution:

1. Adjust the RX rate limit based on the protocol type.

If the rate limit is too small, packet exchanges will be affected. It isrecommended to adjust the RX rate limit as follows:

1) First reduce the rate limit by 50%. For example, if the originaldefault rate limit is 128 pps, it is recommended to change it to 64 pps andcheck for optimization. After the CPU utilization drops to below 50%, it isrecommended to identify the root cause rather than adjust the rate limit again.CPP rate limiting is only a preventive measure, and the solution is to identifyabnormal packet sources.

2) Some types of packets are too important to reduce the original ratelimit, such as ARP packets, BPDU packets, and OSPF packets. Thereason is that setting a rate limit too small may cause protocol packet drop.

Approach to Adjustingthe CPP Rate Limit:

Ruijie(config)#cpu-protect type local-ipv4 bandwidth 50  //Take the command for example

If the fault persists, go to Step 3.

Common Solutions:

Common causes and solutions to high RX rate are listed asfollows:

1. nd-snp-ns-na: After aswitch receives routing/neighbor resolution packets from an IPv6 host, its CPUutilization is high. It is usually because a fault occurs to the host such asvirus attack.

Solution:Reduce the rate limit of the preceding packets and identify the packet sources(see the following description).

Forexample:  

Ruijie(config)#cpu-protecttype nd bandwidth 200

Approachto identifying abnormal packet sources:

1)      Display logs to check whether the syslog contains abnormal packet sources(IP address and MAC address).

2)      Obtain source addresses (IP address and MAC address) of the abnormalpackets by mirroring traffic on suspicious ports. If too many packets gothrough the port, use flow-based mirroring or Wireshark to filter the packets.

3)      Check the ARP table and MAC address table of the switch to identify theport sending the abnormal packets.

2. unknown-ipmc/unknown-ipmcv6:After aswitch receives abnormal multicast packets, its CPU utilization is high .

Solution:Reduce the rate limits of the preceding packets and configure an ACL for antimulticast source spoofing.

Ruijie(config)#cpu-protecttype unknown-v4mc bandwidth 30

Ruijie(config)#cpu-protecttype unknown-v6mc bandwidth 30

Configurean ACL (to filter invalid multicast packets on the PC)

Thereis no security policy for multicast by default. Currently, a large amount ofsoftware installed on a PC can send multicast packets, which may causemulticast floods on the entire network and consume multicast resource entriesand CPU resources of the switch. Therefore, deploy the multicast optimizationpolicy on all multicast-enabled networks.

Multicastsource anti-spoofing (implementation process): Usea specific ACL to allow only multicast packets from valid sources to validdestinations to pass through. (Apply the ACL on the Trunk port or SVI.) Whetherin PIM-DM or PIM-SM mode, deploy the ACL on the entire network.

Optimizationapproach:

Ruijie(config)#ipaccess-list extended deny_mc_source

Ruijie(config-ext-nacl)#10permit igmp any any //Allow all IGMP packets to pass through.

Ruijie(config-ext-nacl)#20permit ip 219.229.134.0 0.0.0.255 239.202.0.00.0.255.255

//Allowmulticast packets from valid sources to valid destinations to pass through. (Update the ARP entry according to thecustomer's requirement.)

Ruijie(config-ext-nacl)#30deny ip any 224.0.0.0 15.255.255.255

//Denyall multicast packets.

Ruijie(config-ext-nacl)#40permit ip any any

//Allowall IPv4 packets.

Tointegrate the above ACL with the old ACL applied on Trunk port or SVI, add ACE40 to the old ACL.

If youare not clear about ACL integration, contact Ruijie technical support forassistance.

3. ip4-packet-other/ip6-packet-other: IPv4 packets not specified in the CPP are usually scanning packets.

Solution:Reduce the rate limits of the preceding packets and identify the packet sources(See above).

Ruijie(config)#cpu-protecttype ip4-packet-other bandwidth 50

Ruijie(config)#cpu-protecttype ip6-packet-other bandwidth 50

Orconfigure NFPP IPv6 guard as follows:

Ruijie(config)#nfpp

Ruijie(config-nfpp)#defineipv6_guard

Ruijie(config-nfpp-define)#matchetype 0x86dd

Ruijie(config-nfpp-define)#global-policyper-src-ip 10 20

Ruijie(config-nfpp-define)#exit

Ruijie(config-nfpp)#defineipv6_guard enable

Step3: Check whetherIP scanning occurs.

1.    Run the show arp and show arpcounter commands.

Ruijie#show arp

Internet  125.39.113.32    <--->     <Incomplete>    arpa  GigabitEthernet 0/22

Internet  125.39.113.35    <--->     <Incomplete>    arpa  GigabitEthernet 0/22

Internet  125.39.113.37    <--->     <Incomplete>    arpa  GigabitEthernet 0/22

Internet  125.39.113.38    <--->     <Incomplete>    arpa  GigabitEthernet 0/22

Internet  125.39.113.34    <--->     <Incomplete>    arpa  GigabitEthernet 0/22

Ruijie#show arp counter

Count of static entries:  1

Count of dynamic entries: 3486 (complete:3607  incomplete: 221)

Total:                    3487

CheckCriterion:

1.Check whether there are many incompleteARP entries containing continuous IP addresses.

If yes, IP scanning occurs in the network.In this case, it is recommended to connect one trunk interface to a PC tocapture packets and check whether the switch sends a large number of ARPrequest packets.

Solution:

If IP scanning occurs, enable IP-guard forthe switch.

IP-guard is enabled by default. If it isdisabled, re-enable it.

·         Step1Run the Ruijie(config)#nfpp command  //enter the NFPP configuration mode.

·         Step2Run the Ruijie(config-nfpp)#ip-guard enable // enable anti-scanning globally. It is enabled by default.

·         Step3Run the Ruijie#show nfpp ip-guard hosts // display the attacked hosts.

NFPP-based hardware isolation is disabledby default. If it is required, confirm with the customer before enabling it inthe case of IP scanning. If PCs are found attacked, the communication fails(for 10 minutes by default).

To troubleshoot the attack source, you needto identify the attack source port (based on the IP address/ARP/MACaddress/port mappings or SPAN packet capturing).

1). Display logs to check whether thesyslog contains abnormal packet sources (IP address and MAC address).

2). Obtain source addresses (IP address andMAC address) of the abnormal packets by mirroring traffic on suspicious ports.If too many packets go through the port, use flow-based mirroring or Wiresharkto filter the packets.

3).Check the ARP table and MAC addresstable of the switch to identify the port sending the abnormal packets.

If the fault persists, go to Step 4.

Step4: Check whetherany loop occurs.

Identify a loop according to the followingsymptoms:

1. Many interface indicatorson the switch rapidly flash at the same time.

2. After connectedto the switch through a Trunk port, the PC receives a large number of packetsrepeatedly.

3. The show interfaces counters rate command outputshows that the traffic on the Trunk port is very high and consistent.

4. When the show mac-address-table command is run forthree consecutive times, the MAC address may move to different ports.

Check Standard:

1. The flashingfrequency of indicators is related to the transmission rate of the interface.If a loop occurs, Layer-2 ports will transmit and receive a large number ofpackets, and the interface indicators will rapidly flash with the samefrequency.

2. A loop meansthat broadcast packets are flooded in a VLAN. After accessing a Trunk port, thePC receives packets from the loop. If a large number of packets are capturedrepeatedly, a loop must have occurred.

3. Check whetherthe broadcast traffic is abnormal on a port. If the broadcast traffic suddenlyincreases to a volume higher than the unicast traffic, a loop must haveoccurred.

4. Check whetherthe MAC address of any port frequently changes. If yes, a loop may haveoccurred.

Solution:

In the network environment where a loopoccurs, eliminate the loop by automatically blocking loop ports throughprotocol (Loop avoidance) or manually disabling ports. Perform the followingoperations:

1.     Enable loop detection on the access-layer switch.

Optional solution 1: Enable RLDP loop detection and check whether the portis automatically disabled and shut down.

Ruijie(config)#rldpenable  //Globally enable RLDP.

Ruijie(config-if-range)#rldp port loop-detect shutdown  //Enable rldp on the Access port.

Ruijie(config-if-range)#spanning-tree bpdufilter enable  //Enable BPDU filtering on an STP port toprevent the port from sending or receiving BPDUs.

Optional Solution 2: Enable PortFast and BPDU guard.

Ruijie(config-if-range)#spanning-treeportfast   //EnableSTP loop guard on the Access port.

Ruijie(config-if-range)#spanning-treebpduguard enable  //Enable STP loop guard on the Access port.

2.     Enable broadcast storm control on the Access port to mitigatethe broadcast storm problem caused by the loop. (Not required on the uplinkport.)

Ruijie(config-if-range)#storm-controlbroadcast

3.       Shut down or unplug cables from suspicious ports one byone to see whether the problem is solved.

If the fault persists, go to Step 5.

Step5: Check whether protocol flapping occurs.

1. Run the show log and show ip ospf nei commands to checkwhether the OSPF/BFD neighbor relationship is frequently down.

2. Run the show spanning-treesummary command (every 5 seconds) three times to check whether the topologychanges frequently.

Check Standard:

1. Check whether the neighbor relationship based on a routing protocol(OSPF/BFD) flaps up or down and whether logs similar to the following aredisplayed:

OSPF logs

*Aug  9 10:26:10: %LINEPROTO-5-UPDOWN: Lineprotocol on Interface GigabitEthernet 0/24, changed state to up.

*Aug  9 10:26:20: %OSPF-5-ADJCHG: Process 10, Nbr10.10.10.1-GigabitEthernet 0/24 from Down to Init, HelloReceived.

*Aug  9 10:26:50: %OSPF-5-ADJCHG: Process 10, Nbr10.10.10.1-GigabitEthernet 0/24 from Exchange to Full, ExchangeDone.

*Aug  9 10:27:44: %OSPF-5-ADJCHG: Process 10, Nbr10.10.10.1-GigabitEthernet 0/24 from Full to Init, 1-WayReceived.

*Aug  9 10:27:55: %OSPF-5-ADJCHG: Process 10, Nbr10.10.10.1-GigabitEthernet 0/24 from Exchange to Full, ExchangeDone.

BFD logs

140092:*Sep 29 19:21:23: %BFD-6-SESSION_STATE_UP: BFD session to neighbor10.242.237.82 on interface TenGigabitEthernet 0/21 is up

140093:*Sep 29 19:21:35: %BFD-6-SESSION_STATE_DOWN: BFD session to neighbor10.242.237.82 on interface TenGigabitEthernet 0/21 has gone down.Reason: Control Detection Time Expired

If there are many logs about neighbor relationship flapping, the protocolfrequently goes up or down.

2. Check whether STP frequently receives topology changes. Focus on the countand duration of topology changes (information in red font).

Ruijie#showspanning-tree     

StpVersion: MSTP <---Current STP mode

SysStpStatus: ENABLED

MaxAge: 20

HelloTime: 2

ForwardDelay: 15

BridgeMaxAge: 20

BridgeHelloTime: 2

BridgeForwardDelay: 15

MaxHops:20

TxHoldCount: 3

######mst 0 vlans map : ALL       <---All VLANs are mapped to Instance 0.

BridgeAddr: 001a.a915.5984        <---Local MAC address

Priority:4096

TimeSinceTopologyChange : 0d:0h:0m:12s  //Check whether the last topology lasts for a short time.

TopologyChanges : 1182 //Check whether the count of topology changesrapidly increases.

DesignatedRoot: 1000.001a.a915.5984  <---MAC addressof Root Bridge for all regions

RootCost: 0

RootPort: 0                          <---Root portselected by the local host in Instance 0. If the Root Bridge is the local host,this value is 0.

CistRegionRoot: 1000.001a.a915.5984  <---MACaddresses of the Root Bridge for Instance 0 in the local region.

If the topology change count rapidly increases and the last change lastsfor a short time, TC packets (STP topology changes) must have been generated inthe network. For details about troubleshooting, see the  "Troubleshooting STP"Section

Solution:

1. If a protocol flapping occurs due to the link fault on a port, you cantemporarily shut down the port or disable the dynamic routing protocol to seewhether the fault is rectified. If not, troubleshoot the link fault on theport.

2. If the topology frequently changes, you can enable TC guard or TCignorance on the port receiving TC packets and then identify the source of TCpackets (described in Step4). After the network isrecovered, remove the makeshift configuration.

tc-guard:

Ruijie(config-if)#spanning-tree tc-protection tc-guard  //Enable TC guard on theport receiving TC packets. The port is connected to an access/convergence switch.

tc ignore:

Ruijie(config-if)#spanning-tree ignore tc  (an optimizationto TC guard) // Enable TC ignorance on the port receiving TC packets. The portis connected to an access/convergence switch.

3. If the fault persists, go to Step 6 to collect information and thencontact Ruijie technical support for assistance.

Step6: Collect information for further analyzing

1) Collect key information about high CPUutilization.

show debugging

show cpu

show cpu-protect mb

show cpu-protect

show cpu

show cpu-protect mb

show cpu-protect

Ruijie#debug su   

Ruijie(support)#tech-support package

exit

show cpu

show cpu-protect mb

show cpu-protect

--------------

2) Collect auxiliary information about highCPU utilization.

show version

show version slots

show run

show log

show ip interface brief

show interface status

show interface counter

show interfaces counters rate

show interface count summary

show interfaces counters rate

show interface count summary

show arp counter

show arp detail

show mac-address-table

show spanning-tree

show spanning-tree summary

show vrrp brief

show cpu-pro

show cpu-pro mb

show cpu-protect slot X  //X indicates the slot number. Forexample, run show cpu-pro slot 4 todisplay information about slot 4 on a standalone S8600E series switch; Run show cpu-pro slot 2/4 to displayinformation about slot 4 on chassis 2 of an S8600E Virtual Switching Unit(VSU).

show arp counter

show arp detail

show mac-address-table

3) Capture packets on the network for 2 to 3 minutes, and enable portmirroring, with the mirror source being a typical service port. (It isrecommended that packets be captured every 10 seconds. If the traffic is verylarge, capture the packets for 1 to 2 seconds multiple times)

Configurationon switches

Ruijie(config)#monitor session 1 source interface gigabitEthernet0/Y      //gigabitEthernet 0/Y indicates a serviceinterface. Multiple service interfaces need to be selected.

Ruijie(config)#monitor session 1 destination interface gigabitEthernet0/X    //gigabitEthernet0/X indicates an interface connected to the PC.

þ Gathering troubleshooting information and then submit a case in theservice portal(case.ruijienetowrks.com),and contact the Ruijie Technical Assistance Center (TAC) for assistance andfurther instructions.