How to do troubleshooting for high CPU utilization - Ruijie Community

The show cpucommand output shows that the CPU utilization increases to 80% - 100%, whichmay cause slow CLI response, high ping loss or latency, and protocol flapping.

Note: If the CPU utilization is much higher than normal(for example, normal state range from 20% to 30%) but does not exceed 80% andservices are not affected, check whether the CPU utilization keeps increasing.It is recommended to leave briefly increasing (for example, for 1s to 2s)unhandled. Instead, keep monitoring it.

1) TheCPU receives a large number ofabnormal protocol packets (for example: packets with TTL value 1, ARP DoSattack packets, DHCP packets, IGMP packets,etc.).

2) Protocol flapping frequently occurs, such as routing protocols, Spanning Tree Protocol(STP), Virtual Router Redundancy Protocol (VRRP), and Bidirectional ForwardingDetection (BFD), and therefore the protocols need to recalculate which consumedCPU resources.

3) A physical loop occurs. A large number of packets are sent to the CPU of devicesthat do not support the CPU Protect Policy (CPP).

4) Whena process is running, many CPUresources are consumed and cannot be released. (For example, the SNMP reads theMIB of the device, or the device prints many syslogs, or many users sendrequests for Web authentication simultaneously.)

5) Afterreceiving some packets, the CPU responds with many protocol packets. (For example, a non-existing address in aVLAN is constantly scanned, causing the switch to send excessive ARP requests.)

Display CPU process information to check whether anyspecial process causes an increase in CPU utilization.

Excute the show cpu command forthree times(every 5 seconds) to display CPU utilization.

Check whether any process occupies many CPU resources for 5 seconds in theshow cpu command output for threeconsecutive times. (The CPU utilization above 20% is usually defined high.)

If you are not clear about the definition of processes with high CPUutilization, contact Ruijie technical support.

When the CPU process is running, a large number of CPU resources areconsumed and cannot be released, causing an increase in CPU utilization.

1) Temporarily disable a function (Ensure this action does not affect the services and youhave confirmed with the customer): For example, ifservices become abnormal due to 100% CPU utilization of the Simple NetworkManagement Protocol (SNMP) process, disable the SNMP agent; if the CPUutilization of the rl_con process ishigh, disable console logging.

2) Use the Access Control List (ACL) to filterprocess-related packets: For example, associate the ACL with Secure Shell (SSH)and SNMP to rectify high CPU utilization caused by abnormal SSH and SNMPpackets.

3) If the CPU is high, check whether CPU Protect Policy(CPP) statistics are abnormal (CPP statistics usually change with CPUutilization). If yes, go toStep 2.

Commonsolutions to high CPU utilization are described as follows: (Note: If you arenot clear about the definition of processes with high CPU utilization, contact Ruijie technical support for more information.)

tnet/tnet6: a process for receiving IPv4/IPv6 packets. If theCPU utilization of this process is high, it is possibly because many ARP/NDpackets or IPv4/IPv6 packets are sent to the CPU.

Solution: Check CPPstatistics and configure the optimization policy according to Step 2.

ssp_flow_rx_task: a process forreceiving data packets. If the CPU utilization of this process is high, it ismostly because many data packets are sent to the CPU. Check CPP information toobtain the information about the packets sent to the CPU.

Solution: Check CPPstatistics and configure the optimization policy according to Step 2.

snmpd: SNMP process. If the CPU utilization of this process is high, it isusually because many SNMP packets are sent to the CPU or the SNMP consumes manyCPU resources when obtaining a MIB node of the device.

1) Excute no enable service snmp-agent command totemporarily disable the SNMP agent.

//If SNMP isdisabled, the administrator may fail to manage devices. Besides, it may cause802.1X/WEB authentication failure. Therefore, confirm with the customer beforethis operation.

rl_con: a process for console logging. If the CPU utilization of this processis high, it is usually because many logs are generated.

Or run the logging rate-limit all 10 exceptemergencies command to limit the output rate of the log.

pimd/pim6d: a process for receiving IGMP/PIM multicastpackets. If the CPU utilization of this process is high, it is usually becausemany unknown multicast packets are sent to the CPU.

Solution:
1) Modify the rate limits of the unknown multicast packets which send to CPU.
Ruijie(config)#cpu-protect type unknown-ipmcpps 30

2) Deploy multicast optimization policy , configure an ACL foravoiding multicast source spoofing (to filter illegal multicast packets fromenduser)

Thereis no security policy for multicast by default. Currently, a large amount ofsoftware installed on a PC can send multicast packets, which may causemulticast packets flooding on the entire network and consume multicast resourceentries and CPU resources of the switch. Therefore, deploy the multicastoptimization policy on all multicast-enabled networks.

Usea specific ACL to allow only multicast packets from valid sources to validdestinations to pass through. (Apply the ACL on the Trunk port or SVI.) Whetherin PIM-DM or PIM-SM mode, deploy the ACL on the entire network.

Ruijie(config-ext-nacl)#10permit igmp any any //Allow all IGMP packets to pass through.

Ruijie(config-ext-nacl)#20permit ip 219.229.134.0 0.0.0.255 239.202.0.00.0.255.255

//Allowmulticast packets from legal sources to legal destinations to pass through. (The legal multicast source and destination should be according to the customer's network requirement.)

Tointegrate the above ACL with the old ACL applied on Trunk port or SVI, add ACE10-30 to the old ACL.

If youare not clear about ACL integration, contact Ruijie technical support forassistance.

ef_res: A process for neighbor resolution. If the CPU utilization of thisprocess is high, it is usually because IP scanning or frequent ARP aging anddeletion occurs (mostly because STP receives TC packets).

Step1 Ruijie(config)#nfpp // Run the nfpp command to enter the NFPPconfiguration mode.

Step2 Ruijie(config-nfpp)#ip-guardenable // Run the ip-guard enable command to enable anti-scanning globally. It isenabled by default.

Step3 Ruijie#shownfpp ip-guard hosts //Runthe show nfpp ip-guard hosts commandto display the attacked hosts.

Step 4 Ruijie(config-nfpp)#ip-guardisolate-period //Run the ip-guard isolate-period command to isolatethe attacked hosts.

NFPP-based hardware isolation is disabledby default. If it is required, confirm with the customer before enabling it inthe case of IP scanning. If PCs are found as attacker will be isolated, thecommunication fails (for 10 minutes by default).

Totroubleshoot the attack source, you need to identify the attack source port(based on the IP address/ARP/MAC address/port mappings or mirror traffic).

1) Display logs to check whether the syslog contains abnormal packet sources(IP address and MAC address).

2) Obtain source addresses (IP address and MAC address) of the abnormalpackets by mirroring traffic on suspicious ports. If too many packets gothrough the port, use flow-based mirroring or Wireshark to filter the packets.

3) Check the ARP table and MAC address table of the switch to identify theport sending the abnormal packets.

If theSTP topology frequently changes, enable tc-guard or tc ignore on the portreceiving TC packets and then identify the source of TC packets to rectify thefault. After the network is recovered, remove the temporary optimizationconfiguration.

1.Run the show cpu-protect mbcommand for three to five times (every 5 seconds)

1. Check whether there is any case of high packet drop with constantgrowth. For example, in a fault case, the number of dropped packets with TTLvalue 1 keeps increasing, as shown in the following figure.

· If many types of protocolpackets are dropped with an increasing rate and the RX rate (pps) is verylarge, it is usually because a loop occurs. Go toStep 4 totroubleshoot the fault.

· The fact that some types ofprotocol packets are dropped with an increasing rate proves that a large numberof abnormal packets flood the network. Optimize the network according to theoptimization policy in the following solution and check the result.

2. Check whether there is any case of high RX rate but less drop in thecpp statistics.

To cater to networks of different sizes, the default RX rate limit is setto an empirical value . For example, the default rate limit of unknownmulticast packets is 128 pps.

The rate of unknown multicast packets is very small (for example, 3 pps)on a healthy network. If it ranges from 30 pps to 100 pps, the CPU utilizationwill also increase even though no drop occurs.

· The fact that some types ofprotocol packets with an increasing rate proves that a large number of abnormalpackets flood the network. Optimize the network according to the optimizationpolicy in the following solution and check the result.

· If a CPU process is abnormalwithout CPP statistics exception (or such statistics cannot be independentlyanalyzed), go to Step 4.3 for troubleshooting.

Run theshow cpu-protect summary command todisplay the default RX rate limit of a device.

If the rate limit is too small, packet exchanges will be affected. It isrecommended to adjust the RX rate limit as follows:

1) First reduce the rate limit by 50%. For example, if the originaldefault rate limit is 128 pps, it is recommended to change it to 64 pps andcheck for optimization. After the CPU utilization drops to below 50%, it isrecommended to identify the root cause rather than adjust the rate limit again.CPP rate limiting is only a preventive measure, and the solution is to identifyabnormal packet sources.

2) Some types of packets are too important to reduce the original ratelimit, such as ARP packets, BPDU packets, and OSPF packets. Thereason is that setting a rate limit too small may cause protocol packet drop.

Ruijie(config)#cpu-protect type local-ipv4 bandwidth 50 //Take the command for example

1. nd-snp-ns-na: After aswitch receives routing/neighbor resolution packets from an IPv6 host, its CPUutilization is high. It is usually because a fault occurs to the host such asvirus attack.

Solution:Reduce the rate limit of the preceding packets and identify the packet sources(see the following description).

1) Display logs to check whether the syslog contains abnormal packet sources(IP address and MAC address).

3) Check the ARP table and MAC address table of the switch to identify theport sending the abnormal packets.

2. unknown-ipmc/unknown-ipmcv6:After aswitch receives abnormal multicast packets, its CPU utilization is high .

Solution:Reduce the rate limits of the preceding packets and configure an ACL for antimulticast source spoofing.

Thereis no security policy for multicast by default. Currently, a large amount ofsoftware installed on a PC can send multicast packets, which may causemulticast floods on the entire network and consume multicast resource entriesand CPU resources of the switch. Therefore, deploy the multicast optimizationpolicy on all multicast-enabled networks.

Multicastsource anti-spoofing (implementation process): Usea specific ACL to allow only multicast packets from valid sources to validdestinations to pass through. (Apply the ACL on the Trunk port or SVI.) Whetherin PIM-DM or PIM-SM mode, deploy the ACL on the entire network.

Ruijie(config-ext-nacl)#10permit igmp any any //Allow all IGMP packets to pass through.

Ruijie(config-ext-nacl)#20permit ip 219.229.134.0 0.0.0.255 239.202.0.00.0.255.255

//Allowmulticast packets from valid sources to valid destinations to pass through. (Update the ARP entry according to thecustomer's requirement.)

Tointegrate the above ACL with the old ACL applied on Trunk port or SVI, add ACE40 to the old ACL.

If youare not clear about ACL integration, contact Ruijie technical support forassistance.

3. ip4-packet-other/ip6-packet-other: IPv4 packets not specified in the CPP are usually scanning packets.

Solution:Reduce the rate limits of the preceding packets and identify the packet sources(See above).

1.Check whether there are many incompleteARP entries containing continuous IP addresses.

If yes, IP scanning occurs in the network.In this case, it is recommended to connect one trunk interface to a PC tocapture packets and check whether the switch sends a large number of ARPrequest packets.

· Step1Run the Ruijie(config)#nfpp command //enter the NFPP configuration mode.

· Step2Run the Ruijie(config-nfpp)#ip-guard enable // enable anti-scanning globally. It is enabled by default.

NFPP-based hardware isolation is disabledby default. If it is required, confirm with the customer before enabling it inthe case of IP scanning. If PCs are found attacked, the communication fails(for 10 minutes by default).

To troubleshoot the attack source, you needto identify the attack source port (based on the IP address/ARP/MACaddress/port mappings or SPAN packet capturing).

1). Display logs to check whether thesyslog contains abnormal packet sources (IP address and MAC address).

2). Obtain source addresses (IP address andMAC address) of the abnormal packets by mirroring traffic on suspicious ports.If too many packets go through the port, use flow-based mirroring or Wiresharkto filter the packets.

3).Check the ARP table and MAC addresstable of the switch to identify the port sending the abnormal packets.

2. After connectedto the switch through a Trunk port, the PC receives a large number of packetsrepeatedly.

3. The show interfaces counters rate command outputshows that the traffic on the Trunk port is very high and consistent.

4. When the show mac-address-table command is run forthree consecutive times, the MAC address may move to different ports.

1. The flashingfrequency of indicators is related to the transmission rate of the interface.If a loop occurs, Layer-2 ports will transmit and receive a large number ofpackets, and the interface indicators will rapidly flash with the samefrequency.

2. A loop meansthat broadcast packets are flooded in a VLAN. After accessing a Trunk port, thePC receives packets from the loop. If a large number of packets are capturedrepeatedly, a loop must have occurred.

3. Check whetherthe broadcast traffic is abnormal on a port. If the broadcast traffic suddenlyincreases to a volume higher than the unicast traffic, a loop must haveoccurred.

4. Check whetherthe MAC address of any port frequently changes. If yes, a loop may haveoccurred.

In the network environment where a loopoccurs, eliminate the loop by automatically blocking loop ports throughprotocol (Loop avoidance) or manually disabling ports. Perform the followingoperations:

Optional solution 1: Enable RLDP loop detection and check whether the portis automatically disabled and shut down.

Ruijie(config-if-range)#rldp port loop-detect shutdown //Enable rldp on the Access port.

Ruijie(config-if-range)#spanning-tree bpdufilter enable //Enable BPDU filtering on an STP port toprevent the port from sending or receiving BPDUs.

Ruijie(config-if-range)#spanning-treeportfast //EnableSTP loop guard on the Access port.

Ruijie(config-if-range)#spanning-treebpduguard enable //Enable STP loop guard on the Access port.

2. Enable broadcast storm control on the Access port to mitigatethe broadcast storm problem caused by the loop. (Not required on the uplinkport.)

3. Shut down or unplug cables from suspicious ports one byone to see whether the problem is solved.

1. Run the show log and show ip ospf nei commands to checkwhether the OSPF/BFD neighbor relationship is frequently down.

2. Run the show spanning-treesummary command (every 5 seconds) three times to check whether the topologychanges frequently.

1. Check whether the neighbor relationship based on a routing protocol(OSPF/BFD) flaps up or down and whether logs similar to the following aredisplayed:

*Aug 9 10:26:10: %LINEPROTO-5-UPDOWN: Lineprotocol on Interface GigabitEthernet 0/24, changed state to up.

*Aug 9 10:26:20: %OSPF-5-ADJCHG: Process 10, Nbr10.10.10.1-GigabitEthernet 0/24 from Down to Init, HelloReceived.

*Aug 9 10:26:50: %OSPF-5-ADJCHG: Process 10, Nbr10.10.10.1-GigabitEthernet 0/24 from Exchange to Full, ExchangeDone.

*Aug 9 10:27:44: %OSPF-5-ADJCHG: Process 10, Nbr10.10.10.1-GigabitEthernet 0/24 from Full to Init, 1-WayReceived.

*Aug 9 10:27:55: %OSPF-5-ADJCHG: Process 10, Nbr10.10.10.1-GigabitEthernet 0/24 from Exchange to Full, ExchangeDone.

140092:*Sep 29 19:21:23: %BFD-6-SESSION_STATE_UP: BFD session to neighbor10.242.237.82 on interface TenGigabitEthernet 0/21 is up

140093:*Sep 29 19:21:35: %BFD-6-SESSION_STATE_DOWN: BFD session to neighbor10.242.237.82 on interface TenGigabitEthernet 0/21 has gone down.Reason: Control Detection Time Expired

If there are many logs about neighbor relationship flapping, the protocolfrequently goes up or down.

2. Check whether STP frequently receives topology changes. Focus on the countand duration of topology changes (information in red font).

TimeSinceTopologyChange : 0d:0h:0m:12s //Check whether the last topology lasts for a short time.

TopologyChanges : 1182 //Check whether the count of topology changesrapidly increases.

DesignatedRoot: 1000.001a.a915.5984 <---MAC addressof Root Bridge for all regions

RootPort: 0 <---Root portselected by the local host in Instance 0. If the Root Bridge is the local host,this value is 0.

CistRegionRoot: 1000.001a.a915.5984 <---MACaddresses of the Root Bridge for Instance 0 in the local region.

If the topology change count rapidly increases and the last change lastsfor a short time, TC packets (STP topology changes) must have been generated inthe network. For details about troubleshooting, see the "Troubleshooting STP"Section

1. If a protocol flapping occurs due to the link fault on a port, you cantemporarily shut down the port or disable the dynamic routing protocol to seewhether the fault is rectified. If not, troubleshoot the link fault on theport.

2. If the topology frequently changes, you can enable TC guard or TCignorance on the port receiving TC packets and then identify the source of TCpackets (described in Step4). After the network isrecovered, remove the makeshift configuration.

Ruijie(config-if)#spanning-tree tc-protection tc-guard //Enable TC guard on theport receiving TC packets. The port is connected to an access/convergence switch.

Ruijie(config-if)#spanning-tree ignore tc (an optimizationto TC guard) // Enable TC ignorance on the port receiving TC packets. The portis connected to an access/convergence switch.

3. If the fault persists, go to Step 6 to collect information and thencontact Ruijie technical support for assistance.

show cpu-protect slot X //X indicates the slot number. Forexample, run show cpu-pro slot 4 todisplay information about slot 4 on a standalone S8600E series switch; Run show cpu-pro slot 2/4 to displayinformation about slot 4 on chassis 2 of an S8600E Virtual Switching Unit(VSU).

3) Capture packets on the network for 2 to 3 minutes, and enable portmirroring, with the mirror source being a typical service port. (It isrecommended that packets be captured every 10 seconds. If the traffic is verylarge, capture the packets for 1 to 2 seconds multiple times)

Ruijie(config)#monitor session 1 source interface gigabitEthernet0/Y //gigabitEthernet 0/Y indicates a serviceinterface. Multiple service interfaces need to be selected.

Ruijie(config)#monitor session 1 destination interface gigabitEthernet0/X //gigabitEthernet0/X indicates an interface connected to the PC.

þ Gathering troubleshooting information and then submit a case in theservice portal(case.ruijienetowrks.com),and contact the Ruijie Technical Assistance Center (TAC) for assistance andfurther instructions.