CPU Tuning

1. CPU Performance Metrics

1.1. CPU Utilization

The percentage of non-idle time out of total CPU time. Depending on the tasks running on the CPU, it is further divided into user CPU, system CPU, I/O-wait CPU, soft interrupts, hard interrupts, etc.

• User CPU: applications.
• System CPU: kernel (excluding interrupts).
• I/O-wait CPU: interaction between the system and hardware devices.
• Soft and hard interrupts: the kernel invokes interrupt handlers.

1.2. Average Load

The average number of active processes in the system. It reflects the overall load of the system and mainly includes three values: the average load over the past 1 minute, 5 minutes, and 15 minutes. Ideally it should equal the number of logical CPUs.

1.3. Context Switches

Process context switching consumes CPU time saving and restoring data such as registers, the kernel stack, and virtual memory, reducing the time available for processes to run. It mainly includes:

• Voluntary context switches caused by being unable to obtain resources.
• Involuntary context switches caused by forced system scheduling.

1.4. CPU Cache Hit Rate

Caches between CPU and memory, mainly L1, L2, and L3.

2. CPU Performance Tools

2.1. top

top

2.2. pidstat

pidstat

2.3. vmstat

vmstat

2.4. perf

perf

2.5. ps

ps

2.6. pstree

pstree

3. How to Analyze a CPU Performance Bottleneck

3.1. Check Whether CPU Is the Bottleneck

• Use top to view CPU load (load average) and CPU utilization (%Cpu). If both are high, it is indeed a CPU bottleneck.

3.2. Find the Process with High CPU Usage

• In top, press P to see which process has high CPU utilization (%Cpu), and record its PID.

3.3. Analyze the Process

• If it is a Java process:
  1. Java Online Problem Troubleshooting
• If it is a Golang process:
  1. pprof
• If it is a MySQL process:
  1. MySQL Online Problem Troubleshooting

4. References

• Troubleshooting and Solving High Load in Linux Systems - 51CTO