2.2 Stress Testing

1. What Is Stress Testing

• Under the premise that performance remains acceptable, test the maximum load that a subsystem or interface can support.

2. Why Stress Testing Is Needed

• Find the bottleneck of a subsystem or interface so that corresponding optimization and scaling can be carried out.
• Verify whether the subsystem or interface can support the estimated load.

3. How to Design a Stress Test and Calculate QPS

3.1. Estimation

• Based on throughput = concurrency / response time:
  • For a compute-intensive application, it depends on the number of CPU cores. Suppose the CPU is 4C, the interface response time is 100 ms, and all 100 ms are used for CPU computation: QPS=4C/100ms=4C/0.1s=40.
  • For an I/O-intensive application, requests waiting for I/O can yield the CPU, so concurrency can exceed the CPU-core count. However, 1600 QPS cannot be derived merely from “four threads per core” and 10 ms of CPU time; actual throughput also depends on concurrency, response time, and system bottlenecks.
• The actual stress-test result should be determined by measurement and cannot be assumed to fall between 40 and 1600.

3.2. Practice

3.2.1. Create a Stress-Test Instance

3.2.2. Gradually Increase Concurrency

• 200 threads, start within 5s, loop 5 times
• 400 threads, start within 5s, loop 5 times
• 600 threads, start within 5s, loop 5 times
• 800 threads, start within 5s, loop 5 times
• …

If QPS cannot reach the target, increase concurrency.

3.2.3. Check QPS

• Check whether the response times on the 90th, 95th, and 99th percentile lines are within the expected range (for example, within 50 ms). If so, the QPS can be seen to be around 500.

4. How to Accurately Evaluate Actual QPS

Stress testing with fabricated data may not accurately reflect the QPS of each machine. This approach can only be used for launch-time evaluation.

For event or promotion scenarios, QPS needs to be evaluated accurately, which requires real data.

1. Capacity stress testing: route real production traffic to the target machine being tested, so the data is real. Configure the service instance to be tested, the service-routing weight growth method, and the threshold levels for key metrics of the single machine under test (CPU utilization, overall system load, QPS, response time, and so on). Stop the stress test automatically when those thresholds are reached to avoid a large impact on production. After the test stops, the platform can show the maximum QPS that the single service instance can provide when system resources reach the threshold.
2. Capacity planning: use the single-machine QPS data of the service, together with analysis of processing-capability differences between server models, to estimate more accurately how many production servers are needed for a large promotion.

5. How to Optimize QPS

Based on QPS = concurrency / response time, there are two directions:

1. Increase concurrency. One approach is to create more threads. Of course, this assumes CPU resources are unlimited, while in reality CPU resources are limited, so creating too many threads will make the CPU busy with thread scheduling and cause QPS to decrease. Another approach is to make the percentage occupied by the interface in the flame graph as high as possible, indicating that CPU time is being used by the interface.
2. Reduce response time. This requires analyzing the bottleneck of the program and optimizing it. A running program uses CPU, memory, disk, and network resources, so the bottleneck must be among these. Linux performance tuning

6. Example

6.1. Frequency-Control System Stress Test

How to design a frequency-control system

7. References

• What Is the Difference Between Stress Testing and Performance Testing? - Zhihu
• PV UV QPS Concurrency - tooltime - CNBlogs
• Java Interface Stress Testing: When the Interface Is Not the Bottleneck, the Impact of CPU and Network Bandwidth on QPS
• The Way of Enterprise IT Architecture Transformation: Alibaba Middle-Platform Strategy and Architecture Practice