NOTE

3.4 Redis Blocking

What Redis blocking is, how to discover it, and internal and external causes including complex commands, CPU saturation, persistence, swap, and network problems.

Redis / CacheCreated Updated 2 min readhistorical

This is a historical learning note and may contain outdated or incomplete understanding.

1. What Is Redis Blocking?

  • Redis takes a long time to respond to client requests.

2. Why Blocking Happens

In essence, Redis has a single-threaded architecture and command execution is completed in this thread.

3. How to Discover Blocking

  • A business-side Redis client times out when connecting to the Redis server, triggering a monitoring alert.

4. Cause Analysis

4.1. Internal

4.1.1. High-Complexity Commands Are Used

  • Unreasonable data structures
  • Too much data in an atomic command, for example mset with 10K pieces of data.
    • How to solve: execute in batches; if atomicity is not required, pipeline can be used.

4.1.2. CPU Saturation (Single-Machine Performance Bottleneck)

  • CPU saturation means Redis drives a single CPU core close to 100% utilization.
  • How to discover:
    • Use the top command to easily identify the CPU utilization of the corresponding Redis process.
  • How to solve:
    • In essence, this is a single-instance Redis performance bottleneck, so Redis Cluster mode is needed to distribute the pressure.

4.1.3. Persistence

  • Some persistence operations can block the main thread: fork blocking and AOF flush blocking.
  • Fork blocking
    • RDB and AOF rewriting fork a child process for processing.
    • How to discover: use info stats and check latest_fork_usec.
    • How to solve: avoid excessively large memory instances.
  • AOF flush blocking

4.2. External

4.2.1. CPU Competition

For example, Redis is deployed together with other services on the same machine.

4.2.2. Memory Swap

Set /proc/sys/vm/swapness to 1, retaining swap while reducing its use.

4.2.3. Network Problems

4.2.3.1. Connection Refusal
  • Network interruption: for example, network switching or bandwidth exhaustion; sar can be used for diagnosis.
  • Redis maximum number of client connections: Redis configuration file.
  • Connection overflow: for example, the Linux process file-descriptor limit.
4.2.3.2. Network Latency

Depends on the network environment from client -> server.

Discussion

Sign in with GitHub to comment. Discussions are stored as GitHub Issues.View on GitHub