ZooKeeper High Availability

1. Leader-Follower Replication

Leader-Follower Replication in Distributed-System Replication Architecture.md

1.1. Leader Election

1.1.1. Election Method

At startup, it is based on the election algorithm. See ZAB Protocol.md.

1.2. Data Synchronization

1.2.1. Synchronization Process

• Full replication: the Leader sends its local log to the Follower, and the Follower restores the data.

1.2.2. Synchronization Method

Asynchronous.

1.2.3. Synchronization Log

Logical log.

1.3. Request Processing

1.3.1. Read Requests

Can be handled by the Leader or a Follower. Under read/write separation, they are generally handled by Followers, so the more nodes there are, the higher the read-request throughput.

1.3.2. Write Requests

Must be handled by the Leader. If a request is routed to the Leader, the Leader handles it and then synchronizes it to the Followers. If a request is routed to a Follower, the Follower needs to forward it to the Leader, or the Follower tells the client to redirect to the Leader. After the Leader handles it, it synchronizes it to the Followers.

1.4. Failure Handling

1.4.1. Failure Detection

The Leader and Followers keep alive through heartbeats.

1.4.2. Failure Recovery

1.4.2.1. Follower Crash

• When a Follower reconnects after crashing, incremental replication is triggered.
• Incremental replication: every time the Leader writes a command, it puts it into a buffer and then sends it to the Follower, and the Follower replays the command.
  • The Follower disconnects.
  • The Leader continues writing into the buffer.
  • The Follower reconnects to the Leader.
  • The Follower sends psync.
  • The Leader sends the data in the buffer to the Follower.

1.4.2.2. Leader Crash

• Trigger the election process to elect a Follower as Leader, and notify the other Followers that the Leader has changed.
• ZAB Protocol.md

2. References

• ZooKeeper Leader/Follower Synchronization - Jianshu