4.1 Distributed-System Replication Architecture: Leader-Leader Replication

1. What Is Leader-Leader?

• There are multiple Leaders, and each Leader has multiple Followers.

2. Leader-Leader Use Cases

• Multiple data centers.
• Applications still need to continue working after the network is disconnected.

3. How Leader-Leader Works

3.1. Leader Election

3.2. Leaders Synchronize Data to Leaders

• Writes are handled by a Leader. After the Leader saves the data locally, it performs two operations:
  • One is to send the data change to Followers in the same data center, and the Followers replay the data change.
  • The other is to send it to Leaders in other data centers. Those Leaders replay the data change and synchronize it to their Followers.

3.2.1. Write-Conflict Problem

• Multiple clients modify the same key at the same time.
  • With a single Leader, the second write blocks and waits for the first write to complete.
  • With multiple Leaders, after writing to the Leader in data center 1, the data is asynchronously replicated to the Leader in another data center, so both writes succeed.

3.2.1.1. How to Solve Write Conflicts

3.2.1.1.1. Synchronous Waiting

• The client requests leader1 to write. leader1 synchronizes to the other Leaders before returning success to the client.
• Disadvantage: synchronous replication has low availability (low performance).

3.2.1.1.2. Avoid Conflicts

• Route writes from the same user to the same Leader.
• Disadvantage: if routing changes, conflicts can still occur.

3.2.1.1.3. Resolve Conflicts

• LWW:
  • Assign a timestamp to each write. Keep the write with the higher timestamp and discard the other writes.
  • Generally, the server adds a version field to each record. The client first reads the data and increments version by 1 when modifying it. The server checks whether the client’s version is greater than the data’s version; if so, modify it, otherwise reject it.
• Merge these values together.
• Custom resolution:
  • Execute on write: once a write conflict is detected, call a conflict handler to process it.
  • Execute on read: when a conflict is detected, all conflicting writes are stored. The next time the data is read, these multiple versions are returned to the application. The application may prompt the user or automatically resolve the conflict and write the result back to the database.

3.3. Failure Handling