5.4 Distributed-System Partitioning: Partition Assignment

Distribute partitions/nodes across machines as evenly as possible.

1. Assignment Methods

1.1. Static Assignment

Create far more nodes than machines.

Advantage: when migrating nodes to other machines, the cluster can still respond externally.

Disadvantage: the maximum number of machines is fixed.

1.2. Dynamic Assignment

Each node coordinates with the others, and each is responsible for a portion of the partitions.

Advantage: nodes can be dynamically added or removed.

Disadvantage: during partition rebalancing, the cluster cannot respond externally.

1.3. Examples

• MySQL is manually specified by the programmer: static assignment.
• Redis has each node responsible for a certain range of slots: dynamic assignment.
• Kafka distributes partitions evenly across brokers: dynamic assignment.
• ZooKeeper has no partitioning.
• Elasticsearch uses dynamic assignment.

2. Assignment Strategies

Similar to the data-splitting strategies in Distributed-System Partitioning: Data Splitting.

2.1. By Order

2.2. By Ratio

Keep the number of nodes/partitions proportional to the number of machines.

3. Partition Rebalancing

When the number of machines increases or decreases, partitions/nodes may no longer be evenly distributed across all machines. Partitions/nodes need to be reassigned to balance the load.

3.1. Manual

When machines are added or removed, manually maintain the balance of partitions/nodes.

Advantage: problems can be handled promptly when they occur.

Disadvantage: troublesome.

3.2. Automatic

When machines are added or removed, the cluster automatically maintains the balance of partitions/nodes.

Advantage: convenient.

Disadvantage: problems cannot be handled promptly when they occur.

3.3. Examples

• MySQL is manual.
• Redis is automatic.
• Kafka is manual.
• ZooKeeper has no partitioning.
• Elasticsearch is automatic.