ThinkerQAQ

NOTE

2.11 Kafka Benchmarking

Kafka producer and consumer benchmarking and a simple way to estimate partition and machine counts.

Message QueuesCreated Updated 1 min readhistorical

This is a historical learning note and may contain outdated or incomplete understanding.

1. Benchmarking

1.1. Producer

kafka-producer-perf-test.sh --topic test_perf --num-records 100000 --record-size 1000  --throughput 2000 --producer-props bootstrap.servers=127.0.0.1:9092

1.2. Consumer

./kafka-consumer-perf-test.sh --broker-list 127.0.0.1:9092 --topic test_perf --fetch-size 1048576 --messages 100000 --threads 1

2. Determining the Number of Partitions and Machines

Same as Release in Business System Design Analysis Approach.

  1. Calculate the QPS that a single partition/machine can support. Refer to Stress Testing.md (the related note has not yet been published).
  2. Calculate the number of partitions/machines = estimated business QPS / QPS supported by a single partition/machine + a little spare capacity.

3. References

Discussion

Sign in with GitHub to comment. Discussions are stored as GitHub Issues.View on GitHub