2.12 Cloud Kafka

This is a historical learning note and may contain outdated or incomplete understanding.

1. Tencent Cloud Kafka

Based on VIP + native Kafka.

VIP: the addressing + health-check + nearby-routing functions of the Polaris registry. CKafka exposes a VIP to clients. After a client connects to the VIP, it obtains metadata for Topic partitions. When an availability zone becomes unavailable, the VIP automatically floats to another available node in the same region, thereby providing cross-availability-zone disaster recovery.
Native Kafka: version 1.1.1; a single Topic supports at most 3,000 partitions (all Leaders + Followers), and at most three replicas (1 Leader + 2 Followers). Each node configuration: 300 GB disk; CPU and memory unknown.

1.1. Deployment

1.1.1. Single Availability Zone in the Same Region

ZooKeeper primary/replicas are in the same availability zone, and Kafka Leader/Follower Partitions are in the same availability zone.

1.1.1.1. Problem

If the single availability zone goes down, clients cannot access the service.

1.1.2. Multi-Availability-Zone Deployment in the Same Region

The ZooKeeper primary is in one availability zone and the replicas are in another availability zone; Kafka Leaders are in different availability zones, and Followers are also in different availability zones. Kafka

1.1.2.1. Problems

There are three availability zones, A, B, and C. ZooKeeper is deployed across the three availability zones, with A as the primary. If the number of Kafka replicas is 2, then they are deployed across 2 availability zones. If the number of Leader partitions is also 2, then the number of Brokers is 2 * 2 = 4. Suppose Broker1 in A is the Controller. The number of Brokers is automatically calculated by the program based on bandwidth and then evenly allocated to the two availability zones.

A single availability zone becomes unavailable: ZooKeeper is the same as Cloud Redis. If availability zone B goes down, availability zone A removes the replica node in B from the cluster. If availability zone A goes down, B and C re-elect the primary. Kafka: if availability zone B goes down, the Controller in availability zone A receives a notification through ZooKeeper’s event callback and removes Broker3 and Broker4 in B from the cluster. If availability zone A goes down, Broker3 and Broker4 in availability zone B elect a Controller through ZooKeeper, and then remove Broker1 and Broker2 in A from the cluster.
Network isolation: if availability zones A and B are isolated from each other, will they become two clusters? ZooKeeper will not, because of the majority mechanism. Kafka can. If the two AZs become isolated and cannot communicate with each other, cluster split-brain may occur: nodes in both availability zones provide service, but writes in one availability zone are treated as dirty data after the cluster recovers.

Consider the following scenario: the cluster Controller node and one ZooKeeper node in the zk cluster become network-isolated from the other nodes. The remaining nodes re-elect a new Controller (because a majority of ZooKeeper nodes can still communicate normally, so the Controller election can succeed), but the isolated Controller still believes it is the Controller node. At this point, the cluster experiences split-brain.

Client writes need to be considered case by case. For example, when the client’s Ack policy equals -1 or all and the replica count is 2, suppose the cluster has 3 nodes and is split 2:1 after split-brain. Writes to partitions whose original leader is in the region with 1 node will fail, while writes on the other side will succeed. If there are 3 replicas and Ack = -1 or all is configured, neither side can successfully write. Further handling therefore needs to be determined according to the actual parameter configuration.

After the cluster network recovers, clients can resume producing and consuming without any action. However, because the server normalizes the data again, data on one split node is directly truncated. For a multi-replica cross-zone data-storage method, this truncation does not cause data loss.

1.2. Consistency

1.2.1. Latency

Within the same availability zone it is generally within 5 ms; across availability zones it is generally 10 ms to 40 ms.

1.3. Scalability

The ability to quickly scale out is essentially Kafka itself.

1.4. Cost

消息队列 CKafka 计费概述-购买指南-文档中心-腾讯云

1.5. Availability

1.5.1. SLA

The service availability of this service is not lower than 99.95%. If the above availability standard is not met (except in cases covered by exemption clauses), compensation can be obtained according to Article 3 (Compensation Plan) of the agreement. Assuming the total monthly service minutes for a single service instance are 30 × 24 × 60 × 99.95% = 43178.4 minutes, there are 43200 - 43178.4 = 21.6 minutes of unavailability.

1.5.2. Failure Handling

1.6. Monitoring

Monitoring granularity: 1 minute.

1.7. Parameters

Retry mechanism: message.send.max.retries=3 retry.backoff.ms=10000 High reliability: request.required.acks=-1 min.insync.replicas=2 High performance: request.required.acks=0 Reliability + performance: request.required.acks=1