How Kafka Topic failover works?

After Kafka cluster has been configured, we need to create a Topic which enables failover and data replication.

The conclusion in advance is that if a Topic’s replication factor is more than 2,

  • Kafka supports automatic leader failover
  • Data rebalance is supported only in manual operation

Test environment

  • Kafka 2.12
  • 3 Kakfa brokers (Id : 0, 1, 2) on a PC

To create a Topic

${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic test-topic

replication-factor 2 means that there are 2 copies for every published item (including leader broker’s)

To check the Topic status

./kafka-topics.sh --describe --zookeeper localhost:2181 --topic test-topic
Topic:test-topic    PartitionCount:2    ReplicationFactor:2    Configs:
    Topic: test-topic    Partition: 0    Leader: 1    Replicas: 1,2    Isr: 1,2
    Topic: test-topic    Partition: 1    Leader: 2    Replicas: 2,0    Isr: 2,0

Above result shows that partitions 0 is located on broker 1 and 2 (check Isr value) and partition 1 is located on broker 2 and 0.

To check the status after a broker’s shutdown

What if a broker is shut down?. For test, I shut down broker 0. (This could affect partition 0)

{KAFKA_HOME}/bin/kafka-topics.sh --describe --zookeeper localhost:2181
Topic:test-topic    PartitionCount:2    ReplicationFactor:2    Configs:
    Topic: test-topic    Partition: 0    Leader: 2    Replicas: 1,2    Isr: 2
    Topic: test-topic    Partition: 1    Leader: 2    Replicas: 2,0    Isr: 2,0

Above result shows that partition 1 is not affected because it’s located on broker 0 and 1. But partition 0’s leader is changed into broker 2. (Leader value has changed from 1 to 2)

The problem is …

In this status, Kafka service continues. But partition 0’s data is only on broker 2. (check Isr value) So while broker 1’s down, if broker 2 dies, partition 0 data is lost. Unfortunately, Kafka doesn’t support automatic data rebalance. (As I know, it’s true.)

So we need to rebalance it manually.

To rebalance the topic manually

Kafka supports data rebalance in 3 steps.

1) to generate reassignment script (topics-to-move.json)

{"topics": [
        {"topic": "test-topic"}
            ],
"version":1
}

With above script, run the following command.

./kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "0,2" --generate

The ouput shows like this.

Current partition replica assignment
{“version”:1,”partitions”:[{“topic”:”test-topic”,”partition”:1,”replicas”:[2,0],”log_dirs”:[“any”,”any”]},{“topic”:”test-topic”,”partition”:0,”replicas”:[1,2],”log_dirs”:[“any”,”any”]}]}

Proposed partition reassignment configuration
{“version”:1,”partitions”:[{“topic”:”test-topic”,”partition”:1,”replicas”:[0,2],”log_dirs”:[“any”,”any”]},{“topic”:”test-topic”,”partition”:0,”replicas”:[2,0],”log_dirs”:[“any”,”any”]}]}

2) to commit reassignment

Save the part of proposed partition reassignment. (i.e as “final-topics-to-move.json”)
And with the new file, run the following command.

./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file final-topics-to-move.json --execute

Now, the Topic has been rebalanced.

3) to verify

To verify the result, run the following command.

./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file final-topics-to-move.json --verify

The output shows like this.

Status of partition reassignment:
Reassignment of partition test-topic-1 completed successfully
Reassignment of partition test-topic-0 completed successfully

To check the status after the rebalance

Now, check the Topic status again.

${KAFA_HOME}/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test-topic
Topic:test-topic	PartitionCount:2	ReplicationFactor:2	Configs:
	Topic: test-topic	Partition: 0	Leader: 0	Replicas: 0,2	Isr: 2,0
	Topic: test-topic	Partition: 1	Leader: 2	Replicas: 2,0	Isr: 2,0

Now, we can verify that partition 0’s data is replicated on broker 2 and 0 (Isr value has changed)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.