Kafka tip – keeping the order of delivery

Kafka guarantees the order of delivery within a partition. That is, the order is not kept among partitions in a topic. There are some tips to keep the order of delivery in a topic.

1. One partition for a Topic

This is what Kafka basically supports.

<Pros>

  • Simple solution

<Cons>

  • A partition is applied to only one consumer, which has limitation in scale out

2. Custom partitioner

Partition number is decided by producer’s partitioner. With default option, org.apache.kafka.clients.producer.internals.DefaultPartitioner is used. DefaultPartitioner decides partition number by the hash value of a key.

Although it is difficult to keep the order of delivery for all messages, we can split it into groups which need the order of delivery. In this case, we can partition by the group info.

Suppose that messages are generated by several departments and the order is important within a department.

In this case,

  1. Prefix a message key with dept code
  2. Apply custom partitioner which decides partition based on dept code

The following is the sample code.

Above partitioner extracts prefix from key and partition by it.

Custom partitioner is set by “partitioner.class” property.

<Message with default partitioner>

Partition 6 for key dept1.key-1
Partition 4 for key dept1.key-2
Partition 8 for key dept2.key-3
Partition 6 for key dept2.key-4
Partition 5 for key dept3.key-5
Partition 7 for key dept3.key-6
Partition 9 for key dept4.key-7
Partition 3 for key dept4.key-8
Partition 2 for key dept5.key-9
Partition 8 for key dept5.key-10

<Message with custom partitioner>

Partition 8 for key dept1.key-1
Partition 8 for key dept1.key-2
Partition 9 for key dept2.key-3
Partition 9 for key dept2.key-4
Partition 0 for key dept3.key-5
Partition 0 for key dept3.key-6
Partition 1 for key dept4.key-7
Partition 1 for key dept4.key-8
Partition 2 for key dept5.key-9
Partition 2 for key dept5.key-10

You can make sure that messages with the same dept id are sent to the same partition.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.