Kafka guarantees the order of delivery within a partition. That is, the order is not kept among partitions in a topic. There are some tips to keep the order of delivery in a topic.
1. One partition for a Topic
This is what Kafka basically supports.
- Simple solution
- A partition is applied to only one consumer, which has limitation in scale out
2. Custom partitioner
Partition number is decided by producer’s partitioner. With default option, org.apache.kafka.clients.producer.internals.DefaultPartitioner is used. DefaultPartitioner decides partition number by the hash value of a key.
Although it is difficult to keep the order of delivery for all messages, we can split it into groups which need the order of delivery. In this case, we can partition by the group info.
Suppose that messages are generated by several departments and the order is important within a department.
In this case,
- Prefix a message key with dept code
- Apply custom partitioner which decides partition based on dept code
The following is the sample code.
Above partitioner extracts prefix from key and partition by it.
Custom partitioner is set by “partitioner.class” property.
<Message with default partitioner>
Partition 6 for key dept1.key-1 Partition 4 for key dept1.key-2 Partition 8 for key dept2.key-3 Partition 6 for key dept2.key-4 Partition 5 for key dept3.key-5 Partition 7 for key dept3.key-6 Partition 9 for key dept4.key-7 Partition 3 for key dept4.key-8 Partition 2 for key dept5.key-9 Partition 8 for key dept5.key-10
<Message with custom partitioner>
Partition 8 for key dept1.key-1 Partition 8 for key dept1.key-2 Partition 9 for key dept2.key-3 Partition 9 for key dept2.key-4 Partition 0 for key dept3.key-5 Partition 0 for key dept3.key-6 Partition 1 for key dept4.key-7 Partition 1 for key dept4.key-8 Partition 2 for key dept5.key-9 Partition 2 for key dept5.key-10
You can make sure that messages with the same dept id are sent to the same partition.