Compacted topics
One way to reduce the disk space requirements in Apache Kafka® is to use compacted topics. This methodology retains only the newest record for each key on a topic, regardless of whether the retention period of the message has expired or not. Depending on the application, this can significantly reduce the amount of storage required for the topic.
To make use of log compaction, all messages sent to the topic must have an explicit key. To enable log compaction, follow the steps described in how to configure log cleaner.
How compacted topics work
An Apache Kafka topic represents a continuous stream of messages that typically get discarded after the message reaches a certain period of time or size. However, for certain use cases it is only needed the most recent value for a certain key.
For example, if there is a topic containing a user's home address, on
every update, a message is sent using user_id as the primary key and
home address as the value:
1001 -> "4 Privet Dr"
1002 -> "221B Baker Street"
1003 -> "Milkman Road"
1002 -> "21 Jump St"
1001 -> "Paper St"
1001 -> "Paper Road 21"
There are three different options to define for how long to retain the messages:
- infinite message retention: all changes to user's address are maintained in the logs. This can lead to the log growing in size without a bound. This option involves the risk of outgrowing the disk capacity.
- simple message retention: older records are deleted after they reach a certain age or size.
- compacted topic: only latest version of the key's value is kept. In the example above, only the current address for a specific user is kept.
With compacted topics, Apache Kafka removes any records from the topics for which there is a newer version (based on the record key) is available in the partition. This retention policy can be set per-topic, so a single cluster can have some topics where retention is enforced by size or time and other topics where retention is enforced by compaction.
Compacted topic example
To understand better how compaction works, we will look at a partition of a compacted topic before and after compaction has been applied.
Continuing the example above, the topic records before the compaction would be:
| Offset | Key | Value |
|---|---|---|
| 1 | 1001 | 4 Privet Dr |
| 2 | 1002 | 221B Baker Street |
| 3 | 1003 | Milkman Road |
| 4 | 1002 | 21 Jump St |
| 5 | 1001 | Paper St |
| 6 | 1001 | Paper Road 21 |
You can notice that there are some records with duplicate keys (1001
and 1002), with the records having offset 4, 5, and 6 being the
addresses updates. When applying compaction, we only keep records with
the latest offset (newest values) and the older ones get discarded. The
end result is the following:
| Offset | Key | Value |
|---|---|---|
| 3 | 1003 | Milkman Road |
| 4 | 1002 | 21 Jump St |
| 6 | 1001 | Paper Road 21 |
Compacted topic details
A compacted topic consists of an head and a tail:
- the head is a traditional Apache Kafka topic where new records are appended. Therefore, the head can contain duplicated keys.
- the tail contains one record per key. Apache Kafka compaction ensures that keys are unique in the tail.
Expanding the example above let's assume that the tail contains the following entries:
| Offset | Key | Value |
|---|---|---|
| 1 | 1001 | 4 Privet Dr |
| 2 | 1002 | 221B Baker Street |
| 3 | 1003 | Milkman Road |
And the head, the newer records including duplicated keys:
| Offset | Key | Value |
|---|---|---|
| 4 | 1002 | 21 Jump St |
| 5 | 1001 | Paper St |
| 6 | 1001 | Paper Road 21 |
During the compaction process, Apache Kafka creates a structure called offset map for the records in the head section, containing for each key, the latest offset.
| Key | Offset |
|---|---|
| 1002 | 4 |
| 1001 | 6 |
The compaction thread then scans the tail, removing every record having a key that is also present in the offset map with an higher offset.
| Offset | Key | Value |
|---|---|---|
| 1 | 1001 (delete) | 4 Privet Dr |
| 2 | 1002 (delete) | 221B Baker Street |
| 3 | 1003 | Milkman Road |
Lastly, the records in the offset map are added in the tail.
| Offset | Key | Value |
|---|---|---|
| 3 | 1003 | Milkman Road |
| 4 | 1002 | 21 Jump St |
| 6 | 1001 | Paper Road 21 |
The compaction occurs per partition: if two records with the same key land in different partitions, they will not be compacted.
This usually doesn't happen since the record key is used to select the partition. However, for custom message routing this might be an issue.