Partitions, offsets and consumer groups

2 min read
On trails:EDAstep 6/8

A stream topic, as in Apache Kafka, is split into partitions, each an independent ordered log. Every record in a partition has an offset, its position in that log, and a consumer group stores one offset per partition: a bookmark for where it has read up to. Think of partitions as lanes, offsets as positions along a lane, and a consumer group as a team of workers sharing the lanes.

A topic with three partitions, each a row of records numbered by offset and labelled with an order id as the key. A consumer group named payments has two consumers: consumer A reads partitions P0 and P1, consumer B reads P2. OFFSET 0 1 2 3 4 5 P0 1042 1051 1042 1060 1042 1051 P1 1043 1043 1057 1043 1057 1043 P2 1044 1058 1044 1044 1058 1061 GROUP: PAYMENTS consumer A reads P0 and P1 consumer B reads P2 each number is an order id, used as the key: a lane holds many keys, but one key stays in one lane highlighted box: the next record to read in each lane, the group bookmark
A lane holds many keys. The group has one bookmark per lane.

Key to partition

The producer chooses the partition, usually by hashing the record’s key. Orders uses the order id as the key, so every event for order 1042 lands in the same partition, and Kafka guarantees that a consumer reads one partition’s records in the order they were written. That gives order per key and none across partitions.

A partition is not one order. It holds many keys, as the diagram shows, and many orders share each lane. Adding partitions to an existing topic does not move existing records, and new records for a key may hash to a different partition, which Kafka’s documentation warns can disturb consumers that rely on key placement.

Offsets

A consumer’s position in a partition is a single number, the offset of the next record to read. The group saves it periodically, so a restarted consumer resumes from the saved offset instead of the start. Because reads do not delete anything, an offset can also be moved back to read old records again, within the topic’s retention.

Consumer groups

Consumers that share a group id form a group, with these rules.

  • Each partition is read by exactly one consumer in the group at a time. A consumer may read several partitions.
  • Partitions are the unit of parallelism, so a topic with 3 partitions keeps at most 3 consumers of one group busy. A fourth consumer in the group sits idle.
  • Different groups are independent. Each has its own offsets and reads every record.

When a consumer joins, leaves or crashes, the group coordinator reassigns partitions, which can briefly interrupt consumption. Kafka 4.0 introduced a new rebalance protocol with less disruption than the classic one.

The catch

A partition is shared by all its keys. If a consumer must keep order and stops on one failing record, every other key in that partition waits behind it, not just the failing one. The partition count also sets the ceiling on parallelism and is awkward to change later, so choose it with growth in mind. See message orderingMessage orderingOrder exists only within one key, and only while one consumer handles that key at a time.Opens in a new tab and dead letters and ordered messagesDead letters and ordered messagesA dead-lettered message leaves the messages after it running on state that never happened.Opens in a new tab.

Sources