Message ordering

2 min read
On trails:EDARELstep 4/9

A broker keeps messages in order within one queue, partition or session, and only while a single consumer handles them one at a time. Across a whole system there is no order. A consumer can rely on the order of events for one order ID, and on nothing about how those events interleave with events for another order.

How each broker gives order

BrokerOrdered unitHowWhere it breaks
KafkaA partitionRecords with the same key go to the same partition, and a consumer reads a partition in the order writtenA partition holds many keys, so there is no order between keys, and no order across partitions
Azure Service BusA sessionMessages share a SessionId; one receiver holds the session lock and gets them in orderSessions must be enabled (Standard or Premium tier). A sequence number alone orders the queue, not the processing
RabbitMQA queue with one consumerA queue delivers in publish orderSeveral consumers on one queue, or a requeued message, change the order
Table 1: where order comes from

Competing consumers lose it

Two consumers share a queue holding messages 1, 2 and 3. Consumer 1 takes message 1 and is slow. Consumer 2 takes messages 2 and 3 and finishes both. The messages were handled in the order 2, 3, 1. More consumers on one queue means more throughput and less order. The way to keep both is to give each key its own lane: a Kafka key, or a Service Bus session, so that all messages for order 42 reach the same consumer in sequence while other orders go elsewhere. See partitions and consumer groupsPartitions, offsets and consumer groupsHow a stream splits a topic into lanes, marks each reader's place, and shares the work among a group.Opens in a new tab and queues and topicsQueue vs topicA queue gives each message to one consumer, a topic gives every subscriber its own copy.Opens in a new tab.

A message that fails and is retried later also arrives after the ones behind it (see retries and backoffRetries and backoffRun a failed message again after a growing, randomised delay, and stop after a limit.Opens in a new tab).

Version numbers

Where order cannot be guaranteed, make it harmless. Each event carries the version of the entity it describes: OrderPlaced is version 1, PaymentReceived is 2, OrderShipped is 3. The consumer stores the highest version it has applied and ignores any message that is not newer.

ApplyIfNewer.csC#
var rows = await db.OrderViews    .Where(v => v.OrderId == msg.OrderId && v.Version < msg.Version)    .ExecuteUpdateAsync(s => s        .SetProperty(v => v.Status, msg.Status)        .SetProperty(v => v.Version, msg.Version), ct); if (rows == 0){    // The view already holds this version or a newer one, so this message is stale.    logger.LogInformation("Ignored stale version {Version} of order {OrderId}", msg.Version, msg.OrderId);}

The condition on line 2 makes the check and the update one statement, so two consumers cannot interleave between them. To also detect a gap, such as version 3 arriving when the stored version is 1, require v.Version == msg.Version - 1 instead and hold back anything else. What to do with a held-back message after a failure is the subject of dead letters and ordered messagesDead letters and ordered messagesA dead-lettered message leaves the messages after it running on state that never happened.Opens in a new tab.

Sources