Dead letters and ordered messages

3 min read
On trails:EDARELstep 7/9

A dead-letter queueDead-letter queuesWhere a message goes when it cannot be processed, so the rest of the traffic keeps moving.Opens in a new tab takes one message out of the flow. The messages for the same order that follow it keep flowing, and they were written on the assumption that the missing one happened. Setting a message aside breaks the order its neighbours depend on.

An example

A consumer that tracks order 42 receives OrderPlaced, PaymentReceived and OrderShipped, in that order. PaymentReceived fails and is dead-lettered. OrderShipped arrives next and is processed.

Messages for order 42 arrive in order. PaymentReceived fails and goes to the dead-letter queue, OrderShipped is processed without it, and after the fix PaymentReceived is replayed and arrives after OrderShipped. ORDER 42, ONE KEY OrderPlaced sequence 1 PaymentReceived sequence 2 OrderShipped sequence 3 processed fails processed, payment missing Dead-letter queue sequence 2 replayed after the fix PaymentReceived arrives last OrderShipped ran without the payment, and the replayed payment now lands after it
The failed message is out of the flow, and when it returns it is late.

Three kinds of damage

  • Orphan. A later message refers to something the failed one should have created. PaymentReceived arrives for an order whose OrderPlaced is in the DLQ, and the payment is stored with no order to attach to.
  • Crash. The handler assumes the earlier state exists, throws on a missing order, is retried, and is dead-lettered too. Every later message for that order follows it into the DLQ.
  • Silent corruption. The later message succeeds against the wrong state. OrderCancelled is applied while PaymentReceived sits in the DLQ. After the fix, the replayed payment sets the status back to Paid on a cancelled order. Nothing failed, and the data is wrong.

Remedies

RemedyWhat it doesCost
Remember poisoned correlation IDsStore the ID when a message is dead-lettered. Park later messages that carry itA store read on every message, and a rule for removing IDs
Publish a failure eventTell other services the step failed, so they or a saga can compensateConsumers must handle the new event
Key by entity, block that keyKeep an entity in one lane and stop only that keyPer-key bookkeeping. Pausing a whole partition stops other keys too
Sequence numbers with dependsOnEach message carries its sequence number and the message it needs. A consumer parks one whose dependency has not been appliedThe producer assigns the numbers and the consumer tracks what it applied
Replay by correlation ID, in orderAfter the fix, resend the dead and parked messages for one ID in sequence order, then unblock itA replay tool, and idempotent consumers
Table 1: five ways to contain a dead letter

Remember and announce

The first two remedies fit in one handler. When a message fails for a permanent reason, record its correlation ID and dead-letter it. Every later message first checks that record and goes to a parking queue if the ID is in it. The handler also publishes a failure event, so the rest of the system hears about it (see sagasSagasA workflow made of local transactions, with an undo step for each one.Opens in a new tab for what a consumer does with it).

PaymentReceivedGuard.csC#
public async Task HandleAsync(PaymentReceived msg, CancellationToken ct){    if (await poisoned.ContainsAsync(msg.CorrelationId, ct))    {        await parking.SendAsync(msg, "an earlier message for this order failed", ct);        return;    }     try    {        await handler.HandleAsync(msg, ct);    }    catch (Exception ex) when (!IsTransient(ex))    {        await poisoned.AddAsync(msg.CorrelationId, ct);        await deadLetters.SendAsync(msg, ex, ct);        await events.PublishAsync(new PaymentFailed(msg.OrderId, ex.Message), ct);    }}

Block the key, not the partition

In Kafka, keying messages by order ID sends every message for order 42 to one partition, which keeps them in order. A partition is not reserved for one order, though. A partition holds many keys. Pausing it with Pause in Confluent.Kafka stops every order whose key hashes there. To block one order, the consumer keeps a set of blocked keys and diverts their messages to a parking topic while the other keys carry on.

Service Bus sessions behave differently. When a message in a session is dead-lettered, the receiver carries on with the next message in that session, so the application has to do the blocking there as well.

Replay in the original order

Replaying a dead letter normally puts it back at the end of the line. On Service Bus a resubmitted message gets a new enqueue time and sequence number, so the original order is lost. Order has to come from the payload: the sequence number, or the version numbers from message orderingMessage orderingOrder exists only within one key, and only while one consumer handles that key at a time.Opens in a new tab. Replay every dead and parked message for one correlation ID in sequence order, then remove the ID from the poisoned set. Because a replay is a redelivery, the consumers must be idempotentIdempotent consumersA consumer that gives the same result whether a message arrives once or five times.Opens in a new tab.

Sources