Eventual consistency

When one service changes its data and tells the others through an event, the others learn of it later. Until every consumer has handled the event, the services disagree about the same order. This is eventual consistency: if nothing changes further, all of them end up with the same picture. Werner Vogels calls the gap the inconsistency window, the period between an update and the moment when any observer is guaranteed to see it.

Why it happens

Each service owns its data, and no service should read another service’s tables or include them in its own transaction. That rules out one transaction across Orders, Payments and Inventory. Changes cross the boundary as eventsEvents, commands and queriesThree kinds of message: an event states a fact, a command asks one owner to act, a query only reads.Opens in a new tab, and an event takes time to be queued, delivered and handled. If a consumer is down or a message is retrying, the window stays open longer.

In the shop

Moment	Orders	Payments	Inventory
Customer submits the order	Placed	No record	No record
OrderPlaced handled	Placed	Charged	Stock reserved
PaymentReceived and StockReserved handled	Confirmed	Charged	Stock reserved

In the first two rows the services hold different facts about the order. Nothing is wrong yet: this is the window.

Designing for it

• Show the real state. Display “Order received” until Orders has handled both PaymentReceived and StockReserved, and only then “Confirmed”.
• Return the new state to the caller from the service that made the change. Asking another service straight afterwards may reach one that has not heard yet. Vogels names this guarantee read-your-writes.
• Write consumers that tolerate repeats and out-of-order arrival: see idempotent consumersIdempotent consumersA consumer that gives the same result whether a message arrives once or five times.Opens in a new tab and message orderingMessage orderingOrder exists only within one key, and only while one consumer handles that key at a time.Opens in a new tab.
• Plan for a later step failing, with a compensating action such as a refund (see sagasSagasA workflow made of local transactions, with an undo step for each one.Opens in a new tab).
• Keep what must always agree inside one service and one database transaction, for example the stock count and the reservation row.

The catch

The window is usually short, but it has no upper bound. A stopped consumer or a message in a dead-letter queue can hold it open for hours, so the size of the gap needs to be measured (see observing event flowsObserving event flowsFollowing one order through several services, and the three queue numbers that warn first.Opens in a new tab).