Queue vs topic

2 min read
On trails:EDAstep 4/8

Messaging systems offer two delivery shapes. In a queue, each message goes to exactly one consumer. In a topic, every subscriber gets its own copy of each message.

Queue

Several workers read from one queue and each message is handled by only one of them, so the workers compete for messages. This is the competing consumers pattern, and it is how work is shared: if Payments falls behind, start more Payments workers on the same queue. The producer and consumers also do not need to run at the same time, because the queue holds messages until a worker is ready. Messages are taken in order, but with several workers they can finish in any order, see message orderingMessage orderingOrder exists only within one key, and only while one consumer handles that key at a time.Opens in a new tab.

A queue fits a command, such as ChargeCard, where one owner must do the work once.

Topic

A producer publishes to a topic, and each subscriber receives a copy. Orders publishes OrderPlaced, and Payments, Inventory and Notifications each get it, without Orders knowing about any of them. In Azure Service Bus, consumers read from a subscription, which behaves like a virtual queue that receives copies of the topic’s messages. In RabbitMQ the equivalent is an exchange that copies each message into every queue bound to it, see routing and topologyRouting and topologyHow a messaging system decides where each message goes: exchange types, subscription filters and partition keys.Opens in a new tab.

A topic fits an event. A subscriber that did not exist when a message was published does not receive it, which is one reason streams exist, see broker vs event streamBroker vs event streamA broker hands messages out and removes them once acknowledged, a stream keeps a log that consumers read at their own pace.Opens in a new tab.

Orders publishes OrderPlaced to a topic. The topic copies it into one queue per subscriber: payments, inventory and notifications. Each queue is read by that service's workers, two for Payments and one each for the others. ONE COPY EACH Orders topic OrderPlaced payments queue inventory queue notifications queue Payments 2 workers Inventory 1 worker Notifications 1 worker each subscriber gets its own queue, so a slow service does not hold up the others
Queues and topics are used together.

They nest

A real system uses both. The topic copies each event into one queue per subscribing service, and that service’s workers compete on its own queue. Each service scales its workers independently, and a backlog in one service’s queue does not slow the others. This is also how a Service Bus subscription works: it receives copies from the topic and supports the same competing consumer pattern as a queue.

The catch

A subscriber’s queue exists only from the moment it is created. A new service subscribing today does not see yesterday’s events in a plain topic. Also, a queue does not by itself say which kind of message it holds. Putting an event on a queue read by one service quietly turns it into a command to that service, see events, commands and queriesEvents, commands and queriesThree kinds of message: an event states a fact, a command asks one owner to act, a query only reads.Opens in a new tab.

Sources