Run a mock round

A service often must both change its database and publish an event. Writing to the database and then to the broker is a dual write: a crash between the two loses the event, and publishing first can announce a change that never commits.

The outbox pattern writes the business data and an outbox record in the same local transaction. A separate relay then publishes outbox rows to the broker and marks them sent.

BEGIN;
UPDATE orders SET status = 'paid' WHERE id = :id;
INSERT INTO outbox (id, type, payload) VALUES (:id, 'order.paid', :json);
COMMIT;

The relay can poll the table or use change data capture (Debezium) to stream inserts. Publishing must be idempotent because the relay may send a row twice, so consumers deduplicate by event id. This guarantees at-least-once publication without distributed transactions, and it pairs naturally with sagas and event-driven read models. Clean up delivered rows and monitor outbox lag.

A topic is a named stream of records. It is split into partitions, which are the unit of ordering, parallelism and replication. Records within a partition are strictly ordered and appended; each gets a monotonically increasing offset.

Consumers read from a partition and commit offsets to remember their position. A consumer group assigns each partition to exactly one consumer, so parallelism is capped by the partition count.

topic: orders
  partition 0: [0][1][2][3]
  partition 1: [0][1][2]

The message key determines the partition, so all records with the same key land in the same partition and preserve order. That is how you keep per-customer or per-order ordering while still scaling. Partitions also determine replication: each has a leader and followers, and the leader handles reads and writes. Increasing partitions later is possible but changes key-to-partition mapping, so plan capacity up front.

Producers publish to an exchange, which routes to queues via bindings. The exchange type decides the routing:

• Direct: routes by exact routing key match. Common for task queues.
• Fanout: ignores the key and copies to every bound queue, which is pub/sub.
• Topic: matches routing keys against wildcard patterns where * is one word and # is zero or more words, for example orders.# or *.error.
• Headers: routes on message header attributes rather than the routing key.

exchange (topic) --binding "order.created"--> queue A
                 --binding "order.*"       --> queue B

A queue can be bound to multiple exchanges, and unroutable messages are dropped unless a mandatory flag or an alternate exchange is configured. Default exchange routes directly to a queue named by the routing key. Choose topic exchanges for flexible event routing, fanout for broadcasting and direct for simple work distribution.

A consumer group is a set of consumers that cooperatively consume a topic. Kafka assigns each partition to exactly one consumer in the group, so the group as a whole processes every record once, while multiple consumers share the load.

group "billing":  consumer A -> partition 0
                  consumer B -> partition 1
                  consumer C -> partition 2

Scaling works up to the number of partitions: a fourth consumer would sit idle. If consumers leave, crash or join, Kafka triggers a rebalance and reassigns partitions. Cooperative rebalancing reduces stop-the-world pauses compared with the older eager protocol.

Different groups are independent; each maintains its own offsets, so the same topic can feed billing, analytics and search independently. Key pitfalls: keep processing bounded so a member is not evicted for missing heartbeats, avoid blocking the poll loop for long periods, and commit offsets after successful processing to preserve at-least-once semantics.

Kafka combines two features:

• Idempotent producer: each producer gets a producer id and sequence numbers. The broker deduplicates retries within a session, so a network retry does not append the same record twice.
• Transactions: the producer atomically writes to multiple partitions and commits consumer offsets in one transaction. Consumers with isolation.level=read_committed only see committed data and do not read aborted or uncommitted records.

enable.idempotence=true
transactional.id=order-processor-1
isolation.level=read_committed

This gives exactly-once processing inside a read-process-write Kafka Streams pipeline. It does not extend to external systems: writing to a database or calling an API can still duplicate or partially commit. For those, use the outbox pattern with an idempotent sink or a two-phase commit-free design.

The cost is higher latency and complexity, so use transactions only where duplicates are genuinely unacceptable and simpler idempotent consumers cannot solve the problem.

• Queue: point-to-point. Each message is delivered to exactly one consumer from a pool, which is ideal for task distribution and load balancing.
• Publish/subscribe: a message is fanned out to all subscribers. Each subscriber gets a copy, which suits event notification.
• Stream (append-only log): messages are retained and ordered, and consumers track their own position. Multiple independent consumers can read the same data, and you can replay history.

Queue:   producer -> [ Q ] -> one of many workers
Pub/Sub: producer -> topic -> every subscriber
Stream:  producer -> [ 0 1 2 3 4 ] <- consumers seek/replay

Kafka and Pulsar are logs; RabbitMQ and SQS are queues (SQS plus SNS gives pub/sub). The distinction affects semantics: a queue usually deletes a message after acknowledgment, while a stream keeps it until a retention policy expires, enabling replay, reprocessing and event sourcing.

Consumer lag is the gap between the latest produced offset and the committed offset. Rising lag means producers outpace consumers, and left unchecked it grows unbounded and delays downstream data.

Diagnose first: is it a traffic spike, a slow downstream, a hot partition or expensive per-message work? Then:

• Scale consumers up to the partition count; beyond that, add partitions.
• Batch and parallelise processing where ordering allows, and increase fetch sizes.
• Optimise the slow dependency, add caching or bulk writes.
• Apply backpressure to producers: rate limit, buffer with bounded queues, or shed non-critical load.
• Move poison messages to a dead-letter queue so one record cannot stall a partition.
• Alert on lag as an SLI and cap retention so unbounded growth is visible.

lag = latest offset - committed offset

Prefer design that keeps consumers fast and stateless so scale-out actually helps, and load-test the pipeline before peak events.

Ordering is usually per partition or per queue, not global.

• Kafka orders records within a partition only. Records with the same key go to the same partition, preserving per-key order. Across partitions there is no order guarantee.
• RabbitMQ keeps FIFO order per queue when a single consumer processes it; multiple competing consumers can interleave messages.
• SQS standard queues offer best-effort ordering; FIFO queues provide ordering within a message group id.

key = customerId -> same partition -> ordered per customer

If you need strict order, ensure single-consumer processing per partition or queue, and avoid parallel retries that can reorder. A common failure is processing per-key events across multiple partitions and assuming they arrive in order. When global order is required, it is usually a sign to serialise on one key or to use timestamps and conflict resolution rather than depending on transport order.

• At-most-once: the broker may deliver a message once or not at all. It never duplicates, so it can lose data. Achieved by acknowledging before processing. Suitable for metrics or non-critical telemetry.
• At-least-once: the broker retries until acknowledged, so nothing is lost, but duplicates are possible. This is the practical default; consumers must be idempotent or deduplicate.
• Exactly-once: each message affects state once. True end-to-end exactly-once across independent systems is extremely hard; Kafka achieves it within its own ecosystem using an idempotent producer and transactions, but once data leaves Kafka for an external system, exactly-once becomes at-least-once plus idempotent writes.

ack before work -> at-most-once
ack after work  -> at-least-once (retry on crash)

Most teams should design for at-least-once and make consumers idempotent, rather than paying the complexity cost of chasing true exactly-once semantics.