Run a mock round

Producers publish to an exchange, which routes to queues via bindings. The exchange type decides the routing:

• Direct: routes by exact routing key match. Common for task queues.
• Fanout: ignores the key and copies to every bound queue, which is pub/sub.
• Topic: matches routing keys against wildcard patterns where * is one word and # is zero or more words, for example orders.# or *.error.
• Headers: routes on message header attributes rather than the routing key.

exchange (topic) --binding "order.created"--> queue A
                 --binding "order.*"       --> queue B

A queue can be bound to multiple exchanges, and unroutable messages are dropped unless a mandatory flag or an alternate exchange is configured. Default exchange routes directly to a queue named by the routing key. Choose topic exchanges for flexible event routing, fanout for broadcasting and direct for simple work distribution.

• Queue: point-to-point. Each message is delivered to exactly one consumer from a pool, which is ideal for task distribution and load balancing.
• Publish/subscribe: a message is fanned out to all subscribers. Each subscriber gets a copy, which suits event notification.
• Stream (append-only log): messages are retained and ordered, and consumers track their own position. Multiple independent consumers can read the same data, and you can replay history.

Queue:   producer -> [ Q ] -> one of many workers
Pub/Sub: producer -> topic -> every subscriber
Stream:  producer -> [ 0 1 2 3 4 ] <- consumers seek/replay

Kafka and Pulsar are logs; RabbitMQ and SQS are queues (SQS plus SNS gives pub/sub). The distinction affects semantics: a queue usually deletes a message after acknowledgment, while a stream keeps it until a retention policy expires, enabling replay, reprocessing and event sourcing.

Retrying immediately in a tight loop amplifies outages. Instead:

• Classify errors: retry transient ones (timeouts, 503, deadlocks) and stop retrying permanent ones (schema violations, 400-class errors).
• Retry with exponential backoff and jitter, and cap attempts.
• Use delayed retry queues or scheduled redelivery so the broker, not the consumer, holds the delay.
• After the cap, route to a dead-letter queue with the error and attempt count, and alert.

max-attempts: 5
backoff: "1s, 5s, 30s, 5m"

Ensure the consumer is idempotent, since retries can process the same message twice. Watch out for the head-of-line problem: a message that always fails can block a partition if retried in place, so move it aside quickly. Also add circuit breakers for downstream calls so retries do not pile onto a service that is already failing, and load-test the retry path.

• At-most-once: the broker may deliver a message once or not at all. It never duplicates, so it can lose data. Achieved by acknowledging before processing. Suitable for metrics or non-critical telemetry.
• At-least-once: the broker retries until acknowledged, so nothing is lost, but duplicates are possible. This is the practical default; consumers must be idempotent or deduplicate.
• Exactly-once: each message affects state once. True end-to-end exactly-once across independent systems is extremely hard; Kafka achieves it within its own ecosystem using an idempotent producer and transactions, but once data leaves Kafka for an external system, exactly-once becomes at-least-once plus idempotent writes.

ack before work -> at-most-once
ack after work  -> at-least-once (retry on crash)

Most teams should design for at-least-once and make consumers idempotent, rather than paying the complexity cost of chasing true exactly-once semantics.

A service often must both change its database and publish an event. Writing to the database and then to the broker is a dual write: a crash between the two loses the event, and publishing first can announce a change that never commits.

The outbox pattern writes the business data and an outbox record in the same local transaction. A separate relay then publishes outbox rows to the broker and marks them sent.

BEGIN;
UPDATE orders SET status = 'paid' WHERE id = :id;
INSERT INTO outbox (id, type, payload) VALUES (:id, 'order.paid', :json);
COMMIT;

The relay can poll the table or use change data capture (Debezium) to stream inserts. Publishing must be idempotent because the relay may send a row twice, so consumers deduplicate by event id. This guarantees at-least-once publication without distributed transactions, and it pairs naturally with sagas and event-driven read models. Clean up delivered rows and monitor outbox lag.

Ordering is usually per partition or per queue, not global.

• Kafka orders records within a partition only. Records with the same key go to the same partition, preserving per-key order. Across partitions there is no order guarantee.
• RabbitMQ keeps FIFO order per queue when a single consumer processes it; multiple competing consumers can interleave messages.
• SQS standard queues offer best-effort ordering; FIFO queues provide ordering within a message group id.

key = customerId -> same partition -> ordered per customer

If you need strict order, ensure single-consumer processing per partition or queue, and avoid parallel retries that can reorder. A common failure is processing per-key events across multiple partitions and assuming they arrive in order. When global order is required, it is usually a sign to serialise on one key or to use timestamps and conflict resolution rather than depending on transport order.

At-least-once delivery means duplicates will happen, so a consumer must produce the same result whether it sees a message once or several times.

Techniques:

• Deduplication table: store processed message ids with a unique constraint; skip ids already seen. Use the insert result to detect duplicates atomically.
• Natural idempotency: use deterministic upserts keyed by business identity, such as INSERT ... ON CONFLICT DO UPDATE.
• State-machine guards: only apply a transition if the current state allows it, for example "mark shipped" only from "paid".
• Idempotent side effects: send a notification keyed by an event id, or use conditional writes.

INSERT INTO processed_messages (id) VALUES (:id)
ON CONFLICT (id) DO NOTHING;

Tie deduplication to the same transaction as the state change so the marker and the effect commit together. Keep the deduplication window at least as long as the retry horizon, and design compensations for effects that cannot be undone, such as emails already delivered.

Kafka combines two features:

• Idempotent producer: each producer gets a producer id and sequence numbers. The broker deduplicates retries within a session, so a network retry does not append the same record twice.
• Transactions: the producer atomically writes to multiple partitions and commits consumer offsets in one transaction. Consumers with isolation.level=read_committed only see committed data and do not read aborted or uncommitted records.

enable.idempotence=true
transactional.id=order-processor-1
isolation.level=read_committed

This gives exactly-once processing inside a read-process-write Kafka Streams pipeline. It does not extend to external systems: writing to a database or calling an API can still duplicate or partially commit. For those, use the outbox pattern with an idempotent sink or a two-phase commit-free design.

The cost is higher latency and complexity, so use transactions only where duplicates are genuinely unacceptable and simpler idempotent consumers cannot solve the problem.

Consumer lag is the gap between the latest produced offset and the committed offset. Rising lag means producers outpace consumers, and left unchecked it grows unbounded and delays downstream data.

Diagnose first: is it a traffic spike, a slow downstream, a hot partition or expensive per-message work? Then:

• Scale consumers up to the partition count; beyond that, add partitions.
• Batch and parallelise processing where ordering allows, and increase fetch sizes.
• Optimise the slow dependency, add caching or bulk writes.
• Apply backpressure to producers: rate limit, buffer with bounded queues, or shed non-critical load.
• Move poison messages to a dead-letter queue so one record cannot stall a partition.
• Alert on lag as an SLI and cap retention so unbounded growth is visible.

lag = latest offset - committed offset

Prefer design that keeps consumers fast and stateless so scale-out actually helps, and load-test the pipeline before peak events.