Message Queues & Streaming Interview Questions and Answers

• Queue: point-to-point. Each message is delivered to exactly one consumer from a pool, which is ideal for task distribution and load balancing.
• Publish/subscribe: a message is fanned out to all subscribers. Each subscriber gets a copy, which suits event notification.
• Stream (append-only log): messages are retained and ordered, and consumers track their own position. Multiple independent consumers can read the same data, and you can replay history.

Queue:   producer -> [ Q ] -> one of many workers
Pub/Sub: producer -> topic -> every subscriber
Stream:  producer -> [ 0 1 2 3 4 ] <- consumers seek/replay

Kafka and Pulsar are logs; RabbitMQ and SQS are queues (SQS plus SNS gives pub/sub). The distinction affects semantics: a queue usually deletes a message after acknowledgment, while a stream keeps it until a retention policy expires, enabling replay, reprocessing and event sourcing.

A topic is a named stream of records. It is split into partitions, which are the unit of ordering, parallelism and replication. Records within a partition are strictly ordered and appended; each gets a monotonically increasing offset.

Consumers read from a partition and commit offsets to remember their position. A consumer group assigns each partition to exactly one consumer, so parallelism is capped by the partition count.

topic: orders
  partition 0: [0][1][2][3]
  partition 1: [0][1][2]

The message key determines the partition, so all records with the same key land in the same partition and preserve order. That is how you keep per-customer or per-order ordering while still scaling. Partitions also determine replication: each has a leader and followers, and the leader handles reads and writes. Increasing partitions later is possible but changes key-to-partition mapping, so plan capacity up front.

A dead-letter queue (DLQ) is a separate queue that receives messages a consumer cannot process after exhausting retries. It prevents one poison message from blocking a partition or queue forever and preserves the payload for diagnosis and replay.

Typical setup: a consumer retries a few times with backoff, then routes the message to the DLQ along with metadata such as the original topic, error, attempt count and timestamp.

retry:
  max-attempts: 5
  backoff: exponential
dead-letter:
  queue: orders.dlq

Operate DLQs actively: alert on non-zero depth, inspect and classify failures, fix the bug, then redrive messages after the fix. Without a redrive process a DLQ becomes a silent graveyard. Distinguish transient failures (network, downstream outage) that deserve retries from permanent ones (malformed schema, unknown type) that should skip retries. Also cap message retention and age.

• At-most-once: the broker may deliver a message once or not at all. It never duplicates, so it can lose data. Achieved by acknowledging before processing. Suitable for metrics or non-critical telemetry.
• At-least-once: the broker retries until acknowledged, so nothing is lost, but duplicates are possible. This is the practical default; consumers must be idempotent or deduplicate.
• Exactly-once: each message affects state once. True end-to-end exactly-once across independent systems is extremely hard; Kafka achieves it within its own ecosystem using an idempotent producer and transactions, but once data leaves Kafka for an external system, exactly-once becomes at-least-once plus idempotent writes.

ack before work -> at-most-once
ack after work  -> at-least-once (retry on crash)

Most teams should design for at-least-once and make consumers idempotent, rather than paying the complexity cost of chasing true exactly-once semantics.

A consumer group is a set of consumers that cooperatively consume a topic. Kafka assigns each partition to exactly one consumer in the group, so the group as a whole processes every record once, while multiple consumers share the load.

group "billing":  consumer A -> partition 0
                  consumer B -> partition 1
                  consumer C -> partition 2

Scaling works up to the number of partitions: a fourth consumer would sit idle. If consumers leave, crash or join, Kafka triggers a rebalance and reassigns partitions. Cooperative rebalancing reduces stop-the-world pauses compared with the older eager protocol.

Different groups are independent; each maintains its own offsets, so the same topic can feed billing, analytics and search independently. Key pitfalls: keep processing bounded so a member is not evicted for missing heartbeats, avoid blocking the poll loop for long periods, and commit offsets after successful processing to preserve at-least-once semantics.

Ordering is usually per partition or per queue, not global.

• Kafka orders records within a partition only. Records with the same key go to the same partition, preserving per-key order. Across partitions there is no order guarantee.
• RabbitMQ keeps FIFO order per queue when a single consumer processes it; multiple competing consumers can interleave messages.
• SQS standard queues offer best-effort ordering; FIFO queues provide ordering within a message group id.

key = customerId -> same partition -> ordered per customer

If you need strict order, ensure single-consumer processing per partition or queue, and avoid parallel retries that can reorder. A common failure is processing per-key events across multiple partitions and assuming they arrive in order. When global order is required, it is usually a sign to serialise on one key or to use timestamps and conflict resolution rather than depending on transport order.

Consumer lag is the gap between the latest produced offset and the committed offset. Rising lag means producers outpace consumers, and left unchecked it grows unbounded and delays downstream data.

Diagnose first: is it a traffic spike, a slow downstream, a hot partition or expensive per-message work? Then:

• Scale consumers up to the partition count; beyond that, add partitions.
• Batch and parallelise processing where ordering allows, and increase fetch sizes.
• Optimise the slow dependency, add caching or bulk writes.
• Apply backpressure to producers: rate limit, buffer with bounded queues, or shed non-critical load.
• Move poison messages to a dead-letter queue so one record cannot stall a partition.
• Alert on lag as an SLI and cap retention so unbounded growth is visible.

lag = latest offset - committed offset

Prefer design that keeps consumers fast and stateless so scale-out actually helps, and load-test the pipeline before peak events.

Producers publish to an exchange, which routes to queues via bindings. The exchange type decides the routing:

• Direct: routes by exact routing key match. Common for task queues.
• Fanout: ignores the key and copies to every bound queue, which is pub/sub.
• Topic: matches routing keys against wildcard patterns where * is one word and # is zero or more words, for example orders.# or *.error.
• Headers: routes on message header attributes rather than the routing key.

exchange (topic) --binding "order.created"--> queue A
                 --binding "order.*"       --> queue B

A queue can be bound to multiple exchanges, and unroutable messages are dropped unless a mandatory flag or an alternate exchange is configured. Default exchange routes directly to a queue named by the routing key. Choose topic exchanges for flexible event routing, fanout for broadcasting and direct for simple work distribution.

Retrying immediately in a tight loop amplifies outages. Instead:

• Classify errors: retry transient ones (timeouts, 503, deadlocks) and stop retrying permanent ones (schema violations, 400-class errors).
• Retry with exponential backoff and jitter, and cap attempts.
• Use delayed retry queues or scheduled redelivery so the broker, not the consumer, holds the delay.
• After the cap, route to a dead-letter queue with the error and attempt count, and alert.

max-attempts: 5
backoff: "1s, 5s, 30s, 5m"

Ensure the consumer is idempotent, since retries can process the same message twice. Watch out for the head-of-line problem: a message that always fails can block a partition if retried in place, so move it aside quickly. Also add circuit breakers for downstream calls so retries do not pile onto a service that is already failing, and load-test the retry path.

At-least-once delivery means duplicates will happen, so a consumer must produce the same result whether it sees a message once or several times.

Techniques:

• Deduplication table: store processed message ids with a unique constraint; skip ids already seen. Use the insert result to detect duplicates atomically.
• Natural idempotency: use deterministic upserts keyed by business identity, such as INSERT ... ON CONFLICT DO UPDATE.
• State-machine guards: only apply a transition if the current state allows it, for example "mark shipped" only from "paid".
• Idempotent side effects: send a notification keyed by an event id, or use conditional writes.

INSERT INTO processed_messages (id) VALUES (:id)
ON CONFLICT (id) DO NOTHING;

Tie deduplication to the same transaction as the state change so the marker and the effect commit together. Keep the deduplication window at least as long as the retry horizon, and design compensations for effects that cannot be undone, such as emails already delivered.

Kafka combines two features:

• Idempotent producer: each producer gets a producer id and sequence numbers. The broker deduplicates retries within a session, so a network retry does not append the same record twice.
• Transactions: the producer atomically writes to multiple partitions and commits consumer offsets in one transaction. Consumers with isolation.level=read_committed only see committed data and do not read aborted or uncommitted records.

enable.idempotence=true
transactional.id=order-processor-1
isolation.level=read_committed

This gives exactly-once processing inside a read-process-write Kafka Streams pipeline. It does not extend to external systems: writing to a database or calling an API can still duplicate or partially commit. For those, use the outbox pattern with an idempotent sink or a two-phase commit-free design.

The cost is higher latency and complexity, so use transactions only where duplicates are genuinely unacceptable and simpler idempotent consumers cannot solve the problem.

A service often must both change its database and publish an event. Writing to the database and then to the broker is a dual write: a crash between the two loses the event, and publishing first can announce a change that never commits.

The outbox pattern writes the business data and an outbox record in the same local transaction. A separate relay then publishes outbox rows to the broker and marks them sent.

BEGIN;
UPDATE orders SET status = 'paid' WHERE id = :id;
INSERT INTO outbox (id, type, payload) VALUES (:id, 'order.paid', :json);
COMMIT;

The relay can poll the table or use change data capture (Debezium) to stream inserts. Publishing must be idempotent because the relay may send a row twice, so consumers deduplicate by event id. This guarantees at-least-once publication without distributed transactions, and it pairs naturally with sagas and event-driven read models. Clean up delivered rows and monitor outbox lag.