Message Queues & Streaming Interview Questions and Answers

• Queue: point-to-point. Each message is delivered to exactly one consumer from a pool, which is ideal for task distribution and load balancing.
• Publish/subscribe: a message is fanned out to all subscribers. Each subscriber gets a copy, which suits event notification.
• Stream (append-only log): messages are retained and ordered, and consumers track their own position. Multiple independent consumers can read the same data, and you can replay history.

Queue:   producer -> [ Q ] -> one of many workers
Pub/Sub: producer -> topic -> every subscriber
Stream:  producer -> [ 0 1 2 3 4 ] <- consumers seek/replay

Kafka and Pulsar are logs; RabbitMQ and SQS are queues (SQS plus SNS gives pub/sub). The distinction affects semantics: a queue usually deletes a message after acknowledgment, while a stream keeps it until a retention policy expires, enabling replay, reprocessing and event sourcing.

A topic is a named stream of records. It is split into partitions, which are the unit of ordering, parallelism and replication. Records within a partition are strictly ordered and appended; each gets a monotonically increasing offset.

Consumers read from a partition and commit offsets to remember their position. A consumer group assigns each partition to exactly one consumer, so parallelism is capped by the partition count.

topic: orders
  partition 0: [0][1][2][3]
  partition 1: [0][1][2]

The message key determines the partition, so all records with the same key land in the same partition and preserve order. That is how you keep per-customer or per-order ordering while still scaling. Partitions also determine replication: each has a leader and followers, and the leader handles reads and writes. Increasing partitions later is possible but changes key-to-partition mapping, so plan capacity up front.

A dead-letter queue (DLQ) is a separate queue that receives messages a consumer cannot process after exhausting retries. It prevents one poison message from blocking a partition or queue forever and preserves the payload for diagnosis and replay.

Typical setup: a consumer retries a few times with backoff, then routes the message to the DLQ along with metadata such as the original topic, error, attempt count and timestamp.

retry:
  max-attempts: 5
  backoff: exponential
dead-letter:
  queue: orders.dlq

Operate DLQs actively: alert on non-zero depth, inspect and classify failures, fix the bug, then redrive messages after the fix. Without a redrive process a DLQ becomes a silent graveyard. Distinguish transient failures (network, downstream outage) that deserve retries from permanent ones (malformed schema, unknown type) that should skip retries. Also cap message retention and age.