Message Queues & Streaming Interview Questions and Answers

Kafka combines two features:

• Idempotent producer: each producer gets a producer id and sequence numbers. The broker deduplicates retries within a session, so a network retry does not append the same record twice.
• Transactions: the producer atomically writes to multiple partitions and commits consumer offsets in one transaction. Consumers with isolation.level=read_committed only see committed data and do not read aborted or uncommitted records.

enable.idempotence=true
transactional.id=order-processor-1
isolation.level=read_committed

This gives exactly-once processing inside a read-process-write Kafka Streams pipeline. It does not extend to external systems: writing to a database or calling an API can still duplicate or partially commit. For those, use the outbox pattern with an idempotent sink or a two-phase commit-free design.

The cost is higher latency and complexity, so use transactions only where duplicates are genuinely unacceptable and simpler idempotent consumers cannot solve the problem.

A service often must both change its database and publish an event. Writing to the database and then to the broker is a dual write: a crash between the two loses the event, and publishing first can announce a change that never commits.

The outbox pattern writes the business data and an outbox record in the same local transaction. A separate relay then publishes outbox rows to the broker and marks them sent.

BEGIN;
UPDATE orders SET status = 'paid' WHERE id = :id;
INSERT INTO outbox (id, type, payload) VALUES (:id, 'order.paid', :json);
COMMIT;

The relay can poll the table or use change data capture (Debezium) to stream inserts. Publishing must be idempotent because the relay may send a row twice, so consumers deduplicate by event id. This guarantees at-least-once publication without distributed transactions, and it pairs naturally with sagas and event-driven read models. Clean up delivered rows and monitor outbox lag.