How does Kafka provide exactly-once semantics?
Assesses fundamental understanding of Message Queues & Streaming conventions, runtime behavior, and memory/performance considerations.
Hiring managers look for precision, avoidance of ambiguous jargon, and ability to explain trade-offs under real production conditions.
Kafka combines two features:
- Idempotent producer: each producer gets a producer id and sequence numbers. The broker deduplicates retries within a session, so a network retry does not append the same record twice.
- Transactions: the producer atomically writes to multiple partitions and commits consumer offsets in one transaction. Consumers with
isolation.level=read_committedonly see committed data and do not read aborted or uncommitted records.
enable.idempotence=true
transactional.id=order-processor-1
isolation.level=read_committed
This gives exactly-once processing inside a read-process-write Kafka Streams pipeline. It does not extend to external systems: writing to a database or calling an API can still duplicate or partially commit. For those, use the outbox pattern with an idempotent sink or a two-phase commit-free design.
The cost is higher latency and complexity, so use transactions only where duplicates are genuinely unacceptable and simpler idempotent consumers cannot solve the problem.
Candidate Response Strategy & Interview Tips
- Start with a concise one-sentence summary: Deliver a direct, confident answer first before expanding into nuances.
- Demonstrate real-world trade-offs: Discuss where this approach excels and when you would avoid it in production systems.
- Discuss complexity & edge cases: Proactively explain time/space complexity or boundary conditions (null values, scale limits).
- Prepare for interviewer follow-ups: Technical hiring panels frequently probe deeper into concurrency, backward compatibility, or alternative libraries.