How should retries and poison messages be handled?
Interviewer Expectations for this Question
01
Core Competency
Assesses fundamental understanding of Message Queues & Streaming conventions, runtime behavior, and memory/performance considerations.
02
Evaluation Criteria
Hiring managers look for precision, avoidance of ambiguous jargon, and ability to explain trade-offs under real production conditions.
Comprehensive Model Answer
Verified Solution
Retrying immediately in a tight loop amplifies outages. Instead:
- Classify errors: retry transient ones (timeouts, 503, deadlocks) and stop retrying permanent ones (schema violations, 400-class errors).
- Retry with exponential backoff and jitter, and cap attempts.
- Use delayed retry queues or scheduled redelivery so the broker, not the consumer, holds the delay.
- After the cap, route to a dead-letter queue with the error and attempt count, and alert.
max-attempts: 5
backoff: "1s, 5s, 30s, 5m"
Ensure the consumer is idempotent, since retries can process the same message twice. Watch out for the head-of-line problem: a message that always fails can block a partition if retried in place, so move it aside quickly. Also add circuit breakers for downstream calls so retries do not pile onto a service that is already failing, and load-test the retry path.
Candidate Response Strategy & Interview Tips
- Start with a concise one-sentence summary: Deliver a direct, confident answer first before expanding into nuances.
- Demonstrate real-world trade-offs: Discuss where this approach excels and when you would avoid it in production systems.
- Discuss complexity & edge cases: Proactively explain time/space complexity or boundary conditions (null values, scale limits).
- Prepare for interviewer follow-ups: Technical hiring panels frequently probe deeper into concurrency, backward compatibility, or alternative libraries.