How would you design a change data capture pipeline?
Interviewer Expectations for this Question
01
Core Competency
Assesses fundamental understanding of Data Engineering conventions, runtime behavior, and memory/performance considerations.
02
Evaluation Criteria
Hiring managers look for precision, avoidance of ambiguous jargon, and ability to explain trade-offs under real production conditions.
Comprehensive Model Answer
Verified Solution
CDC captures row-level changes from a source database and streams them downstream with low latency. A typical design:
- Capture: read the database transaction log with a tool like Debezium on the MySQL binlog or Postgres logical replication, rather than polling, so you get every insert, update and delete without load on the source.
- Transport: publish change events to Kafka, keyed by primary key to preserve per-entity ordering.
- Process: consume events, land them in a bronze table, then merge into silver entities. Handle deletes, schema evolution and ordering.
- Serve: expose current state and history for analytics.
{"op":"u","before":{"id":42,"city":"Paris"},"after":{"id":42,"city":"Lyon"}}
Considerations: initial snapshot plus streaming handoff, idempotent merges, tombstones for deletes, and monitoring replication lag.
Candidate Response Strategy & Interview Tips
- Start with a concise one-sentence summary: Deliver a direct, confident answer first before expanding into nuances.
- Demonstrate real-world trade-offs: Discuss where this approach excels and when you would avoid it in production systems.
- Discuss complexity & edge cases: Proactively explain time/space complexity or boundary conditions (null values, scale limits).
- Prepare for interviewer follow-ups: Technical hiring panels frequently probe deeper into concurrency, backward compatibility, or alternative libraries.