Data Engineering Interview Questions and Answers

Late data breaks the assumption that all events for a time window have arrived. Common strategies:

• Watermarks: define how long to wait for late events before closing a window. A ten-minute watermark holds windows open ten minutes past the latest event time. Events after that are dropped or routed separately.
• Allowed lateness: keep window state longer and update results when late events arrive, emitting revised aggregates downstream.
• Reprocessing: store raw immutable events in the lake, then rerun the affected partition to correct results. Idempotent writes and partitioning by event date make this safe.
• Reconciliation: combine a streaming near-real-time view with a batch correction that overwrites the same partitions.

The right choice balances latency, cost and correctness. Billing pipelines usually favor a batch correction layer, while dashboards may accept approximate streaming results.

CDC captures row-level changes from a source database and streams them downstream with low latency. A typical design:

• Capture: read the database transaction log with a tool like Debezium on the MySQL binlog or Postgres logical replication, rather than polling, so you get every insert, update and delete without load on the source.
• Transport: publish change events to Kafka, keyed by primary key to preserve per-entity ordering.
• Process: consume events, land them in a bronze table, then merge into silver entities. Handle deletes, schema evolution and ordering.
• Serve: expose current state and history for analytics.

{"op":"u","before":{"id":42,"city":"Paris"},"after":{"id":42,"city":"Lyon"}}

Considerations: initial snapshot plus streaming handoff, idempotent merges, tombstones for deletes, and monitoring replication lag.