Why a streaming pipeline needs an idempotent sink
Kafka delivers at least once, so duplicates are normal. Here is how deduplication in the stream and an idempotent write at the end keep a pipeline correct.
Blogs
Notes on data engineering and applied AI. Short, practical and written after the thing worked.
Published
RSS feedOn the list
-
Building production RAG
What goes into the index, and why the data pipeline decides the answer quality.
Planned -
Scaling Kafka pipelines past 10M records a day
Partitioning, back-pressure and the checks that catch problems before users do.
Planned -
Lessons from shipping ML in six months
What changed between a notebook that works and a service that stays up.
Planned
Want one of these sooner? Tell me which.