Pranaav Srinivasan — Blogs https://pranaav.pages.dev/blogs Notes on data engineering and applied AI. en Why a streaming pipeline needs an idempotent sink https://pranaav.pages.dev/blogs/idempotent-sinks https://pranaav.pages.dev/blogs/idempotent-sinks Thu, 05 Nov 2026 00:00:00 GMT Kafka delivers at least once, so duplicates are normal. Here is how deduplication in the stream and an idempotent write at the end keep a pipeline correct.