Streaming data platform
Real-Time FinTech Analytics & Risk Monitoring Platform
Risk monitoring is only as good as the data behind it. This platform treats validation as a pipeline stage, not an afterthought, so every downstream table inherits clean, deduplicated records.
How the data moves
- KafkaEvents arrive as a stream
- PySpark Structured StreamingValidate, enrich, deduplicate
- S3 raw and curatedMedallion layers
- SnowflakeELT transformations
- Analytics and risk viewsDownstream consumers
AirflowRuns batch and streaming workflows as DAGs with monitoring, alerting and retry logic.
What this covers
- End-to-end streaming pipelines on Kafka and PySpark Structured Streaming, with validation, enrichment and deduplication applied at scale.
- ELT transformation workflows in Snowflake, reading from raw and curated layers in AWS S3 laid out as a medallion architecture.
- Hybrid batch and streaming orchestration in Airflow, with DAG-level monitoring, alerting and retry logic.
Design decisions
Clean data in the stream
Validation, enrichment and deduplication run inside the streaming job. Bad and duplicate records are handled before they reach storage, so analysts never have to filter them out later.
Raw and curated layers
Raw events stay in S3 untouched and can be replayed. Curated tables are derived from them, so a fix to transformation logic never means re-collecting data.
Orchestration with retries and alerts
Airflow ties batch and streaming work together. Retries absorb transient failures and alerting surfaces problems early, which protects SLA adherence.