Continuous Data Flow Platform

FluxPipe

Batch and streaming ETL, star-schema warehousing, data quality gates, and full observability — orchestrated across 20 Docker services with a .NET 8 API, Airflow, Spark, and Kafka.

20 Docker Services Real-Time Streaming Snowflake Warehouse MLflow + Spark Multi-Cloud Deploy Helm + Terraform .NET 8 API 55 Automated Tests 3 Airflow DAGs Docker Compose Grafana Dashboards Elasticsearch Logs MongoDB + Redis PostgreSQL Analytics Kubernetes Ready Great Expectations Pipeline Insights API 20 Docker Services Real-Time Streaming Snowflake Warehouse MLflow + Spark Multi-Cloud Deploy Helm + Terraform .NET 8 API 55 Automated Tests 3 Airflow DAGs Docker Compose Grafana Dashboards Elasticsearch Logs MongoDB + Redis PostgreSQL Analytics Kubernetes Ready Great Expectations Pipeline Insights API
Get Started View Live Insights
Pipeline Insights API GET /api/insights/overview
// FluxPipe warehouse dashboard { "overallHealth": "Healthy", "totalPipelineRuns": 8, "totalRevenueLast7Days": 203367, "anomaliesLast24h": 10, "recentPipelineRuns": [ { "dagId": "batch_ingestion_dag", "status": "success" }, { "dagId": "warehouse_transform_dag", "status": "success" } ] }
0
Docker Services
0+
Technologies Integrated
0
Automated Tests
0
Deployment Targets

Platform

Overview

A production-ready data platform integrating batch and streaming processing with monitoring, governance, and machine learning capabilities.

📥

Data Ingestion

Batch Sources: MySQL, PostgreSQL, CSV/JSON/XML files, Data Lakes (MinIO/S3)

Streaming Sources: Apache Kafka for event logs, IoT sensor data, social media streams, and real-time CDC (Change Data Capture)

⚙️

Data Processing

Batch Processing: Apache Spark for large-scale ETL with Great Expectations for data quality

Stream Processing: Spark Structured Streaming for real-time transformations, anomaly detection, and event processing

💾

Multi-Tier Storage

7 Storage Systems: MinIO (S3-compatible data lake), PostgreSQL (analytics + staging), MongoDB (documents), InfluxDB (time-series), Redis (caching), Elasticsearch (search + logs), MySQL (OLTP source)

❄️

Snowflake Data Warehouse

Star Schema: 4 dimension tables, 3 fact tables, 2 aggregation tables

ETL: Automated staging, MERGE-based loading, Snowflake Tasks for scheduled aggregation

Fallback: PostgreSQL warehouse for local development

🔧

.NET 8 REST API

16 Endpoints: Batch ingestion, streaming, warehouse ETL, ML, governance, CI/CD

Enterprise: Serilog logging, Polly retry, Swagger docs, health probes (live/ready/full)

7 Controllers, 10 Services, 6 Health Checks

📊

Monitoring & ML

Observability: Prometheus (9 scrape targets), Grafana dashboards, Elasticsearch logs

MLOps: MLflow experiment tracking with PostgreSQL backend + MinIO artifacts

Quality: Great Expectations validation, Apache Atlas lineage

🚀

Multi-Provider Deployment

Local: Docker Compose (full 20-service or lite 8GB mode)

Cloud: Helm chart with AWS/GCP/Azure/on-prem value overrides

IaC: Terraform (VPC, EKS, RDS, S3) + Argo CD GitOps

CI/CD: GitHub Actions (lint, test, build, integration)

📊

BI Dashboard Integration

Platforms: Tableau, Looker, and Power BI export pipelines

Data: Queries Snowflake warehouse (or PostgreSQL fallback) and uploads aggregated analytics

Exports: Automated CSV/Parquet exports for daily orders, revenue trends, and customer segmentation

System Design

Architecture

Cloud-native, microservices-based architecture designed for scalability, reliability, and maintainability

High-Level System Architecture

graph TB subgraph Sources BS[Batch Sources - MySQL] SS[Streaming - Kafka] end subgraph Orchestration AIR[Apache Airflow - 3 DAGs] KAF[Apache Kafka 7.5.0] end subgraph Processing SPB[Spark Batch ETL] SPS[Spark Streaming] GE[Great Expectations] end subgraph Storage MIN[MinIO - S3 Storage] PG[PostgreSQL 15] MDB[MongoDB 6.0] IDB[InfluxDB 2.7] RED[Redis 7] ES[Elasticsearch 8.11] end subgraph Serving MLF[MLflow 2.9.2] API[.NET 8 API] SF[Snowflake DW] end subgraph Monitoring PROM[Prometheus] GRAF[Grafana] end BS --> AIR SS --> KAF AIR --> SPB KAF --> SPS SPB --> GE SPS --> GE GE --> MIN GE --> PG PG --> SF PG --> MLF PG --> API PROM --> GRAF

Batch Processing Flow

sequenceDiagram participant BS as MySQL Source participant AF as Airflow DAG participant GE as Great Expectations participant MN as MinIO participant SP as Spark Batch participant PG as PostgreSQL BS->>AF: Trigger Batch Job AF->>BS: Extract Data AF->>GE: Validate Quality GE-->>AF: Validation OK AF->>MN: Upload Raw Data AF->>SP: Submit Spark Job SP->>MN: Read Raw Data SP->>SP: Transform + Enrich SP->>PG: Write Processed Data SP->>MN: Archive to S3

Streaming Processing Flow

sequenceDiagram participant KP as Kafka Producer participant KT as Kafka Topic participant SS as Spark Streaming participant AD as Anomaly Detection participant PG as PostgreSQL participant MN as MinIO loop Every 2 seconds KP->>KT: Publish sensor_readings end KT->>SS: Consume Stream SS->>AD: Process Events AD->>AD: Detect Anomalies AD->>PG: Store Anomalies AD->>MN: Archive to S3

Docker Compose Stack (20 Services)

graph TB subgraph Data ZK[Zookeeper :2181] --> KAFKA[Kafka :9092] MYSQL[MySQL :3306] PG[PostgreSQL :5432] REDIS[Redis :6379] MONGO[MongoDB :27017] MINIO[MinIO :9000] INFLUX[InfluxDB :8086] KP[Kafka Producer] end subgraph Orchestration AF_WS[Airflow :8080] AF_SC[Scheduler] end subgraph Processing SM[Spark Master :8081] SW[Spark Worker] end subgraph Serving MLF[MLflow :5001] API[.NET API :5000] end subgraph Monitoring PROM[Prometheus :9090] GRAF[Grafana :3000] ES[Elasticsearch :9200] end KP --> KAFKA AF_WS --> MYSQL AF_WS --> PG SM --> SW SM --> MINIO SM --> KAFKA SM --> PG MLF --> PG API --> PG API --> KAFKA PROM --> GRAF

Stack

Technology Stack

Industry-leading open-source technologies and cloud-native tools

🔄 Data Processing

  • ⚡ Apache Spark 3.5.3 - Batch & Stream Processing
  • 📫 Apache Kafka 7.5.0 - Event Streaming
  • 🦁 Zookeeper - Kafka Coordination
  • ✨ Great Expectations - Data Quality Validation

📊 Orchestration & API

  • 🌀 Apache Airflow 2.7.3 - DAG Orchestration (3 DAGs)
  • 🔧 .NET 8 / C# - REST API (16 endpoints, Swagger)
  • 📦 Serilog + Polly - Structured Logging & Retry
  • 🎯 Dapper - Micro-ORM for DB Access

💾 Databases & Storage

  • ❄️ Snowflake - Cloud Data Warehouse (primary)
  • 🐘 PostgreSQL 15 - Analytics + Staging + Fallback DW
  • 🐬 MySQL 8.0 - Source OLTP Database
  • 🪣 MinIO - S3-Compatible Object Storage
  • 🍃 MongoDB 6.0 - NoSQL Document Store
  • 📈 InfluxDB 2.7 - Time-Series Database
  • ⚡ Redis 7 - In-Memory Cache
  • 🔍 Elasticsearch 8.11 - Search & Log Indexing

🤖 ML & Monitoring

  • 🔬 MLflow 2.9.2 - Experiment Tracking & Model Registry
  • 🔥 Prometheus 2.48 - Metrics Collection (9 targets)
  • 📈 Grafana 10.2 - Dashboards & Alerting
  • 🗺️ Apache Atlas - Data Lineage & Governance

🚀 DevOps & Deployment

  • 🐳 Docker Compose - Local Development (20 services)
  • ☸️ Kubernetes + Helm - Production Deployment
  • 🏗️ Terraform - AWS Infrastructure (VPC, EKS, RDS, S3)
  • 🚀 Argo CD - GitOps Continuous Delivery
  • ⚙️ GitHub Actions - CI/CD Pipeline

📋 Languages & Frameworks

  • 🐍 Python 3.10 - Pipeline Code & DAGs
  • 🔧 C# / .NET 8 - Backend REST API
  • 🌐 HTML5 / CSS3 / JavaScript - Wiki Frontend
  • 🗄️ SQL - Warehouse Schema & Transformations
  • 📝 HCL - Terraform Infrastructure

🧪 Testing & Quality

  • 🐍 pytest - Python Test Framework (49 tests)
  • ✨ Great Expectations - Data Quality Suites
  • 🐳 Docker Tests - Infrastructure Validation
  • 📫 Kafka Tests - Producer & Consumer Checks
  • ❄️ Snowflake Tests - Warehouse SQL Validation

📊 BI & Analytics

  • 📈 Tableau - Enterprise Dashboards
  • 🔍 Looker - Data Exploration & Modeling
  • 📊 Power BI - Microsoft Analytics
  • 🐼 Pandas - Data Export & Aggregation
  • 📋 CSV/Parquet - Interoperability Formats

Capabilities

Key Features

Production data engineering workloads with observability built in

⚡

Real-Time Processing

Process millions of events per second with Spark Structured Streaming and Kafka. Sub-second latency for critical business insights with exactly-once semantics.

📊

Batch Analytics

Scalable batch processing with Apache Spark. Handle petabyte-scale datasets with optimized partitioning, compression, and distributed computing.

🔍

Data Quality

Automated validation with Great Expectations. Define expectations, run validations, and generate data quality reports. Prevent bad data from entering your pipeline.

🗺️

Data Lineage

Track data flow from source to destination with Apache Atlas. Understand data dependencies, impact analysis, and compliance with automated lineage tracking.

📈

Observability

Comprehensive monitoring with Prometheus and Grafana. Track pipeline health, performance metrics, SLA compliance, and receive intelligent alerts.

🔒

Security & Compliance

Enterprise security with encryption at rest and in transit. RBAC, audit logging, secrets management, and compliance with GDPR, HIPAA, SOC 2.

📦

Containerized

Fully containerized with Docker and Kubernetes. Portable, scalable, and cloud-agnostic. Run on AWS, GCP, Azure, or on-premises infrastructure.

🔄

CI/CD Ready

GitOps workflow with Argo CD. Automated testing, deployment, and rollback. Blue/Green and Canary deployment strategies with progressive delivery.

🤖

ML Integration

Seamless MLOps with MLflow and Feast. Track experiments, manage models, serve predictions, and maintain feature stores for ML workflows.

📊

BI Dashboard Exports

Automated analytics exports to Tableau, Looker, and Power BI. Query Snowflake warehouse for daily orders, revenue trends, and customer segmentation with scheduled Pandas-based pipelines.

🏗️

Infrastructure as Code

Full IaC with Terraform for AWS (VPC, EKS, RDS, S3), Helm charts for Kubernetes deployment, and Argo CD for GitOps continuous delivery. Reproducible infrastructure across all environments.

🌀

DAG Orchestration

Three production Airflow DAGs: batch ingestion (MySQL → Spark → PostgreSQL), warehouse transformation (staging → Snowflake star schema), and streaming monitoring with automated alerting and retry policies.

Portfolio

About This Project

A showcase of production-style data engineering by Aryan Gahlot

Aryan Gahlot

Data Engineer · Portfolio Project

FluxPipe demonstrates end-to-end data platform design: batch and streaming ETL, star-schema warehousing, data quality gates, observability, and cloud-native deployment patterns — all orchestrated from a single Docker Compose stack.

  • Batch ingestion with Airflow, Spark, and Great Expectations
  • Real-time Kafka streaming with anomaly detection
  • Warehouse analytics API with live pipeline insights
View on GitHub

Observability

Live Pipeline Insights

Real-time metrics from the FluxPipe warehouse API — start the stack with make up

Connecting to API...

Release

Deployment Strategies

Zero-downtime patterns for production releases

Blue/Green Deployment Zero Downtime

Deploy new version alongside current version, then switch traffic instantly. Instant rollback capability by switching back to previous version.

  • Instant traffic switching between versions
  • Zero downtime during deployment
  • Immediate rollback capability
  • Full testing in production environment
  • Preview environment before promotion

Canary Deployment Progressive Rollout

Gradually shift traffic from old to new version with automated analysis. Detect issues early and rollback automatically if metrics degrade.

  • Progressive traffic shifting (10% → 25% → 50% → 100%)
  • Automated Prometheus metrics analysis
  • Auto-rollback on failure detection
  • Reduced blast radius for issues
  • Real-time performance comparison

Rolling Deployment Incremental Update

Update pods incrementally while maintaining service availability. Ideal for stateless services with minimal resource requirements.

  • Incremental pod updates
  • Configurable update speed
  • Health checks before promotion
  • Pause and resume capability
  • Minimal resource overhead

CI/CD Pipeline Flow

graph LR subgraph Development DEV[Developer] --> GIT[Git Push] end subgraph CI/CD GIT --> GHA[GitHub Actions] GHA --> TEST[Run Tests] TEST --> BUILD[Build Images] BUILD --> PUSH[Push to GHCR] end subgraph Deploy PUSH --> HELM[Helm Chart] HELM --> K8S[Kubernetes] end subgraph Infra TERRA[Terraform] --> CLOUD[AWS/GCP/Azure] CLOUD --> K8S end

Applications

Use Cases

Real-world applications across industries and domains

🛒

E-Commerce & Retail

Real-time recommendations, fraud detection, inventory optimization, customer behavior analysis, and personalized marketing campaigns.

💰

Financial Services

Risk analysis, trade surveillance, fraud detection, regulatory compliance, portfolio analytics, and real-time transaction monitoring.

🏥

Healthcare

Patient monitoring, clinical trial analysis, predictive diagnostics, IoT medical device data processing, and treatment outcome prediction.

🏭

Manufacturing & IoT

Predictive maintenance, supply chain optimization, quality control, sensor data analysis, and production efficiency monitoring.

📱

Media & Social

Sentiment analysis, ad fraud detection, content recommendations, user engagement tracking, and real-time trend detection.

🚗

Transportation & Logistics

Route optimization, fleet management, demand forecasting, delivery tracking, and real-time traffic analysis.

⚡

Energy & Utilities

Smart grid monitoring, energy consumption forecasting, renewable output optimization, outage prediction, and demand-response management at scale.

🏛️

Government & Public Sector

Census data processing, public health monitoring, infrastructure planning, emergency response analytics, and citizen services optimization.

Quickstart

Getting Started

Deploy all 20 services in minutes with a single command

1

Clone the Repository

Get the source code from GitHub and navigate to the project directory

bash 
git clone https://github.com/aryangahlot50/FluxPipe.git
cd FluxPipe
2

Configure & Start

Create environment file and launch all 20 services with Docker Compose

bash 
cp .env.example .env
make build && make up

# Or for 8GB machines:
make up-lite
3

Access the Services

Once all services are running, access the web interfaces:

4

Run Your First Pipeline

Trigger the batch ingestion DAG to see the full pipeline in action

bash 
# Trigger batch pipeline: MySQL → Validation → MinIO → Spark → PostgreSQL
make trigger-batch

# Trigger warehouse ETL: Staging → Snowflake dimensions → facts → aggregations
make trigger-warehouse

# View all DAGs
make list-dags
5

Run Streaming & Spark Jobs

The kafka-producer service auto-generates sensor data. Run Spark jobs for processing

bash 
# Kafka producer is already running (auto-started)
make kafka-topics

# Run Spark batch ETL
make spark-batch

# Run Spark streaming (continuous)
make spark-stream
6

Deploy to Production (Optional)

Deploy to any cloud or on-prem Kubernetes cluster via Helm chart

bash 
# Deploy to any Kubernetes cluster
make deploy-k8s

# Or deploy to specific providers:
make deploy-aws      # AWS EKS (Terraform + Helm)
make deploy-gcp      # GCP GKE
make deploy-azure    # Azure AKS
make deploy-onprem   # On-prem (k3s, kubeadm)

# Check status / teardown
make deploy-status
make deploy-teardown

Additional Resources

📖 Comprehensive README Detailed documentation 🏗️ Architecture Guide System design details 🚀 Quick Start Guide Fast deployment commands 📋 Deployment Strategies Blue/Green & Canary deployments 📓 Jupyter Notebook Interactive tutorial

Start building with FluxPipe

Clone the repo, run make up, and explore batch ingestion, streaming analytics, and live pipeline insights.

Get Started View on GitHub
↑