Skip to content
The Data Foundry machineRaw sources flow into a Bronze chamber, through a Spark core of executor rings, into a Silver chamber with a quarantine pod, and settle as a Gold crystal. An orchestrator DAG floats above; a sentinel orb watches.WIKIDATAWIKICOMMONSWIKIENWIKIZHWIKISOURCEBRONZESPARK CORESILVERQUARANTINEGOLDextractbronzeschemasilvergoldqualityretentionDAGSENTINEL

Sagar KonkanagaonSenior Data Engineer

I build data pipelines that check their own data and are safe to run twice.

  • eBay300+ pipelines migrated from UC4 (Automic) to the in-house DLS platform, plus a 2-year backfill
  • Tiger AnalyticsSpark runtimes cut by 35%, and a medallion lakehouse on Databricks and Delta Lake
  • RedBlackTree · client TraneSchema evolution for telemetry from 10,000+ IoT devices
  • XL DynamicsValidation and reconciliation over 20M+ records
Résumé (PDF)GitHubLinkedInContact
Open to new roles
● Last run passed2026-11-14 17:13 UTC · 4 h ago✓ replayed
Run historylast 11 · 11 passed
pipeline code ↗all reports ↗
Rows landed
4,708
Rows to silver
4,708
Quarantined
0rows
Inject chaossimulationpick a failure

§01 · How to read the machine

Pipeline work is invisible. When it goes right, nobody notices; when it goes wrong, someone gets paged at 3am. This site makes the invisible part visible.

The machine above runs on a real pipeline. Every hour, GitHub Actions samples the Wikimedia recent-changes stream and runs it through Bronze, Silver and Gold Delta tables with PySpark, checks the data against its contracts, and publishes a run report. The machine replays the latest run: its stages, its row counts, the rows it quarantined. When a real run had rejects, a schema change or a failed check, the machine replays that incident.

The run history shows the last 48 runs; open any cell for its full report. Inject chaos is a simulation, labelled as one, and each drill links to the code in /pipeline that handles that failure for real. Click any chamber to see the project behind it.

↓ Scroll to fly through it · or click any chamber

01 · Intake

Intake ports

One real source: every edit on every Wikipedia, sampled each hour.

The machine runs on one real source: Wikimedia's public recent-changes stream, every edit and page creation on every Wikipedia and its sister wikis. Each hour the pipeline in /pipeline samples it for two minutes. The four streams here are the four busiest wikis in the latest run.

The intake's only job is to accept everything and lose nothing: the lines are written untouched to a landing file, so any run can be replayed exactly. Judgement happens later.

Built it · CASE 03Moving 300+ pipelines to a new orchestrator→
02 · Bronze

Bronze chamber

Frosted glass. Records land exactly as they arrived, malformed ones included.

Bronze is the append-only landing zone. Records are written to Delta tables as they arrived, with ingestion metadata attached (_ingested_at, _source_file, _batch_id). Nothing is fixed here, because the raw copy is what lets you replay history when a downstream bug is found.

Malformed records (the red flickers) are kept, not dropped. Throwing data away at the door is how you lose the evidence you need later.

At Trane this was raw HVAC telemetry from 10,000+ devices, landed as Parquet in HDFS before any transformation touched it.

Built with: ADF, ADLS Gen2, HDFS, Parquet / ORC, Delta Lake.

Built it · CASE 01Schema evolution for 10,000+ IoT devices→
03 · Spark core

Spark core

Rings of executors. Records split into partitions, shuffle, and rejoin.

The core is the compute: a Spark cluster on Databricks. Each ring is a set of executors, each node a task working on one partition. When a job joins or aggregates, records shuffle across the rings (the arcs) so that matching keys meet on the same executor, then the results rejoin.

Most of the performance work I do lives here: choosing partition keys, killing accidental shuffles, salting skewed joins, broadcasting small dimensions, and sizing clusters so the bill matches the value of the job.

At Tiger Analytics this work cut distributed pipeline runtimes by 35% through partition pruning, caching, broadcast joins and Adaptive Query Execution.

Built with: PySpark, Spark SQL, Azure Databricks, AQE, Hive.

Built it · CASE 02A faster medallion lakehouse→
04 · Silver

Silver chamber

Clear glass. Records align into a conformed lattice; bad ones go to quarantine.

Silver is where records become trustworthy. Types are enforced, duplicates removed, keys conformed across sources, and late or corrected records merged in with MERGE so history stays consistent.

Every record passes a set of data quality expectations (not-null keys, valid ranges, referential checks). Records that fail are not silently dropped. They are routed to the quarantine pod with the reason attached, so they can be fixed and replayed.

At XL Dynamics I built DQ rule frameworks and reconciliation checks over 20M+ mortgage and loan records; at Trane, schema evolution that absorbed firmware changes without a redeploy.

Built with: Delta MERGE, DQ rule frameworks, reconciliation, SCD Type 1 & 2, lineage and metadata governance (Collibra).

Built it · CASE 04Validating and reconciling 20M+ records→
05 · Gold

Gold chamber

Crystal clear. Business-ready tables, modelled for the people who use them.

Gold is what the business actually sees: aggregated, modelled tables shaped around questions, not sources: equipment health by site, loan performance and risk KPIs, supply-chain throughput.

The crystal rotates slowly because Gold should be boring: stable schemas, documented columns, predictable freshness, and a cost per query someone has actually looked at.

Built with: Delta Lake on Databricks, BigQuery, dimensional modelling with surrogate keys, Tableau.

Built it · CASE 02A faster medallion lakehouse→
06 · Orchestrator

Orchestrator

A holographic DAG above the machine. Each node lights as its task runs.

The DAG above replays the latest real run: GitHub Actions runs the pipeline in /pipeline every hour, one run at a time, and each node lights for as long as that stage really took. A failed stage stays red, and the stages after it never run.

Nothing in a pipeline runs on its own. The orchestrator decides what runs, in what order, and what happens when something fails: retries with backoff, sensors that wait for late files, SLAs that page a human only when it matters.

At eBay I migrated 300+ production pipelines from UC4 (Automic) to Data Lake Studio, and I carry the PagerDuty pager when they misbehave. Before that: Airflow DAGs on Cloud Composer and ADF triggers at Tiger Analytics, shipped through Jenkins.

Built with: UC4 / Automic, DLS, Airflow, Cloud Composer, ADF, Jenkins, PagerDuty.

Built it · CASE 03Moving 300+ pipelines to a new orchestrator→
07 · Sentinel

AI Sentinel

A drifting orb that scans each chamber, diagnoses anomalies and repairs the flow.

The sentinel is the idea I care most about: pipelines that notice their own failures and propose the fix. It reads run metadata, row counts, schema history and DQ results, spots what changed, and either applies a safe, pre-approved remedy or hands a human a diagnosis instead of a stack trace.

It is also what on-call should feel like. At eBay I carry the PagerDuty pager for pipelines I migrated, and the playbook is always the same: detect, diagnose from the run's own evidence, apply the smallest safe fix, then make sure it cannot page twice.

Built it · CASE 03Moving 300+ pipelines to a new orchestrator→
§02/Chaos playbook

Chaos playbook

What the sentinel does when something breaks. The five failures every pipeline eventually meets. Press INJECT CHAOS above to watch each one play out.

Schema drift

warning · bronze
▲ an event arrives with a field the contract doesn't know
◆ registry: new field → recorded once in ops/schema_changes
✓ kept in silver.extras · run continues · no code change

Wikimedia's event schema allows extra fields, and they do appear. Each run compares the fields it sees with a registry seeded from the upstream schema. A new field, or a known field with a new type, is recorded once and the run keeps going.

New top-level fields land in silver's extras column, so nothing is lost and no code changes. A known field whose new type no longer casts goes to quarantine instead.

Late data

warning · silver
▲ events for an earlier hour arrive in this run
◆ the run rebuilds every hour partition its events touch
✓ late hour rebuilt with replaceWhere · nothing counted twice

Events carry the time the change happened, and the pipeline partitions by that hour, not by when the event arrived. A late event reopens its own hour.

Silver and gold are rebuilt for each hour a run touches, from every bronze row in that hour, and written with Delta replaceWhere on exactly those partitions. Late rows are merged in, duplicates are dropped on event_id, and re-running gives the same tables.

Null storm

error · silver
▲ a required field arrives null on many rows
◆ rows fail the silver contract with reason null:<column>
✓ rows quarantined · above 25% the run fails before writing

Every silver column is read and validated from the contract. A row missing a required value, or carrying one that won't cast, goes to silver.quarantine with the reason attached, never silently dropped.

If more than a quarter of a run's rows are rejected, the contract fails the run before silver is written, so a broken source can't overwrite good data.

Skewed partition

warning · spark
▲ one key holds most rows · one task does most of the work
◆ hot key found from a quick groupBy count
✓ salt the hot key across N buckets · balanced join

The hourly pipeline doesn't need this: a few thousand rows an hour can't skew a join. The fix is shown where it can be tested instead, in a separate lab with its own tests.

When one key owns most of the rows, salting adds a random bucket to the large side and replicates the matching rows on the small side, so the hot key spreads across tasks.

Source outage

error · intake
▲ the stream can't be reached, or sends nothing
◆ 3 attempts with backoff, then SourceUnavailable
✓ run marked failed · report still published · next hour retries

The source is outside the pipeline's control. The sampler retries three times with backoff, keeps whatever arrived if the stream drops mid-sample, and gives up cleanly if nothing came at all.

A failed run is still a run: its report is published with the stage and the error, and it shows up red in the run history. The next hourly run tries again.

The foundry at work
§03/Modules

Modules

Docked capsules. Four case studies from production, two open-source tools I built and released, and four mini projects on the problems every pipeline eventually meets.

  1. CASE 01Case study · RedBlackTree for Trane

    Schema evolution for 10,000+ IoT devices

    Schema evolution for telemetry from 10,000+ connected IoT devices.

    1. 10,000+ IoT devices
    2. Telemetry
    3. Schema evolution
    4. Analytics
    • Spark
    10,000+
    IoT devices
  2. CASE 02Case study · Tiger Analytics

    A faster medallion lakehouse

    Cutting Spark runtimes by 35%, and building a medallion lakehouse on Databricks and Delta Lake.

    1. Bronze (Delta)
    2. Silver (Delta)
    3. Gold (Delta)
    4. Databricks
    • Spark
    • Databricks
    • Delta Lake
    −35%
    Spark runtime
  3. CASE 03Case study · eBay

    Moving 300+ pipelines to a new orchestrator

    Migrating 300+ production pipelines from UC4 (Automic) to the in-house DLS platform, plus a two-year backfill.

    1. UC4 (Automic)
    2. Migration
    3. DLS platform
    4. 2-year backfill
    • UC4 / Automic
    • DLS
    300+
    pipelines migrated from UC4 to DLS
  4. CASE 04Case study · XL Dynamics

    Validating and reconciling 20M+ records

    Validation and reconciliation over 20M+ records.

    1. Source records
    2. Validation
    3. Reconciliation
    4. Trusted output
      20M+
      records validated and reconciled
    • OSS 01Open source · v0.1.0 on PyPI

      Number Passport: prove where a reported number comes from

      A regulator points at one number in a capital report and asks where it comes from. Answering means days of reading SQL, dbt and Spark code and running ad-hoc queries.

      1. Report cell
      2. Gold
      3. Silver
      4. Bronze
      5. Source rows
      6. Sealed evidence pack
      • Python
      • sqlglot
      • DuckDB
      • PostgreSQL
      • PySpark
      • dbt
      • Airflow
      • MCP
      • GitHub Actions
      to the cent
      every hop reconciled
    • OSS 02Open source · v0.4.0

      deja: memory for AI coding agents

      AI coding agents forget everything when a session ends, so they rediscover the same missing env var, flaky import and broken command every week, and you pay for the same tokens each time.

      1. Command fails
      2. Fix recorded
      3. Pinned to file hashes
      4. Recalled at the next failure
      • TypeScript
      • Node.js
      • SQLite FTS5
      • MCP
      • Claude Code hooks
      • Codex
      0
      dependencies · extra model calls
    • LAB 01Mini project

      Exactly once, from an at-least-once stream

      Event streams redeliver: retries, replays and mobile clients that come back online send the same event twice and an hour late, and every duplicate inflates a dashboard someone trusts.

      1. Kafka-style event stream
      2. Event-time watermark
      3. dropDuplicates within watermark
      4. Late-row side table
      5. Clean fact table
      • PySpark Structured Streaming
      • Delta Lake
      • Watermarks
      • Python
      0
      duplicates after replaying the same hour twice
    • LAB 02Mini project

      SCD Type 2 from a CDC feed

      A customer moves city and the old address is overwritten, so last quarter's sales by region quietly change every time someone re-runs the report.

      1. CDC change feed
      2. Hash of tracked columns
      3. Delta MERGE
      4. Expire old version
      5. Insert new version
      • PySpark
      • Delta Lake MERGE
      • Surrogate keys
      • SQL
      1
      current row per key, full history kept
    • LAB 03Mini project

      Taming a skewed join with salting

      One customer owns most of the rows, so one Spark task gets most of the join, and a job that should take minutes runs for an hour on a single straggler.

      1. Skewed fact table
      2. Detect hot keys
      3. Salt hot keys (N buckets)
      4. Explode dimension × N
      5. Balanced join
      • PySpark
      • Spark UI
      • AQE skew join
      • Broadcast joins
      1 → N
      tasks sharing the hot key
    • LAB 04Mini project

      A backfill you can run twice

      A two-year backfill fails on day 412, someone restarts it from the top, and now half the history is loaded twice.

      1. Date range planner
      2. One partition per run
      3. replaceWhere overwrite
      4. Row-count audit
      5. Resume from checkpoint
      • PySpark
      • Delta Lake replaceWhere
      • Airflow
      • Python
      1×
      rows per day, however many reruns
    §04/Instrument panel

    Instrument panel

    Every tool on the panel has been run in production. Needle = depth of use (1 to 5), readout = years.

    Compute

    05y
    PySpark
    05y
    Spark SQL
    05y
    Hadoop / Hive
    01y
    Databricks

    Storage

    05y
    HDFS
    01y
    Delta Lake
    05y
    ADLS Gen2
    01y
    BigQuery

    Orchestration

    05y
    Azure Data Factory
    01y
    UC4 / Automic
    01y
    Airflow / Composer

    Cloud

    05y
    Azure
    01y
    GCP
    05y
    On-prem Hadoop

    CI/CD

    05y
    Git
    02y
    Jenkins
    01y
    PagerDuty

    AI

    01y
    AI pipeline docs
    01y
    LLM APIs
    01y
    Self-healing pipelines
    §05/Flight log

    Flight log

    One entry per mission. Most recent first.

    $ tail -f /var/log/foundry/flight.log
    1. →[RUNNING]·Bengaluru

      eBay :: Senior Data Engineer

      domain=E-commerce
      • INFO│Migrated 300+ production pipelines from UC4 (Automic) to Data Lake Studio, rebuilding them on the new orchestration layer.
      • INFO│Ran a two-year historical backfill across high-volume pipelines with full completeness checks.
      • INFO│On-call via PagerDuty: diagnosing Spark and Hive failures before they reach downstream analytics.
    2. →·Bengaluru

      Trane Technologies :: Senior Data Engineer (client engagement)

      domain=Industrial IoT · via=RedBlackTree
      • INFO│Built Hadoop–Spark pipelines for telemetry from 10,000+ connected HVAC devices.
      • INFO│Implemented schema evolution so firmware changes never forced a pipeline redeploy.
      • INFO│Set up validation, lineage and metadata governance for compliance and audit.
    3. →·Bengaluru

      Tiger Analytics :: Data Engineer

      domain=Automotive
      • INFO│Cut Spark pipeline runtime by 35% with partition pruning, caching, broadcast joins and AQE.
      • INFO│Built a Bronze–Silver–Gold lakehouse on Azure Databricks and Delta Lake with SCD Type 1 & 2.
      • INFO│Replaced full loads with watermark-based incremental ingestion on ADF and Cloud Composer.
    4. →·Mumbai

      XL Dynamics :: Associate Data Engineer

      domain=Finance
      • INFO│Built ETL over 20M+ mortgage, loan and risk records with full lineage and audit controls.
      • INFO│Designed DQ rule frameworks and reconciliation checks across every pipeline stage.
      • INFO│Automated ingestion, scheduling and alerting with shell scripts, cutting manual ops work by 40%.
    §06/Ask the engine

    Ask the engine

    Ask about my work or the pipeline on this site. It answers only from what's written here and from the pipeline's run reports, shows its sources, and says so when it doesn't know.

    engine://sagaronline · answers from this site

    engine › Engine online. I answer from this site and the pipeline's run reports, with sources. If they don't say, I'll tell you.

    Answers are written by an AI model from this site and the pipeline's run reports, and cite them. A model can still misread a source, so for anything important, email Sagar.

    §07/Transmit

    Transmit

    I'm actively looking for my next data engineering role. Open a channel: messages go straight to my inbox, and I reply within two working days.

    > channel open · awaiting signal

    Or write directly: sagarkonkanagaon@gmail.com