Sagar KonkanagaonSenior Data Engineer

The pipeline, right now

A real PySpark and Delta pipeline samples Wikipedia's public edit stream every hour on GitHub Actions. These numbers come from its latest run report.

Status

passed

Run

2026-11-11 19:26 UTC

Rows landed

4,405

Rows to silver

4,405

Quarantined

0

DQ checks passed

8 of 8

Run time

156 s

Pipeline codeAll run reportsLatest CI log

Experience

1. Senior Data Engineer · eBay

   Jan 2026 – present · Bengaluru
   • Migrated 300+ production pipelines from UC4 (Automic) to Data Lake Studio, rebuilding them on the new orchestration layer.
   • Ran a two-year historical backfill across high-volume pipelines with full completeness checks.
   • On-call via PagerDuty: diagnosing Spark and Hive failures before they reach downstream analytics.

2. Senior Data Engineer (client engagement) · Trane Technologies (via RedBlackTree)

   Aug 2025 – Jan 2026 · Bengaluru
   • Built Hadoop–Spark pipelines for telemetry from 10,000+ connected HVAC devices.
   • Implemented schema evolution so firmware changes never forced a pipeline redeploy.
   • Set up validation, lineage and metadata governance for compliance and audit.

3. Data Engineer · Tiger Analytics

   Nov 2024 – Aug 2025 · Bengaluru
   • Cut Spark pipeline runtime by 35% with partition pruning, caching, broadcast joins and AQE.
   • Built a Bronze–Silver–Gold lakehouse on Azure Databricks and Delta Lake with SCD Type 1 & 2.
   • Replaced full loads with watermark-based incremental ingestion on ADF and Cloud Composer.

4. Associate Data Engineer · XL Dynamics

   Oct 2020 – Aug 2024 · Mumbai
   • Built ETL over 20M+ mortgage, loan and risk records with full lineage and audit controls.
   • Designed DQ rule frameworks and reconciliation checks across every pipeline stage.
   • Automated ingestion, scheduling and alerting with shell scripts, cutting manual ops work by 40%.

Projects

• case studySchema evolution for 10,000+ IoT devicesSchema evolution for telemetry from 10,000+ connected IoT devices.
• case studyA faster medallion lakehouseCutting Spark runtimes by 35%, and building a medallion lakehouse on Databricks and Delta Lake.
• case studyMoving 300+ pipelines to a new orchestratorMigrating 300+ production pipelines from UC4 (Automic) to the in-house DLS platform, plus a two-year backfill.
• case studyValidating and reconciling 20M+ recordsValidation and reconciliation over 20M+ records.
• open sourceNumber Passport: prove where a reported number comes fromA regulator points at one number in a capital report and asks where it comes from. Answering means days of reading SQL, dbt and Spark code and running ad-hoc queries.
• open sourcedeja: memory for AI coding agentsAI coding agents forget everything when a session ends, so they rediscover the same missing env var, flaky import and broken command every week, and you pay for the same tokens each time.
• labExactly once, from an at-least-once streamEvent streams redeliver: retries, replays and mobile clients that come back online send the same event twice and an hour late, and every duplicate inflates a dashboard someone trusts.
• labSCD Type 2 from a CDC feedA customer moves city and the old address is overwritten, so last quarter's sales by region quietly change every time someone re-runs the report.
• labTaming a skewed join with saltingOne customer owns most of the rows, so one Spark task gets most of the join, and a job that should take minutes runs for an hour on a single straggler.
• labA backfill you can run twiceA two-year backfill fails on day 412, someone restarts it from the top, and now half the history is loaded twice.

Contact

I'm looking for my next data engineering role. Email me, or use the form on the home page.

sagarkonkanagaon@gmail.comOpen the contact form