Operational data into the warehouse on an hourly cadence, zero custom code
MongoDB was the operational store for product data. The business needed that data available for reporting and analysis without building custom extraction code. Fivetran handled the hourly sync via its native MongoDB connector — CDC-based, low latency, no pipeline to maintain. Data landed in Snowflake, transformed through dbt models, orchestrated by Airflow, and surfaced in Power BI. The value of Fivetran here: the team spent zero time on extraction and 100% on the transformation logic that actually mattered.
Sales data in the warehouse hourly so it could be joined with everything else
Salesforce held the CRM data and the business needed to combine it with product usage and financial data that lived elsewhere. Fivetran's Salesforce connector ran hourly, keeping the warehouse fresh without hitting API rate limits manually. Once in Snowflake, dbt models joined CRM data across sources, giving stakeholders a unified view no single system could produce on its own.
Legacy ERP data pulled through Glue — Fivetran would have cost more than the data was worth
A legacy ERP system (Oracle) was the system of record for finance and supply chain data. Fivetran was considered and ruled out on cost — the connector pricing didn't justify the table count. Built the extraction layer in AWS Glue instead: JDBC connection to Oracle, PySpark-based extraction with a full refresh on every run, and direct write to Snowflake. Airflow scheduled and monitored the job. dbt handled the transformation layer the same way it did for every other source.
ERP data in daily CSV dumps — built a custom framework because no connector existed for this pattern
Some ERP data wasn't accessible via a direct database connection — it was only available as daily CSV exports dropped into S3. No API, no connector, no schema contract. Built a custom ingestion framework specifically for this delivery pattern: detect new files, validate schema, handle encoding issues and malformed rows, and load into Snowflake. The framework was generic enough to onboard additional flat-file sources by config rather than writing new code each time. Once in Snowflake, the same dbt → Airflow stack handled transformation and scheduling.
REST APIs pulled directly into Snowflake via stored procedures — no ETL tool in the middle
Some sources exposed only REST APIs — no database connector, no file drop, no Fivetran support. Instead of introducing another ETL tool for a small number of endpoints, used Snowflake stored procedures to call the APIs directly and land the response data into staging tables. The extraction logic lived inside Snowflake itself, keeping the stack lean. From there, the standard dbt → Airflow path handled transformation and scheduling, same as every other source on the platform.
Sigma on top of dbt models — analysts in SQL, not waiting on engineers
The analytics team needed to explore and slice data beyond the fixed dashboards in Power BI. Sigma was chosen because it queries Snowflake directly — no data extract, no cube, no BI layer with its own copy of the data. Analysts wrote SQL-like logic in a spreadsheet-style interface directly on top of the dbt models, which already had the joins, business logic, and governance baked in. Engineers defined the models; analysts owned the analysis. No ticket queue for every new cut of data.
Enriched warehouse data pushed back into Salesforce so the sales team didn't have to leave their tool
After Salesforce data was ingested, joined with product and finance data in Snowflake, and transformed through dbt, the enriched output needed to go back to Salesforce — because the sales team lived in Salesforce, not in BI dashboards. Airflow orchestrated the sync trigger; AWS AppFlow handled the actual transfer: Snowflake as source, Salesforce as destination, field-level mapping defined in config. Led the full production implementation: integration architecture, authentication strategy, and cross-team coordination across Salesforce, DevOps, QA, and Data Engineering.
dbt failures page on-call immediately — no one discovers a broken model from a dashboard
When a dbt model or Airflow DAG fails in production, the team needs to know immediately — not when a stakeholder notices a blank dashboard. Wired Airflow's on-failure callbacks to trigger PagerDuty alerts with full context: which DAG, which task, which run, the error message. On-call engineers got paged directly with enough information to triage without logging into Airflow first. Reduced mean time to awareness from "whenever someone notices" to under a minute, and made data SLAs something the team could actually commit to.
Quality checks run nightly inside Snowflake — anomalies flagged before they reach the transformation layer
Data quality checks ran as Snowflake stored procedures — null checks, row count validations, referential integrity, freshness assertions — executed directly inside Snowflake where the data already lived. Airflow ran the checks every night. When a check flagged an anomaly, PagerDuty alerted on-call so the team knew about it before stakeholders noticed anything wrong in their dashboards. The stored procedure approach kept the logic close to the data and queryable — engineers could debug a failed check with a plain SQL query.
A Redshift cost spike, offloaded to Spark, then opened up across teams
Redshift transform costs were spiking under thousands of jobs at 45–50 PB scale, orchestrated by an in-house-built tool. The solution: unload data from Redshift to CSV in S3, read it in Spark on EMR using Spark SQL for transformations, and write Parquet back into S3 as the data lake — then query it through Redshift Spectrum instead of reloading into Redshift. Amazon had built their own orchestration layer because nothing off the shelf handled the operational complexity at that scale; the in-house tool gave fine-grained control over job dependency graphs, retries, and resource allocation across thousands of concurrent pipelines.
50 tables unloaded from the data warehouse as CSV — landed into a lake for ML
Data science needed warehouse data in a lake to train ML models. Built an ingestion framework to unload ~50 tables from Redshift as CSV files into S3, then process them with Glue and PySpark — converting to Parquet, applying partitioning, and landing in the data lake. Airflow orchestrated the full pipeline. The framework was generic enough to onboard new tables by config rather than new code each time.
45-day compliance SLA on customer data deletion — automated from identification to rewrite
CCPA required deleting customer data from the data lake within 45 days of a request. Evaluated Delta Lake and Apache Hudi — both had early stability issues (2019–2020). Built a custom deletion path on Athena CTAS: identify the affected partitions, rewrite only those partitions with the target records removed, and swap in the cleaned output. Airflow orchestrated the full workflow. The approach kept the lake on standard Parquet with no new format dependencies, and took the deletion SLA from weeks to near real-time.
Player telemetry pipeline — XML off devices, processed in Spark, visualized in Interana
Device telemetry (XML) landed continuously through Kinesis into S3, processed in batch on Spark on EMR, and landed in S3 for the product team to analyze in Interana. A scheduled batch pipeline, not a streaming compute job — Kinesis handled the ingestion side, Spark handled the processing on a regular schedule.
When Lambda hit its limits, the pipeline moved to containers
Lambda's memory and time limits made it the wrong fit for the Python logic driving the live usage dashboard. Containerized the compute, deployed it on ECS, and kept the upstream Kinesis stream unchanged — the only thing that moved was where the Python ran. The dashboard stayed real-time without re-architecting the rest of the pipeline.
Speaker data + mobile data combined into a listening stats dashboard in the Sonos app
Speaker-side player data landed in S3 from the devices; controller-side usage data landed in S3 from the mobile app. Spark joined both sources into a single user-level view, aggregated into Aurora, and served to the Sonos mobile app via a stored procedure. The dashboard showed each user their listening stats: where they listened, how long, at what volume, lifetime hours, and monthly hours — two independent device sources reconciled into one in-app feature.
The framework that moved me from QA into data engineering
Built a custom validation framework that read the same data from two systems—via API calls or JDBC/ODBC—and compared expected vs. actual using SQL intersects and excepts, catching byte-level differences that no other method surfaced. The first version ran on Jenkins: it spun up EC2 instances, installed PySpark, pulled the codebase, and ran Spark jobs against source data in S3 daily. Running it every day flagged data drift and code bugs before they reached production. The framework ran for 5–6 years across multiple orgs — it's the reason I started bringing hard edge-case questions into design discussions instead of finding them at test time, and the direct origin of the risk-first instinct behind every audit I run today.
Simulating a realistic backend to load-test a protocol, not just an endpoint
A WebSocket connection needed performance testing at the protocol level—the three-way handshake itself, then the behavior of the calls that followed once the connection was persistent. Built Nginx-based simulations of the backend API calls so the connection could be load-tested against realistic responses without depending on the real servers being available or in the right state.
Log data queried in Splunk — system behavior visible without grepping raw files
Application and system logs were ingested into Splunk and queried there directly. Built dashboards on top so the team could monitor system behavior, trace issues, and answer operational questions without touching raw log files. Splunk was the right tool for this: ad-hoc querying across high-volume log data with fast turnaround.
Fragmented small datasets moved into the lake — NiFi where a full Spark pipeline was overkill
Several smaller datasets sat in isolated S3 locations and weren't making it into the shared data lake. Data volumes were low — not worth the overhead of a full Glue or Spark pipeline for each one. Used Apache NiFi to handle the movement and light transformation: S3 silo → NiFi → S3 data lake. Right-sized tooling for the problem.
Fixed the root cause, not the symptom — one change, all issues gone
An Athena CTAS app was writing partitioned Parquet, and a Spark app was reading it back. Athena's crawler is case-insensitive; Spark is case-sensitive. The result: schema mismatches, inconsistent column casing, and schema evolution edge cases (different column counts across partitions) that kept accumulating as patches. Instead of handling each case downstream, traced it to the write path — the column names weren't being normalized before writing. Fixed by uppercasing all column names in the Athena CTAS step. Every downstream issue disappeared at once.
Data stayed in place. Access moved instead.
Each team owned a Redshift and shared data by dumping it upstream or downstream — siloed, fragile, high-latency. Instead of moving the data, put a Glue Data Catalog on top of each source and pointed Redshift Spectrum at it. Teams could query each other's data in place, no copies, no hand-off pipelines. The data stayed where it lived; only the access layer changed.
Always-on notebooks, shut down automatically at 6 PM
Glue Notebooks were always running, even when no one was using them. Built a custom framework that spun a notebook up when a user needed one and tore it down automatically at 6 PM every day — going from always-on infrastructure to time-bounded compute. A 70% cost reduction without changing how the team worked.
Pushed back on read-then-cast — schema errors should fail at read, not break Spark mid-job
The team's approach to reading CSV files in Spark was to infer or ignore schema at read time, then cast columns to the correct types inside the transformation logic. The problem: schema mismatches surfaced late, deep inside the job, where they were expensive to diagnose. Pushed back on this pattern and made the case for passing an explicit schema at read time using Spark's built-in schema option. Schema errors then fail immediately at read, before any processing happens, with a clear error message pointing to the source. Bad data caught at the gate rather than discovered when a job crashes halfway through.
Warehouse spend, made visible before it became a problem
Built cost and utilization dashboards to surface expensive queries and unexpected warehouse usage. Following an incident where a runaway ad-hoc query kept a warehouse running 24+ hours unnoticed over a holiday weekend, added execution timeouts, workload-based auto-suspend tuning, and warehouse categorization by workload type — ETL warehouses suspend aggressively, BI warehouses stay warm for the local disk cache.
Read the full story on the blog →Rethought the approach before starting — saved days of compute and engineering time
A 180-day historical backfill was scoped as a day-by-day loop: run the pipeline once per day, 180 times. Before starting, stepped back and asked whether the job actually needed to be structured that way. It didn't — the logic was date-range-capable. Rewrote it to process configurable multi-day windows in a single run, then batched the backfill into a handful of jobs instead of 180. The compute cost dropped significantly, the engineering time to babysit it shrank, and the data science team got their historical data days sooner. Also used the window to cross-train the data science team on the pipeline so they could run backfills themselves going forward.
More compute is the last resort, not the first response
From 100+ parallel EMR clusters hitting out-of-memory errors at Sonos, to understanding shuffle partitions and the catalyst optimizer at Spark Summit 2019, to a single mis-partitioned file crashing a Glue job at Meredith regardless of DPU count — repartitioning fixed what more compute couldn't. The same lesson resurfaced at 45–50 PB scale at Amazon: filter before joins, push predicates early.
Read the full story on the blog →