I've spent the better part of a decade building data lakes. Not consulting on them — building them, from raw infrastructure to the pipelines that run on top. Three companies, very different scales. The lessons don't compound linearly.

Chapter one: Sonos

Sonos was where I first built a data lake at scale. We were handling device telemetry — speaker events, playback data, network diagnostics — flowing in through Kinesis, landing in S3, processed by EMR. At 10 TB, the architecture felt clean. Kinesis handled the ingest burst. S3 absorbed everything without complaint. EMR gave us the compute flexibility we needed.

The lesson Sonos taught me: serverless has a ceiling. We hit it when a batch job that took 40 minutes started taking 6 hours. The job hadn't changed. The data volume had crossed a threshold where our Lambda-based orchestration couldn't keep up. What worked at one scale silently broke at another — no alarm, no obvious failure, just degraded performance that crept up until it was undeniable.

The most dangerous kind of failure in data engineering is the one that looks like slowness before it looks like a failure.

We moved the orchestration to Airflow. Problem solved — but the lesson stuck: architecture decisions have a scale limit baked in. Know what yours is before you hit it.

Chapter two: Meredith

At Meredith (now Dotdash Meredith), the stakes changed. We weren't just building infrastructure — I was the owner. When something broke at 2am, it was my problem. That changes how you design systems.

Meredith is where I learned about compliance as engineering. CCPA came into force while I was there, and suddenly "delete this user's data" became a hard technical requirement on a data lake that had been designed with immutability as a feature. We solved it with Athena CTAS operations — read the partition, filter out the user, write it back, delete the original. It worked, but it was painful, and it would have been half as painful if we'd designed for it from the start.

The ownership model also taught me something about small-scale versus large-scale engineering: small-scale ownership teaches more. When I was one of three engineers responsible for the full stack — ingest, storage, transform, serve — I understood every layer. At Amazon, I had peers who were experts in exactly one component and had no visibility into adjacent systems. Both have value, but the depth you build owning the whole thing is hard to replicate any other way.

Chapter three: Amazon

50 PB. 1,000+ jobs. Multiple teams. A data lake that had been growing for years before I arrived.

At that scale, the technical problems are almost secondary to the organisational ones. The hardest problems I faced at Amazon weren't about storage formats or query optimisation — they were about who owns what, how changes propagate across dependent systems, and what happens when a team upstream changes something without telling anyone downstream.

The answer is data contracts. Not informal agreements — formal, versioned contracts that define what a producer guarantees to its consumers: schema, null rates, freshness SLAs, cardinality expectations. At Amazon's scale, without contracts, changes ripple unpredictably. With them, change becomes manageable.

The other Amazon lesson: observability first, always. At 50 PB with 1,000+ jobs, you cannot monitor by watching. You need automated anomaly detection, data quality checks at every layer, and alerting that fires before a downstream team notices a problem. The teams that operated reliably were the ones who'd invested in observability before the scale made it painful. The teams that hadn't were permanently in reactive mode.

Five lessons that hold across all three

Supreeth M Gowda has spent 12+ years across data engineering, software engineering, performance engineering, and data science. He now consults through Encore — learn more about him and his services.