I didn't plan to end up in big data. Very few people who work in data today did — the field barely existed as a named discipline when most of us started. I came through software testing, then performance engineering, and arrived at data engineering sideways, the way a lot of the best practitioners do.

Here's how it actually went, and what I took from each stop along the way.

CGI and the foundation

My first real role was in software testing at CGI, a large IT services firm. This is where I learned the discipline that I still think is underrated in data engineering: the ability to approach a system with constructive suspicion. Where does this break? Who else touches this? What happens when something unexpected shows up?

Testing is not a glamorous background for a data career. But the engineers I know who think most clearly about data quality, pipeline reliability, and edge cases — a disproportionate number of them came through testing. There's something about spending years finding failure modes that rewires how you think about building systems.

WPI and the pivot

I went back to school — a Master's at Worcester Polytechnic Institute — with the intention of moving from testing into something closer to the data infrastructure I was starting to work adjacent to. WPI is where I first encountered big data as a formal discipline. Hadoop was still the dominant paradigm. Spark was the exciting new thing. The MapReduce mental model was what the textbooks taught.

The academic lens helped me understand the "why" behind tools I'd been using intuitively. It also gave me a framework for thinking about distributed systems that proved more durable than the specific technologies themselves — most of which have since been replaced or significantly evolved.

Sonos: where the real work started

Sonos was my first role that was unambiguously data engineering. We were building the telemetry infrastructure for a consumer electronics company — ingesting device events, processing them, making them available for analytics and product decisions. The scale wasn't enormous, but the requirements were real: low latency, high reliability, data that product teams could actually trust.

At Sonos, I first encountered the gap between "big data" as a concept and big data as an operational reality. The exciting architecture decisions were maybe 10% of the work. The other 90% was monitoring, debugging, handling upstream schema changes nobody told us about, and making sure the numbers analysts saw were actually correct.

The exciting architecture decisions are 10% of the work. The rest is making sure the numbers are right, every day, at 2am when nobody is watching.

Meredith: full ownership

Meredith (now Dotdash Meredith) is where I learned what it means to fully own a data platform. Not as part of a large team where each person owns one component — as one of a small number of engineers responsible for the whole stack, end to end.

Full ownership is uncomfortable in a productive way. You can't specialise your way out of a problem. When the ingestion breaks, you fix it. When the transformation layer has a bug, you fix it. When a compliance requirement arrives — and at Meredith, CCPA arrived while I was there — you figure out how to implement it on infrastructure that wasn't designed for it.

I came out of Meredith with a much more complete picture of what a data platform actually is: not just the pipelines, but the governance, the access controls, the lineage, the operational model that keeps it running when the person who built it isn't around.

Amazon: the scale problem

Amazon (L5 Data Engineer on the Alexa Shopping team) was an education in a different kind of problem: what happens when the scale is large enough that the organisation becomes the bottleneck, not the technology.

50 PB of data. Over a thousand production jobs. Teams that each owned one slice of a much larger pipeline, often without visibility into what happened upstream or downstream. The engineering problems were interesting. The harder problems were coordination ones — how do you change something when your change will affect five teams you've never met?

Data contracts became essential. Not as a concept I read about, but as a practical tool for making cross-team change manageable. A producer commits to a schema, a freshness SLA, a set of quality guarantees. Consumers build against those guarantees. When the guarantees change, there's a process. Without that structure, at Amazon's scale, change is chaos.

The big data vs. warehouse divide (and its resolution)

When I started, there were two camps: the "big data" people (Hadoop, distributed systems, lakes, files) and the "warehouse" people (SQL, structured data, Redshift, Snowflake). The camps were more adversarial than the technical differences warranted.

The field has largely converged. The lakehouse architecture — data lakes with warehouse-quality query and governance on top — is now the default for serious data teams. Apache Iceberg brings ACID transactions, time travel, and schema evolution to S3. dbt brings software engineering practices to SQL. The boundary between "lake" and "warehouse" has blurred to the point where the question isn't which paradigm, but how to wire the pieces together correctly.

What "Data Engineer" means now

The job description has evolved significantly since I started. Early in my career, "data engineer" often meant Hadoop administrator with some ETL experience. Now it spans a much wider range: streaming infrastructure, ML pipelines, data platform products, analytics engineering.

What's stayed constant: the core problem is making data reliably available to people who need it, in a form they can use, with enough confidence that they can act on it. The tools change. The problem doesn't.

After twelve years across six companies, from software testing to petabyte-scale infrastructure, that's still the clearest way I know to describe what I do: I make data available to people who need it, and I try very hard to make sure it's right when they get it.

Supreeth M Gowda has spent 12+ years across data engineering, software engineering, performance engineering, and data science. He now consults through Encore — learn more about him and his services.