I've been in data engineering for twelve years. I've worked across software testing, performance engineering, data lakes, data science platforms, and production ETL at petabyte scale. And after all of it, I keep seeing the same mistake — at startups, at midsize companies, and at one of the largest tech companies in the world.
The mistake is this: we build for the problem we have today, not the problem we'll have in eighteen months.
That sounds obvious. It isn't. The under-building version of this mistake is obvious — people mock it as "move fast and break things." But the over-building version is just as destructive and far less discussed.
The over-engineering trap
Early in my career, I watched a team spend four months building a data ingestion framework that could handle fifteen source types, six output formats, and a pluggable transformation layer. They were processing three data sources.
The framework was elegant. It was also unmaintainable, poorly understood by anyone who hadn't built it, and abandoned within a year when the team changed. The three data sources eventually got rewritten in a weekend using a much simpler approach.
Over-engineering is seductive because it feels like responsibility. You're thinking ahead. You're building something that can scale. But abstraction has a cost: it separates the code from the problem it solves, and that separation makes the code harder to change when the problem turns out to be different than you expected.
The under-engineering trap
The opposite mistake is more visible but equally expensive. A pipeline that works today, built with no thought for tomorrow, accumulates debt at a rate that surprises teams when they finally try to pay it.
I've seen pipelines where every schema change required manually editing twenty separate SQL files. I've seen orchestration built on cron jobs that nobody could tell you were running or broken. I've seen data quality checks that existed in comments but not in code.
Under-engineering feels like speed. It is, briefly. Then it becomes the reason everything takes twice as long — because every new feature requires working around the last cut corner.
The hammer-and-nail problem
The version of this mistake I see most often now is subtler: teams pick a tool and fit every problem to it.
I've seen Spark used for data volumes that would be faster on a single Pandas DataFrame. I've seen Kafka introduced for a use case that a simple database queue would have served for a tenth of the operational cost. I've seen dbt models built for transformations that should have lived in the application layer.
When you're holding a hammer, everything looks like a nail. In data engineering, the hammer is usually whichever technology the team last got excited about.
The solution isn't to avoid powerful tools — it's to pick them for the problem, not for the résumé line. Kafka is extraordinary for what it does. It's also overkill for 90% of the use cases it gets deployed in.
What Amazon taught me about compounding decisions
At Amazon, I worked on infrastructure that had been running for years. The design decisions from five years earlier were still load-bearing — not because they were the best decisions in retrospect, but because the cost of changing them had grown with the scale of the system.
This is the part of data engineering that doesn't get enough attention: design decisions compound. A format choice made at 1 TB is the same format choice you'll live with at 100 TB. A schema convention established in year one shapes what's possible in year three. A monitoring gap that seemed acceptable at a hundred jobs is catastrophic at a thousand.
The teams that operated reliably at Amazon were the ones who treated early decisions as if they were permanent — because they often are. Not by writing endless design documents, but by asking "what breaks in this design when we're 10x bigger?" before committing to anything structural.
The maintenance problem
Here's the thing nobody tells you when you're starting out: maintenance is the real product.
Building a pipeline is a few weeks of work. Maintaining it, evolving it, debugging it at 2am, handling the edge case that only appears on the third Tuesday of a month with a specific upstream condition — that's years. The ratio of building to maintenance in a mature data organisation is probably ten to one in favour of maintenance, and most data engineers are trained almost entirely for the building phase.
This is why I care so much about operational simplicity. The most sophisticated solution is rarely the most maintainable one. The pipeline that can be understood by a new engineer in an afternoon has more long-term value than the one that requires the original author to debug it.
The modern stack era and why it's different
We're living through the most interesting period in data tooling I've seen. dbt changed how SQL transformations are built and tested. Airflow (and its successors) made orchestration visible. Iceberg made data lakes behave more like databases. The modern data stack is genuinely better than what we had five years ago.
But the same mistake persists. Teams adopt the modern stack as if the tools solve the underlying problem. They don't. Adopting dbt doesn't mean you have good data quality — it means you have the infrastructure to build good data quality, if you invest in it. The tool is the skeleton. The discipline is still the thing.
Why I built DataFoundry
After years of watching teams make the same architectural mistakes — over-engineered for the current scale, under-designed for the next one, tool-first instead of problem-first — I wanted to build something that helped teams get the architecture right before they committed to it.
DataFoundry is a free AI-powered tool that generates a data architecture recommendation based on your actual constraints: your data sources, your team size, your compliance requirements, your budget. It's not a vendor recommendation engine — it's a structured way to think through the decisions before you make them.
The same mistake — building for today's problem without thinking about tomorrow's — is one I've spent twelve years watching. DataFoundry is my attempt to make the first conversation about architecture better.
Supreeth M Gowda has spent 12+ years across data engineering, software engineering, performance engineering, and data science. He now consults through Encore — learn more about him and his services.