The conversation in data engineering right now goes something like this: AI will write your pipelines. AI will catch your bugs. AI will review your code.
And it will. It already is.
But here's what nobody is saying loudly enough: an AI-generated DAG still has nowhere safe to run if your environments aren't set up correctly underneath it. A Copilot-written pipeline will still fail in ways you didn't expect. The foundations don't become optional because the tooling got smarter. If anything, they matter more.
This is a piece about those foundations — specifically environments and CI/CD, what they actually look like in practice, and why getting the data flow between them right is harder than most articles admit.
What environments are — and what they're not
An environment is an isolated instance of your data infrastructure — compute, storage, orchestration, configuration — that serves a specific stage of your delivery process.
In theory, the full topology looks like this:
Dev — where engineers write and iterate. Data is synthetic or sampled. Nothing needs to be stable, just fast to work in.
Test / Integration — where code from multiple engineers gets combined and validated together. The first place "it works on my machine" gets disproven.
Pre-Production (PPT / Staging / UAT) — production's mirror. Same infrastructure, same config, same data shape. Where stakeholders validate and where you find the bugs that only appear at production scale.
Performance — a dedicated environment for load testing and throughput benchmarking. For teams with strict SLAs or high-volume pipelines where a slow query in prod costs real money.
Prod — where the business runs.
In practice? I've rarely seen all five. Most companies run three, sometimes four. Pre-prod and performance testing often collapse into the same environment. Some teams treat dev as pre-prod and push straight to prod. Others develop locally, call it dev, then go PPT → prod with nothing in between.
None of these setups is automatically wrong. The mistake is when it happens by accident rather than by design.
The problem nobody talks about: data flow
The environment topology is the easy part. The hard part is what data flows through each environment — and this is where I've seen more teams struggle than anywhere else.
Dev doesn't generate enough data. Real pipeline behaviour only surfaces at volume. So the question becomes: do you develop using test data? Production data? Simulated data? Each choice has trade-offs.
Using production data in dev solves the volume problem but creates a governance one. Using simulated data keeps things clean but means you're developing against data that doesn't reflect reality. Using test data is the middle path — but test data has bugs. It has untested features from other engineers that haven't been validated yet. If your pipeline consumes upstream data that's in an unstable state, your dev work gets contaminated by someone else's in-progress changes.
QA environments have a version of this problem too. The data flows, but it carries whatever is currently being tested — partially implemented features, schema changes mid-migration, transformations that haven't been signed off. If a downstream pipeline depends on something upstream that's in flux, it breaks. And the break looks like your bug, not theirs.
Pre-prod gets production data — or production-like data — which is why it's the most valuable environment for catching real issues before they ship. But it costs more to run, and that cost means teams cut corners on it. Some run full automation suites in PPT specifically to get confidence before promoting to prod. Others treat it as a rubber stamp and barely use it.
The most insidious problem I've seen: orchestration only exists in PPT and prod. Not in QA or dev. Which means engineers working in lower environments are running transformations manually, working on stale data, and making assumptions about pipeline behaviour that they can't actually validate until the code is much further along in the promotion chain. By the time the issue surfaces, it's expensive to fix.
The real architectural decision isn't which environments to have. It's which environments need stable, production-like data, which can tolerate test data with known issues, and which need orchestration wired in versus which can rely on manual runs. That decision has to be made deliberately, with input from both engineering and the stakeholders who depend on the data — because the answer changes depending on how the business actually uses the data.
CI/CD: what it should do and what it usually does
CI/CD — Continuous Integration and Continuous Delivery — is what connects your environments into a delivery system. Without it, environments are just isolated servers. They don't enforce anything. They don't move code systematically.
In practice, most teams have partial CI/CD. Some resources go through an automated pipeline. Others — often the newer, less mature tooling — get deployed manually. This isn't laziness. It's a reflection of where the tooling ecosystem is at. When a tool is new, the infrastructure for automated deployment often hasn't matured yet. Manual deployment is the pragmatic choice until it isn't.
The goal is to get to a place where:
Every code change triggers an automated build and test run. Unit tests, data quality checks, schema validation — all of it runs automatically on every push. The point is to catch regressions before they accumulate.
Once code passes integration, it moves through your environment chain — dev → test → PPT → prod — with validation criteria at each stage. Nothing moves forward until those criteria are met.
Deployment to prod is a boring event, not a stressful one. It's automated, auditable, and has a tested rollback path.
Data engineering CI/CD has specific challenges software teams don't face in the same way. Your pipeline is stateful — a DAG change in Airflow affects which runs have happened, which are scheduled, which are in flight. Schema migrations have to be coordinated with pipeline changes. Backfills have to be planned. You can't treat a data pipeline deployment like a stateless web service deployment.
The other consistent failure: environments sharing configuration they shouldn't. Dev pointing at a prod S3 bucket. Staging writing to a prod schema. These happen because per-environment configuration isn't enforced — it's assumed. The most painful production incidents come from pipelines running in the wrong environment because someone didn't isolate the config correctly.
Airflow makes this concrete
Airflow is where the environment gap becomes most visible, because Airflow manages orchestration state — and orchestration behaviour only reveals itself when it's actually running.
Most teams get this partially right. Having Airflow in two environments — dev and prod, or PPT and prod — is table stakes. Most companies I've seen are already there. DAGs have alerting baked in: task failures trigger email, Slack, Teams, or PagerDuty notifications. Failures don't go unnoticed.
The gap is the third environment.
When Airflow only exists in two places, developers write DAGs in an environment with no real orchestration context — no scheduler running, no upstream dependencies, no realistic data flowing through. They validate logic manually, then promote to PPT or prod to see how it actually behaves under orchestration. That's too late in the cycle. Issues surface after the code has already moved forward, which makes them more expensive to fix.
Adding Airflow to a test/QA environment closes this gap. It gives developers a realistic orchestration context during development — not a production mirror, but enough to see how a DAG behaves when the scheduler is running, when upstream tasks are completing, when real dependencies are in play. The kinds of issues that only appear under orchestration get caught earlier, before they reach PPT or prod.
The decision of where to run Airflow is also a data flow decision. A test/QA Airflow instance needs data to orchestrate against — and that data has the same challenges described earlier: it may be stale, it may have untested upstream changes, it may not reflect production volume. But even an imperfect orchestration environment is more useful than none. The goal isn't a perfect replica. It's enough signal to catch the issues that only appear when the scheduler is actually running.
The fix is deliberate: decide which environments need orchestration, provision Airflow there, and wire it into your CI/CD promotion sequence so that a DAG change is validated under real orchestration conditions before it reaches prod.
Why this matters more in an AI-assisted world
AI makes this problem more acute, not less.
When code generation gets faster and cheaper, the volume of code going into your systems goes up. More DAGs, more pipelines, more transformations, faster iteration cycles. If your environment infrastructure isn't set up to handle this throughput safely — if you don't have the gates and the promotion sequence and the per-environment configuration — you're not moving faster. You're accumulating risk faster.
AI can write a syntactically correct DAG. It can scaffold a CI/CD workflow. It can suggest the right operator for a given task. What it cannot do is tell you whether your staging environment has the right connections provisioned, whether your prod config matches what your DAG expects, or whether a schedule change in PPT is going to cause a data gap when you promote it.
That judgment — the judgment of someone who has debugged a prod incident caused by a mismatched Variable, who has had to explain to a stakeholder why the pipeline worked in staging but failed in prod — that judgment is still yours to carry.
The teams that will move fastest with AI aren't the ones who skip the foundations because AI made it feel optional. They're the ones who get the foundations right and let AI accelerate everything built on top of them.
A practical starting point
If you're inheriting a setup that doesn't have this right, or building one from scratch:
Small team: Dev and prod, with explicit per-environment Airflow Connections and Variables, and a CI/CD pipeline that runs tests on every push and requires passing checks before promotion to prod. No manual deployments to prod.
Growing team: Add a staging environment. Make it a real Airflow instance with production-like config. Use it as the gate before prod promotion. Decide deliberately what data flows through it and how orchestration is wired.
Mature team with SLAs: Add a performance testing environment and a dedicated integration test stage in CI. Every change goes through integration tests in a clean environment before staging. Define clearly which environments need stable data, which can tolerate test data, and where orchestration lives vs. where it doesn't.
The specific topology matters less than the discipline: every environment has its own configuration, every promotion is validated, and the data flow between environments is a deliberate decision — not something that happened by accident and nobody revisited.
The high-level conversations in data engineering are worth having. Architecture matters. AI capability matters.
But the teams that ship reliably aren't the ones with the most sophisticated architecture. They're the ones who got their environments right, understood what data should flow through each one, and built the discipline to respect the boundaries between them.
No amount of AI tooling changes the fact that foundations have to be poured correctly before you build on top of them.
Supreeth M Gowda has spent 12+ years across data engineering, software engineering, performance engineering, and data science. He now consults through Encore — learn more about him and his services.