A green pipeline. A broken system.

10:17 AM. Support pings:

“VIP users are seeing BASIC pricing again.”

You open Grafana.
Everything looks… boring.

• error rate: flat.
• latency: normal.
• CPU: normal.
• last deploy: 3 days ago.
• pipeline: ✅ green.

You try the same flow locally. Works.
In staging. Works.
In production… sometimes.

Someone says: “cache?”
Another: “feature flag rollout?”
Someone else: “is this region-specific?”

You roll back the last release anyway.

Nothing changes.

That’s when the worst feeling appears: The system is broken, but nothing is failing..

What actually changed

Three hours later, you find it.

A producer service had evolved an event schema.
A new field had been added: customerTier.

That field quietly became business-critical.
Routing logic, pricing rules, and entitlements now depended on it.

Most consumers had been updated.
One consumer — in one environment — had not.

That consumer didn’t crash.
It didn’t throw errors.
It didn’t trigger alerts.

It simply defaulted to BASIC.

No exception.
No obvious signal.
Just quietly wrong behavior.

This wasn’t a bad deploy

Nothing failed in CI/CD.

The application artifact was built, tested, and deployed exactly as designed.

What changed happened outside the application lifecycle:

• an external contract evolved
• behavior now depended on it
• not all parts of the system evolved together

CI/CD did its job.

System evolution happened out of band.

The broken assumption

Many teams implicitly believe:

“If it went through CI/CD, it’s controlled.”

That assumption only holds for application code.

Modern systems are no longer self-contained binaries.
They depend on external systems that change over time:

• database schemas
• message contracts and event schemas
• permissions, roles, and policies
• external APIs and SaaS configuration

Those dependencies often evolve independently of the application release.

CI/CD was never designed to coordinate that evolution.

CI/CD’s real boundary

CI/CD is extremely good at answering one question:

“How do we reliably deploy application code?”

It is not designed to answer:

• What external changes must happen with this release?
• In what order should those changes occur?
• Which environments have applied which changes?
• What does rollback mean when the change isn’t a binary?

When teams stretch CI/CD to cover these concerns, they rely on:

• runbooks
• tickets
• manual coordination
• tribal knowledge

That works — until scale makes it collapse.

Why this failure mode is so dangerous

This class of failure is especially painful because:

• nothing crashes
• dashboards look healthy
• alerts don’t fire
• rollback doesn’t help

The system is up, but behavior is wrong.

By the time the issue is detected:

• business impact already happened
• the source of truth is unclear
• responsibility is diffuse

This is drift. Sometimes it’s loud. Often, it’s silent.

Why this keeps getting worse

As systems grow, so does the surface area of change:

• more services
• more teams
• more environments
• more third-party dependencies

Each new dependency introduces assumptions:

• about order
• about compatibility
• about timing

Those assumptions are rarely encoded anywhere.

They live in:

• people’s heads
• outdated docs
• Slack threads
• “don’t forget to…” steps

Over time, assumptions drift.

And drift is hardest to detect precisely because nothing explicitly breaks when it’s introduced.

This is not an argument against CI/CD

CI/CD is necessary.
It just isn’t sufficient.

Deploying applications is not the same as evolving systems.

Modern systems need a way to:

• make external changes explicit
• apply them deterministically
• reason about versions of the entire system, not just the app

Until then, teams will keep shipping green pipelines —
while system behavior quietly diverges underneath.

The question worth asking

CI/CD deploys applications.

So what evolves the systems they depend on —
explicitly, deterministically, and in sync with releases?

That missing layer is where the real work begins.

In the next post, we’ll name this problem — and the discipline emerging to solve it.