Kubernetes Has No "Done": Modeling Rollout Progress and Success

“When should a rollout be considered successful?”

The answer is that Kubernetes gives you the signals to decide, not the decision itself. A controller is not an operation that completes. It is an always-converging loop.

To a user, a rollout is a concrete event: “I deployed version X. Did it succeed or fail?” The controller does not see it that way. It moves current state towards desired state and keeps reconciling whenever they drift apart. The moments when they match are snapshots, not an ending.

A release pipeline cannot wait on “the controller is converging” forever. At some point it needs an answer for this specific rollout. Is it making progress? Did it reach the point where the next stage can safely begin? Or has it failed?

Platforms built on Kubernetes need to synthesize this answer. In practice, the contract has two parts:

1. Progress: is the rollout still moving towards desired state? It tells an orchestrator whether to keep waiting, nothing more.
2. Verdict: has the rollout reached a terminal outcome, succeeded or failed, that a pipeline can branch on?

Progress tells the platform whether to keep waiting. The verdict tells the pipeline whether it can continue. Success means the rollout reached the desired state and held it. Failure means progress stopped or the platform observed a terminal failure.

What Is Progress, Really?

In a converging system, progress means that the current state is moving towards the desired state. For a rolling update (and most other cases), it usually comes down to these observable transitions:

• More replicas have been created from the new pod template.
• Old replicas have been deleted.
• More replicas from the new pod template have become available.

As long as one of these transitions keeps happening, the rollout is making progress. It may be slow, but it is not stuck. Each observed transition resets the clock on progress. The question is not how long the rollout has taken in total, but how long it has been since anything meaningful changed. For Deployment-style rolling updates, the pace of a rollout is shaped by maxSurge and maxUnavailable. What the platform watches for is whether any of these transitions is still happening at all.

In an always-converging system, progress is the primitive you can define. Failure is derived from the absence of progress.

For a discrete rollout result, the platform eventually has to stop waiting and emit a failed verdict for that generation. The controller may keep reconciling, but “still reconciling” is no longer useful information for the upstream system waiting on a verdict.

Conditions are how the platform exposes that contract:

Kubernetes Deployments expose a similar idea through the Progressing condition.

The exact reason names are less important than the shape of the contract. The controller exposes a generation-scoped condition that upstream systems can trust.

That does not mean the Converged condition becomes sticky once it reaches True or False. It should reflect the controller’s current view of reality. When the controller observes new state, the condition should change with it. The terminal verdict is for the upstream system that needed a decision at that point in the rollout.

A subtle but important detail: any orchestration system reading a condition needs to make sure it is looking at status for the current spec. When a condition carries observedGeneration¹, it should match the resource metadata.generation. Otherwise the system may be reading a stale status capturing the result of a previous rollout.

The Progress Deadline Defines Failure, but It Is Not the Right User Knob

Kubernetes exposes this idea through progressDeadlineSeconds on Deployment objects. It is the amount of time the Deployment controller waits for new progress before setting the Progressing condition to False with reason ProgressDeadlineExceeded.

The concept is a useful primitive for the platform, but not the right user-facing knob. What users need is for the platform to apply a progress budget: how long the platform should wait without meaningful movement before classifying the rollout as failed for orchestration purposes.

That budget is rarely something an application team can set well on its own. The application contributes important inputs, but it does not own the whole equation. A good budget combines what the workload declares with what the platform guarantees.

Some parts come from the workload:

• Startup timeout: how long the container may take to start.
• Readiness behavior: how long the container may take before it can serve traffic.
• minReadySeconds: how long a pod should remain ready before counting as available.

Other parts come from the platform:

• Scheduling latency: how long it takes to bind a pod to a node.
• Pool autoscaling: how long it takes to add capacity, if needed, when the rollout triggers a scale-up.
• Node provisioning: how long newly requested machines take to join the cluster.
• Image distribution: image size, pull concurrency, and node cache state.

When we built a CRD for our workloads, exposing this as a user-configurable field seemed natural. Kubernetes has one, so ours should too. But that field made a bad claim: that application teams could choose the progress deadline correctly, even though they only had visibility into part of it.

The failure mode is concrete. If the progress deadline is shorter than a container takes to start up, the rollout fails even though nothing is broken. A validation webhook can catch obvious mistakes, like a deadline shorter than the configured startup and readiness probes, but it cannot model the platform side of the budget.

A better contract is to take the workload-owned inputs and let the platform calculate the deadline from the SLOs it holds for each phase of the rollout path. In practice you can start with percentile bounds for each phase and add them up: scheduling, capacity provisioning, image pull, container startup, readiness, and minReadySeconds. It is not a perfect statistical model. It is an operational budget for how long “no progress” should be tolerated before the rollout is classified as failed.

The budget is a ceiling, not a fixed wait. Some failures are terminal, and waiting out the rest of the budget buys nothing. If the application container is crashing, no amount of additional scheduling, provisioning, or image-pull time will change the outcome, so we exit early and mark the rollout failed rather than burn the remaining budget.

When the budget is exhausted, or a terminal failure is detected, the verdict should explain where the rollout stopped, not just that a deadline passed. We categorize failures based on pod lifecycle and conditions: things like waiting for scheduling, a platform init container failing, an unpullable image reference, or the application container crashing. That taxonomy and the observability around it is a post of its own. My colleague Vasudev Bongale gave an excellent talk at KubeCon NA 2025 on making rollouts observable, and I’d highly recommend it if you’re building a compute platform.

Success Is a Snapshot Contract

A rollout also needs a success signal, but success is different from progress. Progress tells an upstream system that continued waiting is justified. Success tells it that this rollout has reached a snapshot it can safely branch on.

A rollout is rarely the end of a pipeline. It may be followed by canary analysis, promotion to another stage, or rollout to the next production environment. Each of those steps needs a discrete answer before it can move.

Live status is still useful. It lets users see where the rollout is right now. But a pipeline needs a decision: continue, fail, or promote.

A useful definition of success is that the available replicas on the new version have reached the desired count. But a point-in-time match can be fragile. A pod can pass its readiness probe once and fail it a moment later. Treating that instant as success can promote a rollout that will end poorly.

Use minReadySeconds

Kubernetes gives you minReadySeconds for this. When configured, a pod must stay ready for that duration before it counts as available. The default is zero, so the default bar is still “ready once.” Setting it raises the bar to “ready and not flapping,” which is much closer to what a rollout success verdict should mean.

Even then, success is not a permanent guarantee. Pods can become unhealthy later, nodes can fail, or the desired state can change again. That does not mean the rollout never succeeded.

This distinction matters operationally. Rollout success is not the same as ongoing workload health. A rollout answers whether a specific version transition reached its success criteria. Workload health answers whether the running system is healthy now. A pod can become unhealthy after a successful rollout, and that should be handled by health monitoring, remediation, or a new rollout decision. It should not retroactively change the verdict for the rollout that already completed.

The API Lesson

The API design lesson we have learned several times:

Don’t expose a knob just because the system underneath has one.

A good platform API should not be a faithful mirror of Kubernetes. It should expose the outcomes users and higher-level systems need, while the platform owns the machinery required to compute them.

That is the job of a compute platform: turn a continuously reconciling substrate into contracts people can reason about.