A Sync Loop Is Not a Handshake: Keeping External Systems in Lockstep with Kubernetes

“How do you keep an external system in lockstep with what Kubernetes knows?”

Kubernetes is the source of truth for a lot of state. Which pods exist, which ones are ready to serve traffic, which ones are going away. Plenty of systems outside the cluster need that same state. A service discovery registry is the obvious one. It needs to know the exact set of endpoints that can take traffic right now.

The usual answer is a controller. Watch the objects you care about, push the changes to the external system, reconcile on drift. It works most of the time. The problem is the “most of the time” part. A plain sync loop is best-effort and eventually consistent. When it falls behind, or the thing it writes to is down, Kubernetes keeps moving and the external system is left behind.

This post is about closing that gap. I’ll use service discovery as the running example because it makes the failure concrete. But the real subject is two coordination primitives Kubernetes gives you, readiness gates and finalizers, and how they let an external system participate in the pod lifecycle instead of chasing it.

The baseline: mirror EndpointSlices

Kubernetes has a Service, which is an abstraction over a group of pods selected by a label. A controller in the control plane watches Pods and Services and maintains EndpointSlice objects for each Service. An EndpointSlice is a list of endpoints backing the Service: IP, port, pod name, and readiness and termination state. It’s updated as pods come and go.

Say you have a service discovery system that lives outside the cluster, and clients ask it for the endpoints of a given app. You need those endpoints to match reality. The Kubernetes-native way to feed it is a controller that watches EndpointSlices and pushes the endpoints into the external registry.

EndpointSlices are the right primitive here when the rollout controller shares fate with the EndpointSlice controller. In vanilla Kubernetes they do: the Deployment controller and the EndpointSlice controller both live in the kube-controller-manager, the same binary. If it’s down, nothing rolls out and nothing updates endpoints, so a rollout can’t march ahead while endpoint updates quietly fall behind. They fail together. The trouble starts when responsibility is split: one system drives the rollout, and a separate, independent component owns the service discovery endpoints. Now the two can fail on their own, and a rollout can outrun the endpoints.

Where the gap is

Let’s walk through a rollout. Assume the sync controller is unhealthy at the wrong moment, or the external registry is refusing writes.

• A rollout starts. New pods come up. Their containers pass their readiness probes and are ready to serve, so kubelet marks the pods Ready.
• The controller managing the pods sees ready pods and makes progress: it deletes old pods and creates more new ones.
• The rollout finishes. From Kubernetes’ view, everything is healthy. Desired state reached.
• The sync controller was down the whole time (or the registry wasn’t accepting writes), so the external registry never learned about the new endpoints. It still holds the endpoints for pods that no longer exist.

Now the deploy is “done” and there’s nothing for clients to reach. Traffic is black-holed against a rollout that Kubernetes considers a success. Nobody gets paged for a failed deployment, because it didn’t fail. The rollout succeeded, and the registry has stale data.

The root cause is that pod readiness and registration are decoupled. The pod was marked Ready and the rollout moved forward with no acknowledgement that its endpoint had been written to the external registry. Pod readiness says “the container can serve traffic.” It says nothing about whether the system that routes traffic to it has that endpoint yet.

So the fix isn’t a more reliable sync loop. A sync loop can and will fall behind. The fix is to make readiness depend on registration, so a rollout can’t move forward until the external registry acknowledges it received the endpoint of the new healthy pod.

The fix

Concretely, that’s a controller that watches pods instead of EndpointSlices. It uses a readiness gate and a finalizer to participate in the pod lifecycle, so the external registry stays in lockstep with Kubernetes.

Gate “Ready” on registration with a readiness gate

A container has a readinessProbe, and kubelet uses it to set the pod’s ContainersReady condition. Normally the pod’s Ready condition just follows ContainersReady. A pod readiness gate changes that rule. When a pod declares a readiness gate, the pod is Ready only when every container is ready and every readiness gate condition is True.

The gate is a custom condition that you own. You declare it in the pod spec, and flip it to True in the pod’s status when the desired condition is met:

apiVersion: v1
kind: Pod
metadata:
  name: web-1
spec:
  readinessGates:
  - conditionType: discovery.example.com/registered

In our service discovery example, the same sync controller clears the readiness gate. Instead of watching EndpointSlices, it watches pods. When a pod’s ContainersReady condition goes True, it registers that pod’s endpoint with the external system. Only once registration succeeds does it set the gate condition to True:

status:
  conditions:
  - type: ContainersReady
    status: "True"
  - type: discovery.example.com/registered
    status: "True"
  - type: Ready
    status: "True"

The pod is Ready only after all three line up. The Ready condition now carries an extra guarantee: the external system has accepted this endpoint.

Let’s see what that does to the failure above. If the controller is down, or the registry is refusing writes, the gate condition stays False, so the new pods never become Ready. The rollout can’t count them as Available, so it can’t complete. It may still take down a few old pods up to maxUnavailable, but it stalls there and eventually trips its progress deadline (ProgressDeadlineExceeded) and is marked failed. That’s the outcome we want. A deploy that can’t register its endpoints should fail loudly instead of quietly reporting success. The readiness gate turns “the external system is unavailable” into back-pressure on the rollout.

The AWS Load Balancer Controller does exactly this: it injects a pod readiness gate so a pod isn’t marked Ready until its target is registered and healthy in the ALB or NLB target group. That keeps a rolling update from tearing down the old pods before the new ones can actually take traffic. Same primitive, pointed at a different external system.

Gate “Deleted” on cleanup with a finalizer

Registration is only half the story. The other half is deletion, and it has the same decoupling problem in reverse.

When you delete a pod, Kubernetes doesn’t remove it right away. The API server sets metadata.deletionTimestamp to a time in the future: the moment you asked, plus the pod’s grace period. That timestamp is a deadline, not a tombstone. It can be shortened if the pod drains early, but never pushed further out. The pod moves to Terminating, and kubelet runs graceful shutdown: any preStop hook, then SIGTERM, then SIGKILL if the container outlives the grace period. Kubelet stops the containers, but it doesn’t delete the pod object. Once the containers are gone, it sends the API server a final delete, and the API server is what removes the object from the datastore.

Without coordination, here’s the race. The pod object can be gone from the API before the sync controller manages to clean up the external registry. If the controller or the registry is down at that moment, the pod disappears and a stale endpoint is left behind. Now we need an out-of-band garbage collector to find and clean up orphaned records, which is its own source of bugs.

Finalizers help close this gap. A finalizer is a key on an object that blocks its removal until the key is taken off. As long as the finalizers list is non-empty, deletion is blocked. While the finalizer is present, the pod object stays around with its deletionTimestamp set, and the API server won’t remove it.

The finalizer only holds the pod object in the API. It doesn’t keep the containers running: kubelet still drains and kills them on the grace period, finalizer or not. What it buys you is a guarantee that the object sticks around long enough for the de-registration to run.

apiVersion: v1
kind: Pod
metadata:
  name: web-1
  finalizers:
  - discovery.example.com/deregistered

Both the finalizer and the readiness gate have to be on the pod from the start. So whatever creates the pods needs to put both in place up front, either in the pod template it stamps out or through a mutating admission webhook that injects them.

Now the controller has two jobs across the teardown, and the order matters:

1. When the deletionTimestamp first appears, stop the registry from handing this endpoint to new clients. Exactly how depends on the system: a health-aware registry can mark it unhealthy or draining, another one may only support removing it. The point is to cut off new traffic without pulling the pod out from under its in-flight requests.
2. Wait until the pod has actually drained, then remove the endpoint for good and take the finalizer off.

For step 2 you need a signal that the containers are gone. For a normal pod, that’s every entry in status.containerStatuses sitting in a terminated state. There are edge cases: init containers and restartable sidecars show up there too, and a pod that never started or whose node went unreachable may never report terminated. The controller needs a fallback for those, a timeout or a force path, or the finalizer will keep the pod object in the API forever.

The finalizer is what makes step 2 safe. The object can’t be reaped until de-registration has succeeded, so a failed de-registration or an unhealthy controller holds the pod instead of leaking a record. The cleanup happens before Kubernetes removes the object, so you don’t need a separate job chasing leaks.

The two gates bracket the pod’s life. A new pod can’t take traffic until the registry has it, and a finished pod can’t be reaped until the registry has let it go. The rollout waits on the registry at the front, cleanup waits on it at the back.

The tradeoff: rollouts now depend on the registry

None of this is free. A plain sync loop is best-effort, but it never blocks a deploy. With readiness gates and finalizers, both rollouts and teardown take a hard dependency on the controller and the external registry. If either is unhealthy, rollouts stall and pods pile up in Terminating.

For service discovery that’s the desired failure. A pod that can’t be registered shouldn’t take traffic, and a pod that can’t be de-registered shouldn’t vanish and leave a dangling endpoint. So reach for this when a stale record actually causes harm, not by default.

A couple of caveats:

• Readiness can flap after registration. If a running pod’s ContainersReady goes back to False, the controller should mark that endpoint unhealthy in the registry, the same move as the start of a graceful teardown, so clients stop routing to it. If the registry distinguishes health and registration, don’t de-register it: the pod may recover, and you want it back without churn. If the controller or the registry is unhealthy at that instant, the registry keeps advertising a stale endpoint until things recover, so clients still need to tolerate an endpoint that errors and retry another one. Coordination makes the system resilient but doesn’t remove the need for clients to be defensive.
• Finalizers can wedge deletion. The flip side of “the pod can’t disappear until cleanup succeeds” is that a broken controller can leave pods stuck in terminating. That’s a deliberate choice: a stuck delete you can see and alert on beats a silent leak you can’t. But it means the controller that owns the finalizer is now on the critical path for teardown, and you should treat it that way.
• Watching pods is heavier than watching EndpointSlices. EndpointSlices aggregate up to 100 endpoints per object, so the baseline controller watched relatively few of them. This one watches pods, one object each, and pods churn on every status update, not just readiness changes. The informer cache is much larger and the event stream much busier. On a large fleet, that’s real load to size for.

The pattern

Service discovery is just the example. The pattern generalizes to any external system that has to stay in lockstep with Kubernetes data.

A sync loop is a one-way push. It sends updates and hopes they land, and Kubernetes never waits for an answer. A readiness gate and a finalizer turn that push into a handshake: a pod isn’t Ready until registration succeeds, and it isn’t deleted until de-registration does. Kubernetes waits for the acknowledgement. That’s the difference between eventually consistent and in lockstep.