The Defaults
Nobody Declared

Detecting drift sounds like a simple diff: take what's declared in Git, take what's actually running, compare the two, flag what doesn't match. Validating that against a real cluster instead of hand-written fixtures meant applying a known-good manifest to a disposable kind cluster, dumping the live state with a genuine kubectl get -o yaml, and comparing that real capture against the Git source it came from — the same source, seconds after a clean apply, before anything had a chance to change.

A Diff That Should Have Been Empty

It wasn't. The comparison flagged drift across nearly every resource, on fields nobody had touched: spec.strategy, imagePullPolicy, resources, dnsPolicy, securityContext, and more, all present in the live capture and absent from the Git manifest. Not because someone had changed the cluster out-of-band — this was the very first read, immediately after apply. The API server and its admission defaulting had filled in every one of those fields on their own, the moment the resource was created, exactly as Kubernetes is designed to do. A Deployment with no strategy specified doesn't run without one; the API server picks RollingUpdate and writes it back into the object's own spec. A container with no imagePullPolicy gets one assigned based on the image tag. None of this is drift. All of it looked exactly like drift to a checker doing a naive full-object comparison.

The Fix Wasn't Smarter Diffing — It Was a Smaller Diff

The fix is an explicit allowlist: a fixed set of keys — SERVER_DEFAULTED_KEYS — known to be commonly filled in by the API server or its admission controllers, excluded from the comparison before drift is evaluated. Not inferred at runtime, not guessed from context — a maintained list of the specific fields a cluster is expected to add on its own, checked once against a real cluster's actual behavior rather than assumed from documentation. A field on that list showing up in the live capture but not in Git no longer counts against the resource; a field not on that list doing the same thing still does, correctly.

A second, smaller issue rode along with the first: containers are a list, and a naive list comparison fails the whole list the moment any one container in it has a defaulted field the others don't — even if every container is otherwise identical to what Git declared. The fix there was to diff containers by name rather than by list position, so one container's legitimate defaulting doesn't drag every sibling container in the same Deployment into a false positive alongside it.

Why This Is Worth Getting Right

A drift check that cries wolf on every fresh apply doesn't get ignored gently — it gets disabled, or worse, everyone learns to skim past its output because it's never actually clean. The entire value of a drift signal depends on silence meaning something: no output means nothing has actually diverged. A checker that can't tell "the platform did this automatically, as designed" from "someone changed this by hand, out-of-band, in a way Git doesn't know about" can't produce that silence, no matter how correct its comparison logic is otherwise.