A Benchmark
That Can Never Hit 100

Reusing an anti-pattern checker as a benchmark for how well an AI agent authors Kubernetes manifests is a natural idea: the checker already exists, already scores objectively, already covers a documented catalog of real mistakes. Point the checker at whatever an agent produces, and the pass count becomes a number worth tracking across models, prompts, or catalog revisions. Nine of the catalog's categories scored cleanly this way from the start. Getting to eighteen took three more rounds of extending what the benchmark's submission format could accept — a promotion pipeline file, an app-registration file, a whole config directory instead of a single manifest. One category never joined the other eighteen, and it isn't going to.

What the Nineteenth Category Actually Measures

Most of the catalog checks something present in what an agent writes: does the Deployment have resource limits, does the Ingress have TLS, does the HorizontalPodAutoscaler have a sane range. The one holdout — drift, the gap between what Git declares and what's actually running on a cluster — measures something categorically different: a divergence that can only exist after a manifest has already been authored, already applied, and something (a person running kubectl edit, an operator reconciling a different intent, anyone changing the live state out-of-band) has since changed the cluster without updating Git to match.

An authoring benchmark scores what an agent writes. It has no mechanism to introduce drift, because drift isn't a property of a YAML file — it's a property of the relationship between a YAML file and a cluster's state hours, days, or months later, shaped by events the authoring step has no way to cause or prevent. Asking an agent's manifest to demonstrate "no drift" is asking it to prove a fact about a future it doesn't control.

Two Different Kinds of Missing

A checklist with an item nobody's gotten to yet and a checklist with an item that structurally cannot apply look identical if you only read the score: 18/19 either way. They call for opposite responses. The first is a backlog entry — schedule the work, and the number climbs to 19 eventually. The second is a fact about the tool's shape, and treating it as a backlog entry invites exactly the wrong instinct: someone eventually tries to make the number hit 19 anyway, which usually means fabricating a proxy signal for something the artifact under test genuinely cannot demonstrate.

The honest fix wasn't a fix at all — it was a documented, permanent exclusion. The scoreable ceiling for this benchmark is stated as 18 out of 19, not 19 out of 19, in the same doc that defines the scoring itself. Not a caveat buried in a footnote; the number the benchmark reports is defined, from the start, to never include the category that can't apply.

The General Principle

Any scorer, checklist, or benchmark that aggregates multiple checks into one number needs to distinguish two different reasons a check might be missing: not implemented yet, which is a plan, and cannot be measured by this kind of artifact, which is a permanent property of what's being scored. Conflating them either wastes effort chasing a ceiling that was never reachable, or — worse — creates pressure to fake the missing signal well enough to claim the full score. A benchmark that states its own ceiling honestly, in the same place it reports results, removes that pressure before it starts.