Kubernetes · AI Agents
A Tied Score,
Two Different Kinds of Wrong
A check that confirms a NetworkPolicy exists can't tell you what the policy lets in, and a check that confirms a promotion pipeline exists can't tell you whether it points at anything real. Two models given the same Kubernetes manifest-authoring task both scored 9/9 on the same automated checker, an exact tie, independently reproduced. Reading what each one wrote found two real defects the checker had no way to see, and they didn't cancel out. Each model was wrong in a way the other wasn't.
The scorer comes from my example project that catalogs common Kubernetes deployment mistakes, each paired with a check that finds it in a manifest. The task: add an internal-only service tracking parcel shipments, with bursty nightly-batch traffic, a database credential, and a required dev→staging→prod promotion pipeline with a verification gate at each stage. Two models, identical prompt, run in separate worktrees so neither could see the other's work. Both were told to run the scoring script themselves and iterate until every applicable check passed.
The Score Told Nothing Apart
Both self-reported 9/9 applicable checks passed (1 N/A excluded). Independently rerunning the checker against each model's committed files, from the canonical checkout rather than trusting the self-report, reproduced both numbers. On the structural axis the checker measures (resource limits, probes, TLS, RBAC scope, autoscaling sanity, promotion-gate presence), there was no daylight between them at all. Not a near-tie. An identical result, verified twice.
A checker that produces a tie is doing its job; it isn't obligated to find a difference where none exists on the axis it checks. The interesting part starts where a tied score would normally end the comparison.
A NetworkPolicy That Exists but Lets Everything In
Reading one model's NetworkPolicy directly turned up a rule that passes the check and shouldn't. Alongside a correctly-scoped ingress rule naming the expected webhook-gateway namespace, it added a second rule with namespaceSelector: {} — an empty selector, which Kubernetes matches against every namespace in the cluster, not "other internal services," as the rule's own comment claimed. That makes the first, carefully-scoped rule pointless: the policy as a whole accepts traffic from any pod in any namespace on the service's port. A real least-privilege violation, invisible to the check, because check_networking.py's netpol rule only confirms a NetworkPolicy exists in the namespace, not that what it allows matches what it's supposed to allow. The other model's ingress rules named two real namespaces, with no catch-all anywhere.
A Pipeline That Exists but Points at Nothing
The gap ran the other direction on a different file. The checker's own minimal fixture for the promotion check contains only Stage resources, deliberately minimal, since its only job is to be scored. A real Kargo pipeline also needs a Warehouse, the object a Stage's requestedFreight[].origin points at as its freight source, and typically a Project to contain both. One model's pipeline mirrored the minimal fixture closely enough to pass the check — and referenced a Warehouse that was never defined anywhere in its submission. Passed the check. Would never discover freight on a real cluster; the reference points at nothing. The other model's pipeline included the matching Project and Warehouse, meaning it had read further into the documentation than the minimum needed to satisfy the scorer, and produced something that would work if applied for real.
The model with the self-defeating NetworkPolicy had the more complete, deployable promotion pipeline. The model with the correctly-scoped NetworkPolicy had the promotion pipeline that references a resource that doesn't exist. A tied structural score sat on top of two independent, unrelated quality gaps, one per model, in opposite files.
One More Difference the Score Never Asked About
Neither runAsNonRoot difference was scored, but one model's container securityContext went further than the other's: readOnlyRootFilesystem: true, allowPrivilegeEscalation: false, and capabilities.drop: [ALL], stacked on top of the baseline runAsNonRoot the check looks for. The check's own least-privilege item only verifies the one field it was written to verify, so a fuller answer to the same principle simply doesn't register as a higher score.
What a Tie Means
A tied structural score does not mean tied output quality, not in either model's favor, and not by a small margin either time; a self-defeating catch-all firewall rule and a pipeline that would fail without an error to discover freight are both the kind of defect that matters in production. What it means is narrower and more useful. The checker measures what it was built to measure, correctly, and anything outside that (whether a rule's logic does what its comment claims, whether a referenced resource exists elsewhere in the same submission) has to be checked by reading the output directly, every time, independent of which model produced it or how the two scores happen to compare.
docs/benchmark.md (the full task, both submissions, and the complete quality-gap analysis, in my example project that catalogs common Kubernetes deployment mistakes with a checker for each)