A Tied Score,
Two Different Kinds of Wrong

The task: add an internal-only service tracking parcel shipments, with bursty nightly-batch traffic, a database credential, and a required dev→staging→prod promotion pipeline with a verification gate at each stage. Two models, identical prompt, run in separate worktrees so neither could see the other's work. Both were told to run the scoring harness themselves and iterate until every applicable check passed.

The Score Told Nothing Apart

Both self-reported 9/9 applicable checks passed (1 N/A excluded). Independently rerunning the harness against each model's actual committed files, from the canonical checkout rather than trusting the self-report, reproduced both numbers exactly. On the structural axis the harness measures — resource limits, probes, TLS, RBAC scope, autoscaling sanity, promotion-gate presence — there was no daylight between them at all. Not a near-tie. An identical result, verified twice.

A NetworkPolicy That Defeats Itself

Reading one model's NetworkPolicy directly turned up a rule that passes the check and shouldn't. Alongside a correctly-scoped ingress rule naming the expected webhook-gateway namespace, it added a second rule with namespaceSelector: {} — an empty selector, which Kubernetes matches against every namespace in the cluster, not "other internal services," as the rule's own comment claimed. That makes the first, carefully-scoped rule pointless: the policy as a whole accepts traffic from any pod in any namespace on the service's port. A real least-privilege violation, invisible to the check, because check_networking.py's netpol rule only confirms a NetworkPolicy exists in the namespace — not that what it actually allows matches what it's supposed to allow. The other model's ingress rules named exactly two real namespaces, with no catch-all anywhere.

A Promotion Pipeline That References Nothing

The gap ran the other direction on a different file. This repo's own minimal fixture for the promotion check contains only Stage resources — deliberately minimal, since its only job is to be scored. A real Kargo pipeline also needs a Warehouse, the object a Stage's requestedFreight[].origin actually points at as its freight source, and typically a Project to contain both. One model's pipeline mirrored the minimal fixture closely enough to pass the check — and referenced a Warehouse that was never defined anywhere in its submission. Passed the check. Would never discover freight on a real cluster; the reference points at nothing. The other model's pipeline included the matching Project and Warehouse, meaning it had read further into the actual documentation than the minimum needed to satisfy the scorer, and produced something that would work if applied for real.

One More Difference the Score Never Asked About

Neither runAsNonRoot difference was scored, but one model's container securityContext went further than the other's: readOnlyRootFilesystem: true, allowPrivilegeEscalation: false, and capabilities.drop: [ALL], stacked on top of the baseline runAsNonRoot the check actually looks for. The check's own least-privilege item only verifies the one field it was written to verify — a fuller answer to the same principle simply doesn't register as a higher score.

What a Tie Actually Means

A tied structural score does not mean tied output quality — not in either model's favor, and not by a small margin either time; a self-defeating catch-all firewall rule and a pipeline that would silently fail to discover freight are both the kind of defect that matters in production. What it means is narrower and more useful: the harness measures what it was built to measure, correctly, and anything outside that — whether a rule's logic actually does what its comment claims, whether a referenced resource actually exists elsewhere in the same submission — has to be checked by reading the output directly, every time, independent of which model produced it or how the two scores happen to compare.