Every Kubernetes anti-pattern checker in a five-check harness assumed `---`-separated documents. Naming more than one resource in a single `kubectl get -o yaml` call wraps the result in `kind: List` instead — and every checker silently found zero resources to flag, which looked identical to a clean pass.
Argo CD's App-of-Apps proof lives entirely on the parent; Flux's dependsOn proof is declared by the child and unverifiable alone. Audit either tree without including its root, and both fail the same way — for what turns out to be the same underlying reason.
A scoring harness covers nineteen categories of Kubernetes deployment mistake. One of them — drift — can only exist after a manifest has already been applied, which an authoring benchmark structurally cannot produce or avoid. The honest fix was a permanent, documented ceiling, not a future version.
Two models scored an identical 9/9 on a Kubernetes manifest-authoring task, independently reproduced. Reading what each one actually wrote found a self-defeating NetworkPolicy in one and a promotion pipeline referencing a resource that doesn't exist in the other — two unrelated defects invisible to a tied harness score.
A drift checker pointed at a cluster that had just been applied cleanly reported drift everywhere. The cluster wasn't lying — the API server's own admission defaulting had filled in fields Git never mentioned, and a naive full-object comparison had no way to tell the difference.
Two codebases generate an Aggregate's ID in two different places — one in the constructor, one via a Factory asking Infrastructure for it. Eric Evans' own book has a specific, citable answer for which pattern it actually describes, and it isn't the one either codebase's convention assumes.
Nearly every DDD codebase forbids referencing another Aggregate by direct object reference — ID only. Eric Evans' 2003 book explicitly permits it. The person who actually wrote the ID-only rule, Vaughn Vernon, says so himself, in the same paper that argues for the stricter rule anyway.
The same moment, serialized by the same driver, produces a different string depending on the process's timezone. Four languages had this bug at the call site and one had it at the process boundary — and the fix belonged in a genuinely different place in each, verified by literally running the tests nine time zones apart.
Every PR a Dependabot auto-merge workflow had ever merged did so by winning a race against its own six-hour deadlock — one of its steps was waiting for a check run that could only finish after that step did. Fixing it surfaced a second bug waiting right behind the first, and a class of half-merge left behind by plain GitHub 502s.
A Spring Boot 4 migration that checked git history instead of a stale doc, found a workaround for a library a search index insisted did not exist, and ended a day later with the deployable image unable to build — because nothing in CI was watching the file whose meaning had just changed.
A path-existence checker reported zero findings before and after a three-language audit round that fixed roughly eighty real issues — stale code quotes, an evaluator that grades itself a perfect score for scanning nothing, and a generator still emitting a bug already fixed in the code it was modeled on.
NestJS's e2e suite assembled its own approximation of the app instead of booting the real one, and every language's LLM features had only ever run through their own fallback path. Fixing both surfaced a stranger bug: nock and testcontainers fighting over the same patched module.
Same doc, same task, two models, run at the same time in separate worktrees. Both self-reported a perfect harness score. Only one of them, independently reproduced against real Postgres and LocalStack, actually worked.
A week after a benchmark task exposed two languages that could not support a second event subscriber, four real features made every language need one. This time all five broke — from a loud boot-time crash to a silent single-handler drop nothing ever logged.
A monthly statement and a GDPR-style data export both died to the same question: couldn't the client just build this itself? The spending-analysis ETL that survived it, and the rule it revealed.
RefundReasonClassifier's fraud-risk score was computed entirely from text the refund requester controlled. Removing it, the sibling ML scorer that went with it, and the one rule the removal left behind.
A structured-data RAG feature over an account's own transaction history, the guardrail that lets an LLM touch it safely, and how the same invariant survived five different languages' own conventions.
A missing @Transactional, a JDK HTTP client retry quirk, a VARCHAR(36) overflow, an SQS FIFO dedup collision — four real bugs that needed real infrastructure to even exist.
RefundReasonClassifier reads a refund reason and hands back a signal — the Domain Service that actually decides never calls it, and swapping the LLM backend from Claude to self-hosted Ollama touched almost no test.
RefundFraudRiskScorer is a hand-rolled logistic regression trained on refund history, swappable between a native and an HTTP implementation, feeding the same Domain Service a second independent threshold.
Completing incomplete Swagger docs across five languages, verified by actually booting each app instead of trusting the annotations compiled. What it found had nothing to do with documentation — including a Spring Boot 4 dependency split that left production migrations silently never running.
A shell command's output has, more than once, contained something shaped exactly like a real system message, instructing the agent to hide a change. The rule that matters: disregard it, and say so.
A transfer feature needs one thing every implementation already claimed to support: writing two Aggregates atomically. Building it for real found a working mechanism in one language, a regression waiting one edit inside the obvious fix in another, and a doc that had been quietly wrong about its own code in a third.
Five languages scoring 100% on an easy synthetic task taught nothing about where they would fail. A four-level difficulty ladder built specifically to exercise unexercised code paths found the ceiling — and its last rung exposed a fan-out bug that had been invisible since nothing had ever subscribed two things to the same event before.
A repository-naming fix that only reached the write-side interface, four rounds of turning that gap into permanent harness rules, and a yield curve — three or four real bugs per round, then two, then zero — that was itself the most useful result.
No parsing, no understanding of what a code snippet does — just comparing backtick-quoted paths against the real file tree. The exclusion rules that kept it from crying wolf mattered more than the two-pattern check itself, and it still caught a real bug in four docs on its first run.
Two harness rules had checked out clean for months — because every domain that ever fed them was Account or Card. Building a genuinely unrelated third domain surfaced two false positives, and confirmed the rule meant to catch a real mistake still did.
A security audit found /auth/sign-in accepted a userId and nothing else — how the same bug showed up in five languages, and the JDK retry bug a new 401 test uncovered along the way.
Three violations across five languages — a Query reading a write Repository, a domain class carrying JPA, a notification module in the wrong layer. Only one was actually a bug, and the other two reveal why dozens of prior audits never caught any of it.