Tooling · Architecture
Zero Findings,
Eighty Bugs
A check that reports zero findings has told you what it looked at, not what is there. A docs-drift checker built to catch renamed and moved files reported zero, the same zero it always reports. A parallel audit across three languages the same week found roughly eighty real issues: stale code quotes, a check that gives itself full marks for scanning nothing, and a domain generator still emitting a bug already fixed in production weeks earlier. None of it was invisible by accident. Each one was invisible for its own specific, honest reason.
The docs-drift script is, by design, a path-existence checker wearing a code-review costume. It compares backtick-quoted paths against the real file tree and has no opinion on whether a quoted snippet still matches what the named file contains. I ran it on my example project, which implements the same backend design in five languages side by side, across the NestJS, Go, and FastAPI implementations both before and after a full audit of those three, and it reported the same thing both times: zero findings. In between those two runs, roughly eighty real issues got fixed. The gap between those two numbers isn't a bug in the checker. It's the checker doing what it was built to do, at a scale nobody had tested it against before.
Code Quotes Rot Faster Than Version Numbers
Every concrete version string in every doc across all three languages (framework versions, base image tags, dependency pins) checked out clean. What had drifted, consistently, in all three, were doc blocks labeled with a real file path and presented as the actual code living there: bootstrap sequences, entity shapes, method signatures, module exports, each one or two feature changes behind the real file. A reader checking only "does this doc name a file that exists," the one question the automated checker can ask, gets no signal at all. A reader who diffs the quoted snippet against what the named file contains today finds this constantly. The two questions sound similar. Only one of them was being asked.
A Check That Grades Its Own Homework
NestJS's architecture checker (a script that statically scores code against the documented rules) has a dto-validation evaluator. It matched files named *.dto.ts, a naming pattern that doesn't exist anywhere in the codebase's own convention, which spells out request-body and request-querystring DTOs differently. Every run scanned zero files and reported a perfect 25 out of 25, unconditionally, for as long as the rule had existed. Separately, four other evaluators had fallen out of the score-category breakdown entirely: 85 points present in the raw total but absent from any bucket a reader would look at. Neither of these shows up by reading the checker's own output; a perfect score doesn't announce which fraction of itself never ran.
The Generator With the Bug Its Own Codebase Had Already Fixed
A real production bug had been fixed weeks earlier: an event with two subscribers silently dropping one of them, in a specific shape unique to how FastAPI's own consumer dispatched handlers. The domain-scaffolding generator, a script meant to produce a skeleton domain in the codebase's own house style, had never been told the shape changed. It still emitted the old, pre-fix wiring: a bare handler where the fixed code now expects a list. A domain generated from that template would swallow a second subscriber's failures behind a TypeError nobody would see unless they happened to generate a domain with two subscribers and run it. The static architecture checker looks at structure, not runtime behavior, so it had no way to tell the fixed shape from the broken one; they look identical on the page.
All Green, and the App Doesn't Boot
NestJS's real entity-registration list was missing one entity the running application needed to start. Every end-to-end spec, though, assembled its own hand-picked entity list rather than importing the real one. It was a shortcut that had been in place long enough that nobody remembered it meant the test suite never once booted the actual composition root. Full build green, full test suite green, architecture check green. The real application: does not start.
None of these were invisible by accident. Each had a specific, locatable reason a check that existed didn't see it: a tool built to check one layer (paths) while the bug lived in another (content); an evaluator that can self-report success without ever running against real input; a generator nobody re-runs after fixing the thing it generates; a composition root nothing in the test suite assembles. "The checks are green" is a claim about what got checked. It was never a claim about what's true.
scripts/check_docs_drift.py (the docs-drift checker, and exactly what it does and doesn't ask, in my example project that implements the same backend design in five languages) · A Path-Existence Checker Found a Real Bug on Day One (how the same checker was designed, and the limits it states up front)