Tooling · Architecture

Zero Findings,
Eighty Bugs

The docs-drift checker that catches renamed and moved files reported zero findings — the same zero it always reports. A parallel audit across three languages the same week found roughly eighty real ones: stale code quotes, a check that gives itself full marks for scanning nothing, and a domain generator still emitting the exact bug a different round had already fixed in production weeks earlier. None of it was invisible by accident. Each one was invisible for its own specific, honest reason.

The docs-drift script is, by design, a path-existence checker wearing a code-review costume — it compares backtick-quoted paths against the real file tree and has no opinion on whether a quoted snippet still matches what the named file contains. It ran across NestJS, Go, and FastAPI both before and after a full audit round on those three languages, and reported the same thing both times: zero findings. In between those two runs, roughly eighty real issues got fixed. The gap between those two numbers isn't a bug in the checker. It's the checker doing exactly what it was built to do, at a scale nobody had tested it against before.

Code Quotes Rot Faster Than Version Numbers

Every concrete version string in every doc across all three languages — framework versions, base image tags, dependency pins — checked out clean. What had drifted, consistently, in all three, were doc blocks labeled with a real file path and presented as the actual code living there: bootstrap sequences, entity shapes, method signatures, module exports, each lagging the real file by a round or two of feature work. A reader checking "does this doc name a file that exists" — the only question the automated checker can ask — gets no signal at all. A reader who actually diffs the quoted snippet against what the named file contains today finds this constantly. The two questions sound similar. Only one of them was being asked.

A Check That Grades Its Own Homework

NestJS's dto-validation evaluator matched files named *.dto.ts — a naming pattern that doesn't exist anywhere in this repository's own convention, which spells out request-body and request-querystring DTOs differently. Every run scanned exactly zero files and reported a perfect 25 out of 25, unconditionally, for as long as the rule had existed. Separately, four other evaluators had quietly fallen out of the score-category breakdown entirely — 85 points present in the raw total but absent from any bucket a reader would actually look at. Neither of these shows up by reading the harness's own output; a perfect score doesn't announce which fraction of itself never actually ran.

The Generator With the Bug Its Own Codebase Had Already Fixed

An earlier round had fixed a real production bug: an event with two subscribers silently dropping one of them, in a specific shape unique to how FastAPI's own consumer dispatched handlers. The domain-scaffolding generator — the tool meant to produce new code in the repository's own house style — had never been told the shape changed. It still emitted the old, pre-fix wiring: a bare handler where the fixed code now expects a list. A domain generated from that template would silently swallow a second subscriber's failures behind a TypeError nobody would see unless they happened to generate a domain with two subscribers and actually run it. The static harness checks structure, not runtime behavior, so it had no way to tell the fixed shape from the broken one — they look identical on the page.

All Green, and the App Doesn't Boot

NestJS's real entity-registration list was missing one entity the running application actually needed to start. Every end-to-end spec, though, assembled its own hand-picked entity list rather than importing the real one — a shortcut that had been in place long enough that nobody remembered it meant the test suite never once booted the actual composition root. Full build green, full test suite green, harness green. The real application: does not start.

What all four share

None of these were invisible by accident — each had a specific, locatable reason a check that existed didn't see it: a tool built to check one layer (paths) while the bug lived in another (content); an evaluator that can self-report success without ever running against real input; a generator nobody re-runs after fixing the thing it generates; a composition root nothing in the test suite actually assembles. "The checks are green" is a claim about what got checked. It was never a claim about what's true.

Further reading in the repo

scripts/check_docs_drift.py — the checker, and exactly what it does and doesn't ask · A Path-Existence Checker Found a Real Bug on Day One — where this tool's stated limits were first written down