Tooling · Documentation

A Path-Existence Checker
Found a Real Bug on Day One

A heuristic script that does nothing but compare backtick-quoted paths in the docs against the real file tree (no parsing, no understanding of what the code inside a snippet does) caught a real bug in four Kotlin docs on its first run. The design decisions that kept it from crying wolf ended up mattering more than the check itself.

A check that only asks whether a quoted file path exists sounds too dumb to be worth running. I built one anyway, for a specific reason. I had just closed a batch of doc/code gap issues in my example project, which implements the same backend design in five languages side by side, and every one of them had been found the same way: a human or an agent reading a doc, separately reading the code it described, and noticing they'd stopped agreeing. That's a repeatable pattern, and repeatable patterns are worth automating even in a cheap, dumb form.

Cheap and Dumb, on Purpose

scripts/check_docs_drift.py checks two things, both pure string matching against the file tree, with zero understanding of what any code does:

  • STALE-ABSENCE — the doc says something doesn't exist yet, and it actually does.
  • PHANTOM-PRESENCE — the doc labels a snippet "actual code" and names a real-looking path, and no such file exists.

That's the entire detection surface. It doesn't parse the snippet, doesn't diff it against the named file, and has no opinion on whether the code shown is what the file contains. It only checks whether the path named next to it exists. The honest way to describe it is a path-existence checker wearing a code-review costume.

The Interesting Part Was All in the Exceptions

A checker this literal is mostly a false-positive generator until it's taught what not to flag, and each exclusion came from testing against the docs and finding a specific way the naive version was wrong:

  • A code-block header that says "to add," "proposed," or "target shape" is never read as STALE-ABSENCE. In these docs that phrasing overwhelmingly means "add this to a file that already exists," and testing against files that are always present (build.gradle, main.go, application.yml) showed reading it as "doesn't exist" was wrong every single time.
  • A qualified reference like pkg.path.TypeName is recognized by capitalization. If the segment after the last dot starts uppercase, it's a type reference, not a file path, and gets skipped.
  • Any path or header containing ... is skipped outright, since it's an elision, not a path.
  • .md cross-references between docs are excluded from the existence check entirely, since a doc-to-doc link is essentially always present and checking it proves nothing.

None of these are clever. All of them came from a false positive first, then got written down as a rule.

What It Found on Day One

Wired into CI to run on every push touching **/*.md or implementations/*/examples/**, the first run caught something worth catching. Four Kotlin docs (config.md, module-pattern.md, observability.md, secret-manager.md) each cited a code block's header as notification/infrastructure/X.kt. The real path was account/infrastructure/notification/X.kt: the domain prefix was missing and the two segments were in the wrong order. Four docs, the same wrong path, fixed in the same commit that shipped the checker.

What it deliberately doesn't check

Prose that describes a gap without a backtick-quoted path ("the app service isn't in compose" written as a sentence, no path) is invisible to it. So is everything inside a snippet: whether the code shown still matches what the named file contains is a question this tool has no way to ask. It knows one thing, whether the path exists, and answers only that.

Why the Cheap Version Still Earns Its Keep

Most doc drift in a codebase like this isn't "the logic subtly changed and the doc's explanation is now wrong." It's "the file moved, or was renamed, and the one line naming it in a doc never got updated." That's a mechanical mistake, and a mechanical check catches it without needing to understand a single line of the code it's checking. The engineering effort here went almost entirely into the four exclusion rules, not the two detection rules. Teaching a literal-minded script what to ignore turned out to be the part that decided whether anyone would trust its output.

Further reading

scripts/check_docs_drift.py (the full checker, under 250 lines, in my example project that implements the same backend design in five languages) · docs/docs-drift-check.md (what it checks and what it deliberately doesn't)