Can an AI Agent
Follow Your Architecture?

Measuring that takes three decisions, and each is easy to get wrong. The task has to be sparse, so that finding the conventions is part of what gets tested. The score has to come from the person running the test, not from the agent's own report. And the difficulty has to rise one decision at a time, because a task every agent passes tells you nothing. I tried this on my example project, which implements the same backend design in five languages side by side. It already had an objective scorer that had never been used this way: an architecture checker that mechanically scores whether code follows the documented rules, with no human review needed. Turning it into a test for AI agents needed no new infrastructure at all, just a different way of framing the task.

Keep the Task Sparse

The prompt given to the agent contains three things, and nothing about how to implement it. First, the business rules for a domain that doesn't exist in the code yet. Second, an instruction to follow the existing conventions, plus the minimal entry point of where to start reading (that language's CLAUDE.md, with no other doc path given). Third, the completion criterion, which is simply to run the checker itself and iterate until it passes. Whether the agent finds the relevant docs on its own, by following the doc index instead of being handed a reading list, is itself part of what's being measured.

Does It Find the Conventions Unprompted?

The first real test gave an agent a Subscription domain: owner and plan name, PENDING on creation, a simple activate() transition, and a cancel(reason) deliberately described only as "other parts may need to react to it," with no further hint about what that meant technically. Nothing pointed the agent at the reference template, or at the script that generates a skeleton domain from a name.

The agent followed the CLAUDE.md index on its own, discovered that generator script unprompted, generated a skeleton with it, then interpreted "other parts may need to react" as precisely what it needed to mean in this codebase (a Domain Event plus the Outbox pattern), implementing cancel() to publish an event while leaving activate() as a plain, event-free transition. Independent re-verification matched the self-report: A (100/100, raw 630/630). Reading the domain code confirmed the business logic matched the spec too, not just the checker's score.

Do Independent Agents Make the Same Call?

A Voucher domain (issue, redeem, expire, with the same "other parts may need to react" hint attached only to expire()) ran simultaneously across all five implementations, one agent per language, each given only its own CLAUDE.md as the entry point.

Every language scored perfectly, and, more interesting than the perfect score itself, every one independently chose to attach a Domain Event only to expire(), never to redeem(), applying the same underlying pattern ("a transition nobody reacts to has no event; one something needs to react to does") that a single root doc had described once. One doc, five independent agents, one identical architectural judgment.

The Java mismatch is the more important result of this test. The independent re-verification disagreed with the self-report, and the cause wasn't the code at all. A stale Gradle build cache directory was being scanned by the checker as if it were real source, producing a false "no layer directory" positive. Deleting the build artifact and re-running matched the self-report. This is the scenario the "never trust the self-report" rule exists for: the discrepancy pointed at a real bug, just not the one anyone expected.

Raising the Difficulty, One Notch at a Time

A task where every language scores perfectly on the first try has no discriminating power. It can't tell you anything about where an agent (or a doc) might fail. Each later task deliberately added one new decision point.

A rule spanning two Aggregates. The next task required a Domain Service coordinating two Aggregates within the same BC (a Booking and a Cancellation, where the cancellation is only valid if the original booking is confirmed and the requested count doesn't exceed the original), with no mention of the one existing precedent for this shape anywhere in the prompt. Every language independently found that precedent and separated the judgment into a stateless Domain Service. Every one even caught a subtle spec distinction on its own: because the task said the invalid request itself must never be created, all five changed the Domain Service to throw immediately rather than return a rejection object to save, a different behavior from the existing precedent, correctly detected as different. NestJS was the first submission in any of these tests that wasn't perfect from the start, scoring 96 before self-correcting a real defect (a raw string thrown instead of the typed enum).

A synchronous lookup across a boundary. The next one needed a synchronous cross-BC lookup with no vocabulary hint at all, just the plain sentence that another BC's status had to be checked before allowing creation. All five picked the synchronous Adapter/ACL pattern, found the existing precedent, and kept the ACL discipline of never exposing the other BC's status enum directly, translating it into a boolean first. Independent verification caught another checker bug here too: three more evaluator files, beyond the one fixed earlier, shared the same stale-build-artifact blind spot.

An asynchronous reaction. The task after that flipped the axis entirely: an asynchronous reaction to another BC's event, deliberately designed to contrast with the synchronous lookup before it. Every language correctly chose to subscribe to an Integration Event instead of adding another synchronous call, and every one proved it end-to-end by suspending an account through the real API and polling until the reaction happened. This task's most important result wasn't about the agents at all. It exposed that two of the five languages' Outbox consumers could only ever register one handler per event type, breaking without an error the moment a second BC tried to subscribe to the same event. That's a real architectural gap the task found by touching a code path nothing had exercised before, not a contrived edge case.

Combining Every Decision at Once

The most demanding task combined three previously-separate axes into one task: a recurring transfer that runs automatically on a monthly schedule (the same batch/Task Outbox pattern as interest payments), requires a Domain Service to judge eligibility (the same shape as the earlier Refund precedent), and must isolate one rule's failure from every other rule's processing.

Only three of five languages passed cleanly on the first submission. The other two had a perfect checker score and perfect unit tests, and still shipped a real bug that only an end-to-end test against real infrastructure caught: one saved a generated reference ID into a database column sized for a shorter format, failing on every retry scenario a full month later; the other's end-to-end test suite hit a FIFO queue's deduplication window across separate test methods within the same run, dropping every enqueue after the first and nearly producing a false-positive pass.

What This Is Measuring

Every step above tested a different thing about the documentation, disguised as a test of the agent: whether a rule described once in a root doc produces the same judgment across five independent implementations and, by extension, across new engineers who've never seen this codebase before. A score that's identical across languages says the docs communicate the pattern with real precision. A test run that finds a bug in the checker itself, or a structural gap the codebase never happened to exercise, is doing something a passing test suite alone can't: proving the specification is complete, not just self-consistent.