Can an AI Agent
Follow Your Architecture?

This repo already had an objective scorer sitting around unused for this purpose — the harness, which mechanically scores whether code follows documented architectural rules, with no human review needed. Repurposing it as an AI-agent benchmark turned out to need no new infrastructure at all, just a different way of framing the task.

The Task Format Is Deliberately Sparse

The prompt given to the agent contains exactly three things, and nothing about how to implement it: the business rules for a domain that doesn't exist in the repo yet; an instruction to follow the repo's existing conventions, plus the minimal entry point of where to start reading — that language's CLAUDE.md, with no other doc path given; and the completion criterion, which is simply to run the harness itself and iterate until it passes. Whether the agent finds the relevant docs on its own, by following the doc index instead of being handed a reading list, is itself part of what's being measured.

Run One: A Domain Nobody Had Ever Mentioned

The first real run gave an agent a Subscription domain — owner and plan name, PENDING on creation, a simple activate() transition, and a cancel(reason) deliberately described only as "other parts may need to react to it," with no further hint about what that meant technically. Nothing pointed the agent at the scaffolding generator or the reference template.

The agent followed the CLAUDE.md index on its own, discovered the scaffolding generator unprompted, generated a skeleton with it, then interpreted "other parts may need to react" as precisely what it needed to mean in this codebase — a Domain Event plus the Outbox pattern — implementing cancel() to publish an event while leaving activate() as a plain, event-free transition. Independent re-verification matched the self-report exactly: A (100/100, raw 630/630). Reading the actual domain code confirmed the business logic matched the spec too, not just the harness score.

Run Two: The Same Task, All Five Languages at Once

A Voucher domain — issue, redeem, expire, with the same "other parts may need to react" hint attached only to expire() — ran simultaneously across all five language implementations, each given only its own CLAUDE.md as the entry point.

Every language scored perfectly, and — more interesting than the perfect score itself — every one independently chose to attach a Domain Event only to expire(), never to redeem(), applying the same underlying pattern ("a transition nobody reacts to has no event; one something needs to react to does") that a single root doc had described once. One doc, five independent agents, one identical architectural judgment.

The Java mismatch is the more important result of this run. The independent re-verification disagreed with the self-report, and the cause wasn't the code at all — a stale Gradle build cache directory was being scanned by the harness as if it were real source, producing a false "no layer directory" positive. Deleting the build artifact and re-running matched the self-report exactly. This is precisely the scenario the "never trust the self-report" rule exists for: the discrepancy pointed at a real bug, just not the one anyone expected.

Raising the Difficulty, One Notch at a Time

A task where every language scores perfectly on the first try has no discriminating power — it can't tell you anything about where an agent (or a doc) might fail. Later runs deliberately added one new decision point each time.

Level 2 required a Domain Service coordinating two Aggregates within the same BC (a Booking and a Cancellation, where the cancellation is only valid if the original booking is confirmed and the requested count doesn't exceed the original) — with no mention of the one existing precedent for this shape anywhere in the prompt. Every language independently found that precedent and separated the judgment into a stateless Domain Service. Every one even caught a subtle spec distinction on its own: because the task said the invalid request itself must never be created, all five changed the Domain Service to throw immediately rather than return a rejection object to save — a different behavior from the existing precedent, correctly detected as different. NestJS was the first case across any run to not be perfect from the start, scoring 96 before self-correcting a real defect (a raw string thrown instead of the typed enum).

Level 3 needed a synchronous cross-BC lookup with no vocabulary hint at all — just the plain sentence that another BC's status had to be checked before allowing creation. All five picked the synchronous Adapter/ACL pattern, found the existing precedent, and kept the ACL discipline of never exposing the other BC's status enum directly, translating it into a boolean first. Independent verification caught another harness bug here too: three more evaluator files, beyond the one fixed in the earlier round, shared the exact same stale-build-artifact blind spot.

Level 4 flipped the axis entirely — an asynchronous reaction to another BC's event, deliberately designed to contrast with level 3's synchronous lookup. Every language correctly chose to subscribe to an Integration Event instead of adding another synchronous call, and every one proved it end-to-end by actually suspending an account through the real API and polling until the reaction happened. This run's most important result wasn't about the agents at all — it exposed that two of the five languages' Outbox consumers could only ever register one handler per event type, silently breaking the moment a second BC tried to subscribe to the same event. That's a real architectural gap the benchmark task found by touching a code path nothing had exercised before, not a contrived edge case.

Level 5: Combining Everything at Once

The most demanding run combined three previously-separate axes into one task — a recurring transfer that runs automatically on a monthly schedule (the same batch/Task Outbox pattern as interest payments), requires a Domain Service to judge eligibility (the same shape as the earlier Refund precedent), and must isolate one rule's failure from every other rule's processing.

Only three of five languages passed cleanly on the first submission. The other two had a perfect harness score and perfect unit tests, and still shipped a real bug that only an end-to-end test against real infrastructure caught: one saved a generated reference ID into a database column sized for a shorter format, failing on every retry scenario a full month later; the other's end-to-end test suite hit a FIFO queue's deduplication window across separate test methods within the same run, silently dropping every enqueue after the first and nearly producing a false-positive pass.

What This Is Actually Measuring

Every level above tested a different thing about the documentation, disguised as a test of the agent: whether a rule described once in a root doc produces the same judgment across five independent implementations and, by extension, across new engineers who've never seen this codebase before. A benchmark score that's identical across languages says the docs communicate the pattern with real precision. A benchmark run that finds a bug in the harness itself, or a structural gap the codebase never happened to exercise, is doing something a passing test suite alone can't — proving the specification is actually complete, not just self-consistent.