A Perfect Score,
A Broken Feature

Every benchmark run described in an earlier post held the model constant and varied the language or the task's difficulty. This run inverted that: same language, same task, same prompt, and the only thing allowed to vary was which model received it — the first time this repo actually ran the "comparing across models" idea it had only speculated about before.

The Task

A level-4-style domain, SavingsPocket — ownerId, accountId, label, ACTIVE on creation. If the linked Account is later suspended, the SavingsPocket must automatically become FROZEN; if closed, CLOSED. The reaction has to happen automatically when the Account's status changes — never through a direct API call on SavingsPocket itself. Both models got exactly the same rule and exactly the same entry point, implementations/nestjs/CLAUDE.md, run simultaneously in separate git worktrees so neither could see the other's work.

Both models produced a perfect harness score. Only one of them actually worked.

The Same Pattern, Wired Two Different Ways

Sonnet's implementation was independently reproduced end-to-end: suspending or closing a real Account through its real HTTP API actually flips the linked SavingsPocket to FROZEN or CLOSED, through the real Outbox → SQS → OutboxConsumer path. Haiku's implementation wired the identical architectural pattern — Integration Event subscription via EventHandlerRegistry, correctly even supporting the existing 1:N handler contract — and it was completely plausible on inspection. Nothing about the code itself looked wrong.

Notice also what Haiku's own self-report actually said: "event registrations are correct and handlers are properly wired." That's a true statement about the code's structure, and it is not a claim that the test run passed — Haiku never said that, because the tests never passed. The gap wasn't a model lying about its results; it was a model correctly describing structure while a reader could easily mistake that description for a claim about behavior.

Root Cause

Rerunning the E2E test Haiku itself had written showed all three assertions failing — the handler's own log line never even printed, meaning it was never invoked. Haiku's e2e test file didn't override NotificationService with a no-op stub the way every existing e2e test in this repo does (card.e2e-spec.ts, for instance). Instead it tried to make real SES delivery work through a LocalStack email-identity verification call — and that path never completed cleanly enough for the reaction to actually run.

Whether that specific choice was the exact failure mechanism or a symptom of a broader setup problem in Haiku's test wasn't chased any further, because the decisive finding — the reaction the task asked for measurably doesn't happen — was already independently confirmed. There was nothing more to prove.

Not a Docs Bug This Time

Several earlier runs in this benchmark series turned up real defects in this repo's own harness or docs — a stale build-artifact blind spot, evaluator files sharing the same false positive. This one is different: it's a mistake inside code Haiku itself wrote, not a gap in the shared harness, docs, or scaffolding. There was nothing to fix in the repo. Neither worktree was merged.