The Fraud Signal
That Trusted the Fraudster

Before a signal goes into a security- or money-relevant judgment, ask who controls its input. In my example project that implements the same backend design in five languages side by side, a refund decision had two fraud signals feeding it: RefundReasonClassifier, an LLM reading the reason the requester typed, and RefundFraudRiskScorer, a model scoring the requester's history. Both were cleanly layered Technical Services, and both worked, right up until a design review turned up a channel problem that no amount of clean layering could paper over. Both are gone now. This is what was wrong, what got cut, and the one rule the removal was for.

What It Did

RefundReasonClassifier was a Technical Service: an Application-layer interface, with the implementation (a self-hosted Ollama model, qwen2.5:1.5b) living in Infrastructure. It read a refund's free-text reason and returned a category plus a fraud-risk score from 0 to 1. That score fed RefundEligibilityService, a Domain Service, which rejected the refund outright once the score crossed a threshold:

// domain/refund-eligibility-service.ts — the branch that got removed
if (classification.category === 'fraud_suspected'
    && classification.fraudRiskScore >= 0.7) {
  return {
    approved: false,
    reason: 'This refund reason was flagged as high '
      + 'fraud risk and requires manual review.'
  }
}

On paper, this is a textbook Technical Service: an LLM call abstracted behind an interface, feeding a plain judgment into a Domain Service that never knew an LLM was involved, tested with a mocked classifier and no network call. Everything about the layering was accurate. The layering was never the problem.

The Channel Problem

The reason field is only what the person requesting the refund typed. If someone intends to defraud the system, they control the one input the fraud judgment depends on. Nothing stops them from writing "the item arrived damaged" instead of the truth. The classifier has no way to tell the difference, because there is no difference visible to it. It's the equivalent of verifying a sworn statement by re-reading the statement.

A judgment meant to catch bad-faith actors was built entirely out of a channel bad-faith actors fully control. The one case it was designed to catch is the one case guaranteed to sail through it.

The Cut, and How Far It Went

The fix was to remove RefundReasonClassifier outright, across all five language implementations. A second signal sat right next to it, RefundFraudRiskScorer, an ML model scoring the requester's own refund/payment history (frequency, amount ratio, time since payment). That input isn't something a requester can rewrite on a whim, so it doesn't share the flaw above. It stayed, at first.

Then the decision changed partway through: cut that one too, as a separate simplification call rather than because it shared the flaw. Five languages, two removals each, each one independently re-verified (build, lint, unit tests, e2e tests, the architecture checker, a docs-drift checker over the whole project) rather than assumed correct by analogy to the others. A fix that's obviously right in one codebase still has to prove itself again in every other implementation carrying the same logic.

What's left of RefundEligibilityService is two structural checks and nothing else:

// domain/refund-eligibility-service.ts — everything that's left,
// no fraud judgment of any kind
public evaluate(payment: Payment, refund: Refund): RefundDecision {
  if (payment.status !== PaymentStatus.COMPLETED) {
    return { approved: false, reason: '...only be requested for a completed payment.' }
  }
  if (refund.amount > payment.amount) {
    return { approved: false, reason: '...cannot exceed the payment amount.' }
  }
  return { approved: true }
}

The shared Python microservice the ML scorer's HTTP variant called was deleted last, once nothing referenced it anymore. No orphaned service was left running for its own sake.

The Principle

Removing a feature is only useful if it leaves behind a rule that outlives it:

An LLM may narrow what an authorized user sees. It must never decide who is authorized — or approve/reject a security- or money-relevant action — when its input is free text the affected party can shape.

Narrow versus decide is the whole distinction. An LLM filtering a list, summarizing a document, or ranking search results produces a worse answer when it's wrong. An LLM approving a refund, granting access, or flagging fraud produces a wrong outcome when it's wrong. And if the party who benefits from that wrong outcome is also the one who supplied the input, "wrong" becomes "exploitable."