ETL · Architecture

Not Every Report
Needs a Server

The request was to add an ETL feature. Every proposal that followed died to the same one-line question — until one didn't, and the reason it survived is the actual rule.

"Add a data ETL feature" is an easy request to say yes to and a surprisingly hard one to fill in. The obvious moves — a monthly account statement as a CSV, a GDPR-style "download all my data" export — both sounded like real backend work. Both died to the same question, asked plainly: couldn't the client just call the existing read endpoints and build that file itself?

The Question That Kept Killing Ideas

For the monthly statement: the account's transaction history is already fully queryable through GetTransactions. A client fetching a month's worth of rows and rendering a CSV is client work, not backend work — the server doing it instead is a convenience, not a necessity. For the data export: same shape, larger scope. Aggregating Account+Card+Payment+Refund into one file is more tedious for a client to build, but "tedious" and "impossible" aren't the same claim, and only one of them justifies putting it on the server.

Two proposals, two honest admissions that the server wasn't actually required. That's a good sign the wrong question was being asked — not "can the server do this," but "does the client have some real reason it can't."

The Line That Actually Matters

A server-side job earns its place for one of a few real reasons, not because it happens to be possible:

  • The client can't reach the underlying data at all — it belongs to other users, or to nobody in particular (an internal ops report, a settlement file consumed by an external system with no human client in the loop).
  • Delivery has to be push, not pull — something has to happen on a schedule whether or not anyone asks for it.
  • The value is in precomputing an aggregate a client would otherwise have to re-derive from potentially many raw rows on every request — not in producing a file, but in not repeating expensive work.

Neither statement nor export cleared any of these. A monthly spending-pattern analysis — total/average withdrawal, month-over-month %-change, a trend label — did, on the third reason: computing that from raw transactions is exactly the kind of aggregation nobody wants running live, on every request, for every account.

What Survived, and Why

The whole feature is: a Cron enqueues a Task on the 1st of the month; the Task paginates every active account, aggregates last month's (and the month before's) withdrawals, and writes one precomputed row per account per month. A new query endpoint serves that row directly — no live aggregation, ever, on the read path:

// domain/spending-analysis.ts — the one real "transform" step
public static create(params: {
  accountId: string
  analysisMonth: string
  totalAmount: number
  transactionCount: number
  previousTotalAmount: number
}): SpendingAnalysis {
  const averageAmount = params.transactionCount > 0
    ? Math.round(params.totalAmount / params.transactionCount) : 0

  const changeFromPreviousMonth = params.previousTotalAmount === 0
    ? (params.totalAmount === 0 ? 0 : 100)
    : Math.round(((params.totalAmount - params.previousTotalAmount) / params.previousTotalAmount) * 100)

  let trend: SpendingTrend = 'STABLE'
  if (changeFromPreviousMonth > 10) trend = 'INCREASING'
  else if (changeFromPreviousMonth < -10) trend = 'DECREASING'

  return new SpendingAnalysis({ accountId: params.accountId, analysisMonth: params.analysisMonth,
    totalAmount: params.totalAmount, transactionCount: params.transactionCount,
    averageAmount, changeFromPreviousMonth, trend })
}

That's the entire "T" in ETL — two numbers in, a percentage and a label out. Extract is the existing per-account transaction table; Load is one upsert-shaped row, idempotent via a (accountId, month) unique constraint, the same two-layer pattern the repo's card-statement job already used. Nothing here needed inventing — the win was recognizing that a CQRS read-model, expressed as a batch job, was the shape that survived the question the report ideas didn't.

Real numbers from the actual test

The e2e test backdates two withdrawals — 30,000 and 20,000 — into "last month," runs the real scheduler, and reads back a row with totalAmount: 50000, transactionCount: 2, averageAmount: 25000, and — since there's no prior-prior-month history to compare against — changeFromPreviousMonth: 100, trend: 'INCREASING'. Re-running the same month's job a second time doesn't produce a second row.

The Rule, Stated Plainly

"Can the server do this" is nearly always yes. The question that actually filters ideas is narrower: does the client have a real reason — not a convenience one — that it can't do this itself? Most report-shaped requests fail that question quietly, because report-shaped is UI work wearing a backend costume. The one that survives is usually the one that isn't shaped like a report at all.

Further reading

Scheduling and the Task Outbox Pattern — the Cron→Task Queue infrastructure this feature reuses without needing anything new · CQRS in Practice — the Query-side discipline this feature's read model has to answer to