Observability Is a
Design Decision, Not an Afterthought

Observability tends to get treated as infrastructure you bolt on — pick a logging library, wire up a dashboard, done. The parts that actually matter are decisions, not tools: what gets logged at what level, which layer is responsible for logging what, and how a single request's story stays traceable once it's scattered across dozens of log lines from multiple processes.

Five Levels, Strictly Enforced

error is for request-handling failures and external system outages — a DB connection failure, an external API returning 5xx, an unhandled exception. warn is for normal operation that still needs attention — a call to a deprecated endpoint, a retry occurring, approaching a threshold. log covers key business events and state changes — an order created, a payment completed, the app starting or stopping. debug is detailed info for development — query parameters, intermediate computed results. verbose is maximum detail, like a full request/response payload.

Production emits only error, warn, and log. Development and staging emit everything. Unnecessary logging in production doesn't just cost money — it buries the important lines in noise exactly when you need to find them fastest.

Who Logs What

The Interface layer (a Controller) logs request errors, caught in a catch block. The Application layer logs business events and the results of external system calls. Infrastructure logs external-integration failures and retries, and abnormal query performance. The Domain layer never logs, full stop — it stays framework-independent, and the result of domain logic gets logged one layer up, in the Application layer that called it.

// forbidden — using a logger/framework in the Domain layer
import org.slf4j.Logger;          // forbidden
import org.slf4j.LoggerFactory;   // forbidden

public class Order {
    private static final Logger log = LoggerFactory.getLogger(Order.class);  // forbidden

    public void cancel(String reason) {
        log.info("Order cancelled");  // forbidden
        ...
    }
}

This isn't a purity rule for its own sake. A Domain layer that logs has taken a framework dependency it's supposed to have none of, and now every domain unit test has to either mock a logger or tolerate log noise it never asked for.

Structured Logs, and Why the Field Names Matter

When integrating with an external monitoring system — Datadog, CloudWatch, Grafana Loki — logs should be structured JSON, with field names in snake_case:

// a business-event log
log.info("Order created", kv("order_id", orderId), kv("user_id", userId), kv("amount", amount));

// an error log
log.error("SQS send failed", kv("event_id", event.getEventId()), e);

The reason for snake_case specifically, not camelCase, is unglamorous but concrete: most monitoring platforms parse snake_case fields by default. A field-name mismatch doesn't just look inconsistent — it silently breaks indexing, so a query that should find every log for a given order_id quietly returns nothing.

Correlation ID: Making One Request Traceable Across Everything

To trace a single request across multiple services in logs, every log entry includes a Correlation ID. If the client sends an x-correlation-id header, it's used as-is; otherwise the server generates one. The header is forwarded on every downstream call, and returned in the response too.

The ID is generated or extracted at the request entry point — the Interface layer, as a Servlet Filter — and propagated via SLF4J's MDC (Mapped Diagnostic Context), a ThreadLocal-backed map that the Logback JSON encoder reads automatically, so every later layer can read the current request's Correlation ID with no argument threaded through method signatures:

// at request entry — a Filter in the Interface layer
String correlationId = Optional.ofNullable(request.getHeader("X-Correlation-Id"))
        .orElseGet(() -> UUID.randomUUID().toString().replace("-", ""));
MDC.put("correlation_id", correlationId);
try {
    chain.doFilter(request, response);
} finally {
    MDC.remove("correlation_id");
}

// when logging, anywhere downstream — no argument needed, MDC is read automatically
log.info("Order created");

This is the exact same shape as the request-scoped user-context pattern used for authentication — a value generated once at the edge, read from storage everywhere else, with no request object passed around to get at it. The two problems (who is the current user, what request produced this log line) are different, but the mechanism that solves them is identical on purpose.

Metrics and Tracing: Directional, Not Mandated

This isn't tied to one specific stack, but a few things are worth alerting on regardless of which one you pick: the HTTP 5xx rate, p99 response time, DB connection pool saturation, message-queue DLQ depth greater than zero, and the queue's ApproximateAgeOfOldestMessage — a metric that catches a stalled consumer long before anyone notices requests are actually failing.

For tracing, OpenTelemetry auto-instrumentation collects HTTP, DB, and message-queue spans with minimal manual wiring. At an asynchronous boundary — a Task Queue, an Integration Event — including traceparent in the Outbox payload propagates the trace context across the gap, linking an HTTP request straight through to the event processing that happened seconds or minutes later, as a single trace instead of two disconnected ones. Including trace_id in log records lets you jump from a trace directly to its logs, which is usually the difference between "I can see something slowed down" and "I can see exactly which query slowed it down."

The Principle That Ties It Together

Always log the error in a catch block before rethrowing — never swallow an exception silently just because it's going to propagate anyway. A silently-swallowed exception and a correctly-rethrown one look identical to the caller; only the log tells you afterward that something went wrong at all.