Logging That Actually Helps in Production
Most application logs are noise pretending to be signal. Here is what we actually log across four production APIs — and what we deliberately do not.
Logging is one of those topics where the gap between common practice and useful practice is enormous. Open most production codebases and you will find logs like "Entering function processOrder", "Got result", "Done". None of these tell you anything when the system is actually broken at three in the morning.
After running four production APIs for a few months, the logging patterns that have actually paid off look almost nothing like the patterns I used to write. Here is what we keep, what we cut, and why.
Log lines are answers to questions you have not asked yet
The right mental model for a log line is: at some future moment, someone will be debugging a problem and reading this. What do they need to know? Not "what did the code just do" — that is what stack traces are for — but "what was the situation."
The situation is request ID, user/tenant ID, the high-level operation, the inputs that mattered, the outcome, and the timing. Everything else is decoration.
The 10-field schema we actually use
Every log line at our INFO level has the same shape. Not "should have" — actually does, enforced by the logger:
ts— ISO 8601 UTC with millisecondslevel— DEBUG, INFO, WARN, ERRORservice— which API (documint, cronping, flagbit, webhookvault)request_id— UUID per request, propagated to background jobstenant_id— the API key owner (hashed, never the key itself)route— the matched route, not the raw path (so/v1/invoices/:id, not/v1/invoices/abc123)method— HTTP verbstatus— HTTP status codelatency_ms— server-side processing timemsg— short human-readable summary, present-tense verb
If the line is about an error, it gets error (the message), error_class (the type), and error_id (a short hash you can give to a user without leaking detail). Stack traces only at ERROR level — they explode log volume otherwise.
What we deliberately do not log
Request bodies. They contain credentials, PII, and giant base64 blobs. We log a one-line summary instead — for a webhook, the event type and source; for an invoice, the line item count and total; for a flag evaluation, the flag key and result.
Response bodies. Same reasons.
Headers other than safe ones. We allowlist a small set: User-Agent, Content-Type, X-Request-Id. Everything else, including Authorization and Cookie, is dropped at the logger boundary, not at the field level. Drop at the field level and one developer somewhere will accidentally bypass it.
Routine successes at high cardinality. We do not log "200 OK" for every health check ping or every status badge fetch — those would be 90% of our log volume and they tell us nothing.
The request ID middleware trick
The single highest-leverage piece of logging infrastructure is a request ID middleware that runs first. It generates a UUID, stashes it in a context variable accessible to every subsequent handler and every log call, and emits it in the response X-Request-Id header.
Now when a customer reports a bug with a screenshot, the request ID is right there in the response headers. You grep your logs for that one ID and see every line of activity for that request, across services, in order. This single primitive collapses most production debugging from hours to minutes.
For background jobs, propagate the request ID into the job payload. The job logs include the ID of the request that scheduled it. The full chain stays traceable.
Log levels are about audience, not severity
Common confusion: people use INFO for "important" and DEBUG for "less important." That is not what they mean.
INFO is for things that should be visible in production at all times — request boundaries, lifecycle events, security-relevant decisions. If you would not want it scrolling through the production stream, it is not INFO.
DEBUG is for things that are useful while debugging but not worth the volume during normal operation. In production we run at INFO. When investigating, we flip a single tenant or a single route to DEBUG.
WARN is for things that are unusual but recoverable — a retried API call, a fallback path triggered, a slow query. WARN should be rare; if it is firing on every request, it should be DEBUG or it should be ERROR.
ERROR is for things that need a human to look. Not "we returned 400 because the user sent invalid input" — that is INFO. ERROR is for "we returned 500 because we cannot reach the database." If your ERROR rate is non-zero in steady state, your alerting is broken or your usage of ERROR is wrong.
Log volume is a leading indicator of confusion
If your log lines per request creep upward over months, that is a signal — usually that someone added logs while debugging and never removed them. Schedule a quarterly review where you sample a few requests and ask: which of these lines did I look at the last time I debugged? The unread ones are candidates to delete.
The opposite mistake is also real: services where you cannot tell what happened from the logs because everything is summarized to a single line. There is a happy medium, and it is closer to "five to fifteen lines per request" than to "one line" or "two hundred lines."
Structured beats prose, every time
Logs as JSON, not as prose. {"event": "stripe_webhook_received", "type": "invoice.paid", "amount": 1900} is queryable. "Got Stripe webhook of type invoice.paid for $19.00" is grep-able and that is all.
The cost of structured logging is small (one wrapper around your logger) and the payoff is massive: log aggregators can index every field, you can compute SLO metrics directly from logs without instrumenting separately, and you can answer questions like "what is the p99 latency for write requests from tenants on the starter plan" with a query rather than a code change.
What we run in production
Across DocuMint, CronPing, FlagBit, and WebhookVault, the logging stack is unromantic: stdout JSON, captured by Docker, shipped to a log aggregator. No agent, no sidecar, no sampling, no tracing system. Sub-million requests per day.
We will reach for distributed tracing when we have multiple services per request that can fail independently. Until then, the request ID and structured logs do everything we need at a fraction of the operational cost.
Logging is a product feature
The logs that survive contact with a real outage are the ones that were designed for the moment of debugging, not the moment of writing. The test is simple: when something breaks, can you tell what happened from the logs alone, without re-running the request?
If yes, the logs are working. If no, they are decoration.