Skill levelIntermediateApplies toDocker, Python, Node.js, any containerized serviceTime8 min
Your service crashes. You check the logs. The last line is from two minutes before the crash. Whatever happened at the moment of failure — gone.
This is not a mystery. This is buffering.
The Default Behavior
When a process writes to stdout in a terminal, it flushes after every newline. This is line buffering, and it is the behavior you see during development: you write a log line, it appears immediately in the terminal.
When the same process writes to stdout inside a Docker container, stdout is a pipe, not a terminal. Pipes get full buffering by default. The C runtime — and most language runtimes built on top of it — buffers output in 4 to 8 KB chunks and flushes only when the buffer fills, when you explicitly flush, or when the process exits normally.
If the process exits due to a crash — SIGSEGV, SIGKILL from the OOM killer, an unhandled exception — the buffer may never flush. The log lines you wrote in the last few seconds, describing exactly what went wrong, disappear with the process.
Python
Python buffers stdout by default when stdout is not a terminal. You have three options:
# Run with unbuffered flag
python3 -u your_script.py
# Set the environment variable
PYTHONUNBUFFERED=1 python3 your_script.py
# Or add to your Dockerfile
ENV PYTHONUNBUFFERED=1The Dockerfile approach is the right default for containerized services. Add it once and forget about it. FastAPI, Django, and Flask containers should all have this set. If yours does not, add it now.
The -u flag forces unbuffered binary mode for both stdout and stderr. The environment variable does the same thing. Both also affect the sys.stdin buffer, which rarely matters for services but is worth knowing.
Node.js
Node.js uses libuv rather than the C runtime stdio, and stdout is generally not buffered the same way. But there are two situations where you lose tail output:
First, if you are writing to a stream and not handling backpressure correctly, writes that exceed the internal high-water mark are buffered in memory. On fast exit, those buffered writes may not complete.
Second, most production Node.js logging libraries write asynchronously for performance. If your process exits before the async writes complete, those log records are lost. Ensure your SIGTERM handler flushes all transports before calling process.exit().
process.on('SIGTERM', async () => {
await logger.flush(); // depends on your logging library
process.exit(0);
});Docker Log Drivers
Once data reaches Docker's logging subsystem, a second buffering layer applies. The log driver you configure determines what happens next:
- json-file (the default): Writes to disk on the host. Survives process exit. Supports
docker logs. The right choice for most deployments. - syslog: Sends to the local syslog daemon. Typically fast. Syslog itself may buffer, but it is process-independent, so it survives container crashes.
- fluentd / awslogs / gelf: Send over the network. If the remote endpoint is slow or unreachable, Docker will drop log messages rather than blocking your container. You will lose logs. The default mode is blocking, which at least preserves data — but blocks container stdout writes when the log endpoint is slow, which is its own problem.
- none: Discards everything. Useful for high-volume services where you handle log shipping in-process, using a log rotation library writing to files on a bind-mounted volume.
For network log drivers, check the mode setting. The default is blocking, which preserves logs but adds latency to your container's stdout writes when the log endpoint is under pressure. The alternative is non-blocking with a max-buffer-size limit — but non-blocking silently drops the oldest messages when the buffer fills. Neither is free.
Recommended Configuration
For most small production deployments, json-file with rotation is correct:
# /etc/docker/daemon.json
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
}
}This caps log storage at 30 MB per container, retains the last 3 rotations, keeps everything local where docker logs can read it, and adds no network dependency to your logging path. Add a log shipper (Filebeat, Fluentd, Vector) that reads from disk if you need centralized log aggregation — the shipper reads from the json-file output, not from Docker's log driver directly, which avoids the network-failure-drops-logs problem.
Structured Logging
If you are writing structured JSON to stdout, ensure each log record is a single line. Multiline output is captured as separate log entries by most drivers, breaking JSON parsing in your aggregator. Use a logging library that formats records as single-line compact JSON.
Exception tracebacks are the common problem. A Python exception with a 10-line traceback produces 10 separate log entries in json-file format, none of which parse as complete JSON. Serialize the traceback to a string field within your log record, so the entire exception appears as a single JSON object on a single line.
What to Fix First
If logs disappear on crash: start with PYTHONUNBUFFERED=1 (or the equivalent for your runtime). That resolves the majority of cases in containerized Python services and takes ten seconds to add.
If logs disappear under load: check your log driver. Network drivers under load drop messages. Switch to json-file and add a log shipper that reads from disk.
If logs disappear at shutdown: add a SIGTERM handler that flushes all log transports before the process exits. Do not rely on the runtime's normal shutdown sequence to do this for you — under container orchestration, SIGKILL follows SIGTERM after a grace period, and a logger that defers flushing to the interpreter shutdown hook may not get there in time.
The buffering problem is not subtle. It is the default behavior of every stdio pipe in every Unix system. The fix is one environment variable and an understanding of which log driver you are actually using.
Building something real and want to track it publicly? builds.anethoth.com is a directory for indie SaaS founders building in the open — real metrics, real stages.