Designing File Upload APIs That Don't Become Security Holes

File upload endpoints look like a solved problem from the outside. The framework gives you a multipart parser, the browser gives you an input element, the storage backend gives you a put-object call, and the obvious code path connects them. The obvious code path is also responsible for an outsized share of production incidents in any web service that accepts user-supplied files. The bugs hide in the gap between what the framework defaults give you and what a real adversary can send.

This post is the longer version of "what does a file upload endpoint actually need to do" — the validation that matters, the storage isolation that prevents pivoting, the streaming pattern that survives large files, and the failure modes that show up in production a year after the endpoint shipped. We have shipped four products that accept user input, and the file-upload-shaped subsets are the ones that need the most defensive engineering.

What naive uploads look like

The framework default for a file upload typically does three things: it parses the multipart body into memory or a temporary file, it gives your handler a reference to the file content, and it lets you write that content to your storage backend with one more line of code. None of these defaults are wrong individually. The collection of them produces an endpoint that accepts arbitrary content with arbitrary filename and arbitrary size and writes it to a path the user can influence. Each adjective in that sentence is a separate bug class.

The fix is not a magic decorator. It is a small list of explicit decisions made before the upload reaches storage, each of which closes one bug class. None of them are exotic and none of them are defaults.

Size limits before parsing

The first decision is the maximum upload size, and it has to be enforced before the multipart body is parsed. Frameworks that enforce size limits inside the parser have already allocated the memory or written the temporary file by the time they reject — at scale this is a denial-of-service primitive where an attacker sends concurrent oversized uploads to exhaust disk or RAM before the server sees them.

The right enforcement layer is the reverse proxy. Caddy's request_body { max_size }, nginx's client_max_body_size, and the equivalents in HAProxy and Envoy reject oversized bodies at the connection layer before the application sees a single byte. Set this limit per route — your invoice generation endpoint accepts five-megabyte JSON bodies, your file upload endpoint accepts fifty-megabyte multipart bodies, and the limits are documented in the same Caddyfile that wires the routes.

The application then enforces a second size limit after parsing, defensively, to handle the case where the proxy was bypassed or the limit was misconfigured. Defense in depth here costs nothing and catches real bugs.

Content-type validation that does not trust the client

The Content-Type header on a multipart part is supplied by the client and means nothing. The filename extension is supplied by the client and means nothing. The only way to know what kind of file you actually have is to read the first few bytes and compare them to known magic numbers — PNG starts with 89 50 4E 47, PDF starts with 25 50 44 46, JPEG starts with FF D8 FF, and so on.

The library to use is whatever your language's magic binding is — libmagic in C, python-magic in Python, file-type in Node, mime/multipart with explicit sniffing in Go. Build an allowlist of magic numbers your application accepts, reject everything else with a structured error, and never trust the client-supplied metadata for anything other than display.

The allowlist matters more than it sounds. "Accept any image" is not a security-meaningful constraint — SVG is an image and is also an HTML document with script execution. PDF is a common allowlist member and contains JavaScript. The right discipline is to enumerate the specific formats your application can actually do something with, validate against that list, and add new formats explicitly when product requirements expand.

Filename sanitization without losing the original

The user-supplied filename is for display purposes only. The storage path must be a server-generated identifier that does not include any user-supplied content. This closes path traversal, filename collision, case-sensitivity confusion across filesystems, and the long tail of bugs caused by filenames containing characters the storage backend treats specially.

The pattern that works is to store the file under a UUID-based path, store the original filename in a database row alongside the UUID, and serve downloads with a Content-Disposition header that supplies the original filename for the user's browser. This separates storage concerns from display concerns cleanly.

Storage isolation: where the bytes actually live

Uploaded files should never be written to a directory that the web server serves directly. The pattern that has caused the most real incidents is "files saved to /var/www/html/uploads/ and the web server serves them with whatever Content-Type the framework guessed." This is how PHP-with-uploaded-shells worked for fifteen years and how a depressing number of modern systems still get popped.

The right pattern is to store uploads in a directory that is not part of any web server's document root, serve them through an explicit application route that sets a safe Content-Type and Content-Disposition, and never execute uploaded content under any circumstances. For object storage backends like S3 or R2, configure the bucket to deny public access and serve through signed URLs or a server-side proxy. The bytes never live in a place where a misconfiguration turns them into executable code.

Streaming for large files

Loading the entire file into memory before processing it is fine for small uploads and disastrous for large ones. The right pattern for files over a few megabytes is to stream the upload directly to storage as it arrives, with validation happening on a streaming basis. WeasyPrint and most PDF tools work on a file path, so the streaming target is a temporary file with a guaranteed-cleanup wrapper. Object storage backends accept multipart uploads natively — initiate, write parts as they arrive, complete or abort on the final byte.

The streaming pattern also lets you enforce the size limit incrementally. Count bytes as they arrive, abort the connection if you exceed the limit. This handles the case where the client lies in the Content-Length header.

Malware scanning when it earns its weight

If your service stores user-uploaded files that other users will download, you have an implicit obligation to do something about malware. The honest options are: scan with ClamAV or a commercial equivalent at upload time, scan asynchronously after upload with quarantine semantics, or rely on browser sandboxing for downloads and accept that some uploads will be malicious.

The first two cost real CPU and add latency. The third is honest about the limits of server-side scanning — modern malware specifically targets the gap between "scanner says clean" and "browser executes anyway" — and shifts the threat model to "make sure downloads are clearly marked as user-supplied content."

For developer-tool APIs where uploads are private to the customer and not redistributed, malware scanning is usually overengineering. For consumer-facing services with social distribution, it is table stakes.

The five tests that find the bugs

The bugs in file upload endpoints hide in the failure modes manual testing does not exercise. The tests that catch them are: (1) upload with the wrong magic number for the claimed Content-Type — does the server detect the mismatch? (2) upload with a filename containing path traversal characters — does the storage path stay sanitized? (3) upload at exactly the size limit, one byte over, and ten times over — does each get the right error? (4) upload with a connection that drops mid-stream — is the partial file cleaned up? (5) upload concurrent duplicates of the same file — does deduplication work or does each upload create an independent storage entry?

Each of these is a one-time test that takes ten minutes to write and prevents a real production incident. The cost of writing them is much smaller than the cost of explaining to a customer why their corrupted file ended up referenced by another customer's record.

Where this matters across our products

File uploads appear in different forms across our four products. DocuMint accepts logo uploads for invoice templates with image-format validation and per-customer storage isolation. CronPing accepts no file uploads on customer endpoints. FlagBit accepts CSV uploads for bulk targeting rule import with format validation and async processing. WebhookVault accepts arbitrary HTTP request bodies including file uploads as the entire point of the service, treats them as opaque bytes, and isolates them per endpoint with explicit retention windows. Each product treats the upload surface differently because the threat model differs, and that is the right discipline. The pattern that does not work is to apply identical defaults across surfaces with different requirements.