Designing API File Uploads: Resumable Uploads, Deduplication, and the Patterns That Work

The naive file upload endpoint is a POST that accepts multipart form data, stores the result, and returns 200. It works for small files on reliable networks, which is most uploads on desktop browsers. It fails badly on large files (a 30-minute upload that drops at minute 28 loses the work and the bandwidth), on unreliable mobile networks (every interruption restarts), on duplicate uploads (multiple clients re-uploading the same file consume storage and processing), and on partial failures (an interrupted upload may leave inconsistent state).

The patterns that scale handle these cases without making the simple case complicated. The five concerns that matter are size limits enforced before parsing, resumable upload protocol for large or unreliable cases, content-addressed deduplication, validation at the right layer, and bounded storage cost via retention or quota enforcement.

Size limits enforced at the right layer

The first failure mode of naive upload endpoints is that the size limit is enforced after the data has already been received. The customer sends a 10GB file, the server buffers it (or worse, stores it), and only then does the size check happen. The bandwidth and storage cost has already been incurred, and the customer gets a 413 response after a long wait.

The correct pattern is to enforce size limits at the reverse proxy layer before the request body is parsed. Nginx has client_max_body_size, Caddy has request_body max_size, and most cloud load balancers have equivalent settings. The reverse proxy returns 413 immediately when the Content-Length header exceeds the limit, before the body is transferred. This costs the customer one round trip and a small response, not the full upload time and bandwidth.

The application-layer size limit is a backup in case the reverse proxy is misconfigured, and a finer-grained check for per-account or per-endpoint variation. The application limit should be slightly larger than the proxy limit to make misconfiguration obvious in monitoring (proxy 413s indicate normal operation; application 413s indicate the proxy is not enforcing limits correctly).

Resumable uploads via the upload-and-finalize pattern

For files large enough that a single-shot upload is impractical, the right pattern is a three-step protocol: initiate, upload-chunks, finalize. The initiate call creates an upload session with a server-generated ID and returns the chunk size and the URL pattern for chunk upload. The chunk-upload calls send byte ranges with Content-Range headers and store each chunk durably. The finalize call assembles the chunks into a complete file and returns the resource ID.

The chunk size choice is a trade-off between round-trip overhead and resumption granularity. 5-10MB chunks are typical for B2B SaaS use cases. Smaller chunks mean more overhead per chunk but finer-grained resumability. Larger chunks mean less overhead but more lost work on resume.

The resumption protocol requires the server to track which chunks have been received and which are still missing. The most direct implementation is a server endpoint that returns the list of received chunks (or equivalently, the next chunk number to send) so the client can resume after interruption. AWS S3 multipart upload, Google Cloud resumable upload, and Tus.io all implement variants of this protocol.

The session lifetime question is whether to clean up sessions that never finalize. Storage cost compounds quickly if abandoned sessions persist indefinitely. The standard policy is 24-hour to 7-day TTL on incomplete sessions, with cleanup running as a periodic background job. The TTL needs to be long enough that legitimate slow uploads complete (a 50GB upload over a slow connection can take days) but short enough that abandoned uploads do not accumulate.

Content-addressed deduplication

Many upload workloads involve substantial duplication: the same image uploaded by multiple users, the same PDF re-uploaded due to a customer mistake, the same input file processed multiple times. Content-addressed storage stores each unique content blob exactly once, with multiple references to it from different metadata records.

The implementation is a content hash (SHA-256 is the standard choice) computed during upload, used as the storage key. The upload endpoint hashes the content as it arrives, checks whether the hash already exists in storage, and only stores the actual bytes if this is a new blob. The metadata record stores the hash plus user-visible attributes like filename and upload timestamp.

The savings vary by workload. For deduplication-friendly workloads (image hosting, document repositories, code archives) the storage savings can be 30-70%. For deduplication-hostile workloads (encrypted user data with unique IVs, randomly-generated files) the savings are essentially zero and the hashing cost is pure overhead. The implementation cost is low enough that the default-on choice is usually right unless the workload is known to be deduplication-hostile.

The privacy implication is that two users uploading the same file will have their uploads linked at the storage layer, which has implications for some compliance contexts. The mitigation is per-tenant deduplication scopes (each tenant has its own content-addressed store) at the cost of giving up cross-tenant savings.

Validation at the right layer

Upload validation has three concerns: format, content, and security. Format validation answers whether the file matches the declared content type via magic-number detection (file headers, not file extension or Content-Type header which both lie). Content validation answers whether the file content is well-formed and parseable. Security validation answers whether the file contains content that should not be accepted.

The order matters because each layer is more expensive than the previous. Format validation is microseconds via a magic-number table. Content validation depends on the format but is typically milliseconds (PDF parsing, image decoding). Security validation can be tens or hundreds of milliseconds for full virus scanning. The right pattern is to validate cheaply and fail fast, only running expensive validation if the cheap checks pass.

The Content-Type and filename header trust question deserves explicit handling. Customer-supplied Content-Type and filename are advisory at best and adversarial at worst. The server should derive its own Content-Type from magic-number detection and should sanitize or replace the filename for storage. The server-generated storage key should never include any customer-supplied string that could affect filesystem paths.

Bounded storage cost

Upload endpoints without storage limits eventually become storage cost-runs. The pattern that prevents this is per-account quota enforcement, with quota counted against a measured value that the customer understands (typically bytes stored or files stored).

The quota check happens at upload time against current usage, with the upload rejected if it would exceed the quota. The customer-facing error code distinguishes quota exceeded from other failure modes (a 507 Insufficient Storage is the right code, with a body that explains current usage and quota).

The retention policy is the complement of the quota policy. Files that are no longer referenced (for example, drafts that were never published, or uploads to deleted resources) should be cleaned up on a schedule. The retention rules need to be customer-visible because data loss as a side effect of unclear retention is one of the more damaging failure modes.

Cold storage tiering is the natural extension once the storage cost becomes meaningful. Files that have not been accessed for 30+ days move to cold storage with higher retrieval latency but lower per-byte cost. The transition is usually invisible to the customer except in the first retrieval after the tier transition, which is several seconds slower. The cost savings can be substantial for write-once, read-rarely workloads.

Patterns across our products

Our use of upload endpoints varies across the four products. DocuMint has the heaviest upload surface because invoice generation accepts HTML and template overrides as input; the size limits are set conservatively at 500KB for inline content with no resumable protocol because the use case does not need it. WebhookVault receives webhook payloads which arrive in the request body rather than as separate uploads, so the upload concerns reduce to size limits and parsing safety. CronPing and FlagBit currently have no upload surface because their data model is small structured values, not files.

The deeper observation is that file upload is one of the API surfaces where the gap between "works for the happy path on a small test file" and "works under production traffic at scale" is largest. The five concerns (size limits, resumability, deduplication, validation, quota) each address a real failure mode that production traffic eventually exposes. Designing them in from the beginning is much cheaper than retrofitting them after the first incident.

Our products: DocuMint (PDF invoice generation API), CronPing (cron job monitoring with status pages), FlagBit (feature flags API for modern teams), and WebhookVault (webhook capture and replay) keep the lights on.