Postgres TOAST Compression: LZ4 vs PGLZ and the Tuning Choice Most Schemas Skip

Postgres pages are 8KB. Values larger than roughly 2KB do not fit alongside other values in a row, and Postgres deals with the size problem by transparently compressing the value and possibly storing it out-of-line in a side table. The mechanism is called TOAST, for The Oversized Attribute Storage Technique, and it has shipped in Postgres since 2001. For most of that history the compression algorithm was a single in-tree variant called PGLZ. Postgres 14 added LZ4 as an alternative, configurable per column, and the choice changes both compression ratio and CPU cost in ways worth understanding.

The defaults are conservative. Most schemas inherit the global default_toast_compression = pglz and never change it. The result is fine for most workloads and slightly worse than necessary for workloads with substantial large-value columns.

What TOAST actually does

When Postgres writes a row whose total size would exceed the page limit, it picks the largest variable-length attribute and tries four strategies in order: compress in place, store out-of-line uncompressed, store out-of-line compressed, or some combination, until the row fits. The strategy chosen per column is configured via ALTER TABLE ... ALTER COLUMN ... SET STORAGE with values PLAIN, EXTENDED, EXTERNAL, or MAIN. EXTENDED is the default for variable-length types and allows both compression and out-of-line storage.

The compression step uses whichever algorithm the column declares — either pglz (the default) or lz4 (Postgres 14+). Out-of-line values live in a side table called the TOAST relation, one per main table that has compressible columns. Reads against the main table fetch only the row's metadata and the in-line attributes; out-of-line attributes require a separate index lookup against the TOAST relation.

What PGLZ optimizes for

PGLZ is a custom LZ77 variant designed for the constraints of a database. It compresses moderately well — typically 2-4x ratio on text and JSON — and decompresses fast enough to not bottleneck normal queries. It does not require any external library. The algorithm has been stable since the early 2000s, which means existing TOAST data continues to decompress correctly across major version upgrades.

The cost is that PGLZ is slow to compress relative to modern algorithms. Bulk loads into tables with large compressible columns spend substantial CPU time in the compressor. Workloads that write a few large rows per second do not notice; workloads that ingest gigabytes per hour notice.

What LZ4 changes

LZ4 is a widely deployed compression library tuned for speed. Compression throughput is typically 4-10x faster than PGLZ for similar input. Decompression is also faster, sometimes by an order of magnitude. The compression ratio is usually slightly worse than PGLZ — closer to 2-3x on the same text and JSON workload. The trade-off is favorable for write-heavy workloads where the storage saved by a higher ratio is less valuable than the CPU saved by faster compression.

The configuration is per column and irreversible only in the sense that existing data stays in its original format until rewritten:

ALTER TABLE webhook_events ALTER COLUMN payload SET COMPRESSION lz4;

After the ALTER, new rows compress with LZ4 and existing rows decompress with PGLZ as they always have. Old data stays PGLZ until the row is updated or the table is rewritten via VACUUM FULL, CLUSTER, or pg_repack. Mixed-algorithm storage is a normal state and Postgres handles it correctly.

The compression ratio question

The right way to compare algorithms on actual workload is to copy representative data into a test table and measure both ratios and timing. pg_column_compression(column_name) returns the algorithm used for a particular value. pg_size_pretty(pg_total_relation_size(table_name::regclass)) reports the total size including TOAST. The diagnostic loop is straightforward:

-- Create twin tables with different compression
CREATE TABLE payload_pglz (id bigserial, body text);
ALTER TABLE payload_pglz ALTER COLUMN body SET COMPRESSION pglz;

CREATE TABLE payload_lz4 (id bigserial, body text);
ALTER TABLE payload_lz4 ALTER COLUMN body SET COMPRESSION lz4;

-- Load identical data into both
INSERT INTO payload_pglz (body) SELECT body FROM source_sample;
INSERT INTO payload_lz4 (body) SELECT body FROM source_sample;

-- Compare sizes
SELECT 'pglz' AS algo, pg_size_pretty(pg_total_relation_size('payload_pglz'))
UNION ALL
SELECT 'lz4', pg_size_pretty(pg_total_relation_size('payload_lz4'));

Typical results on natural-language text and structured JSON show LZ4 producing 5-15% larger total relation size compared to PGLZ. The variance is wider on small samples and tighter on representative samples. The diagnostic is repeatable and the answer for any specific workload is empirical, not predictable from algorithm theory.

The compression CPU question

Timing the bulk load reveals the other half of the trade-off. The same INSERT against the LZ4 table typically completes 3-8x faster than against the PGLZ table for compression-heavy inputs. The difference is roughly proportional to the fraction of total insert time spent in compression — for small rows with little compressible content, the difference disappears. For multi-kilobyte text or JSON, the difference dominates.

Decompression is faster too, but the difference is smaller as a fraction of query time because decompression is fast either way. Workloads that read a large fraction of TOASTed values per query see modest improvement; workloads that read mostly metadata see none.

When to switch and when to leave it

Workloads where the switch pays off cleanly include high-volume ingest of large JSON or text payloads, audit log tables with structured content, and any case where bulk loads bottleneck on CPU rather than disk. The storage cost is typically 5-15% and the CPU saving is typically 50-80%, which is favorable when storage is cheap and CPU is the binding constraint.

Workloads where PGLZ remains the right choice include cold archival tables where storage cost matters more than load speed, tables that compress poorly under both algorithms (already-compressed binary blobs, encrypted content), and tables small enough that the compression cost is invisible against query cost.

The global default is set via default_toast_compression in postgresql.conf. Changing the global default affects new tables and new columns; existing schemas are unaffected. Most teams that adopt LZ4 do so per-column on the handful of large-payload tables and leave the rest alone.

What we use across the four products

Our four products run on SQLite which does not have TOAST. The closest analog is the row-overflow page mechanism that handles values exceeding the page size by moving them to a chain of overflow pages without compression. SQLite trades compression for simplicity, which is the right choice for our scale.

The Postgres migration plan for the products specifies LZ4 for the columns that are obvious candidates: WebhookVault captured_request_body (multi-kilobyte JSON payloads ingested at high volume), DocuMint generation_log.context (template-rendered HTML kept for debugging), CronPing monitor_check.response_body (HTTP response bodies kept for diagnostic), and FlagBit evaluation_log.context (evaluation context blobs). For each, the compression CPU cost dominates over the marginal storage benefit, and the LZ4 default applies cleanly.

The smaller text columns — names, descriptions, error messages, configuration values — stay on default PGLZ because they rarely exceed the in-line threshold and the algorithm choice is mostly irrelevant to performance.

What this tells us about Postgres

Compression is one of those topics where Postgres provides a sensible default and a tunable knob, and the default is fine for most workloads and slightly worse than necessary for workloads with substantial compressible content. The pattern is recognizable across many Postgres features — autovacuum tuning, shared_buffers sizing, default statistics target, work_mem allocation, default index types. The defaults are correct for the default workload, which is a hypothetical average across all possible workloads, which means they are nobody's optimum.

The diagnostic loop for any of these knobs is the same: measure on representative workload, change one variable, measure again, decide based on the actual cost-benefit. For compression specifically, the measurement is unusually easy because the trade-off is between two scalar quantities (storage size, CPU time) and the algorithm choice is reversible by rewriting the data. The reversibility is what makes LZ4 worth trying — the worst case is reverting to PGLZ via another ALTER and another rewrite, which costs operational time but no permanent commitment.

Our products: DocuMint (PDF invoice generation API), CronPing (cron job monitoring with status pages), FlagBit (feature flags API for modern teams), and WebhookVault (webhook capture and replay) put these patterns into production.