zstd vs lz4 Compression for Binlog Archiving #

Choosing a compressor for binlog archiving looks like a micro-optimization until a busy primary’s archiver starts stealing CPU from the database during a peak write window, or an egress bill arrives fatter than expected. The two realistic choices — zstd and lz4 — sit at different points on the same curve: lz4 compresses and decompresses extremely fast at a modest ratio, while zstd achieves a substantially higher ratio at tunable CPU cost, reaching lz4-like speed at its lowest levels. This page measures them against the constraints that actually matter for archiving row-based binlogs: CPU headroom on the primary, storage and egress cost, decompression speed during a time-critical recovery, and how each pairs with the streaming encryption the archive requires. The answer is rarely universal; it depends on where your bottleneck is.

Context & Prerequisites #

This decision refines the compression stage of Compression & Encryption Workflows for MySQL Binary Log Archiving and PITR Automation; read that for how compression sits ahead of client-side encryption in the pipeline. You need Python 3.10+ with the zstandard and lz4 bindings (or the zstd/lz4 CLIs), MySQL 8.0.22+ producing binlog_format=ROW segments, and the archiving daemon from Automated Binlog Archiving to Object Storage. Row-based binlogs compress well because of repeated column images, which is what makes the ratio difference meaningful.

Step-by-Step Implementation #

1. Measure the ratio on your own binlogs #

Compression ratio is data-dependent; measure on real segments rather than trusting generic benchmarks.

# Python 3.10+
import lz4.frame, zstandard as zstd
from pathlib import Path

def ratios(path: Path) -> dict[str, float]:
    raw = path.read_bytes()
    lz = lz4.frame.compress(raw)
    z3 = zstd.ZstdCompressor(level=3).compress(raw)
    z9 = zstd.ZstdCompressor(level=9).compress(raw)
    n = len(raw)
    return {"lz4": n / len(lz), "zstd-3": n / len(z3), "zstd-9": n / len(z9)}

PITR relevance: a higher ratio shrinks both storage and egress for every archived segment, and egress is the cost you pay again on every recovery fetch, so ratio compounds.

2. Measure CPU cost against primary headroom #

Time compression per MiB and compare it to the CPU the primary can spare during its busiest window.

# Python 3.10+
import time

def throughput_mib_s(compress, raw: bytes) -> float:
    t0 = time.monotonic()
    compress(raw)
    return (len(raw) / (1 << 20)) / (time.monotonic() - t0)

PITR relevance: if compression saturates a core the database needs, archiving lag rises and RPO widens — the compressor must fit inside spare CPU, not compete for hot CPU.

3. Choose the level, not just the algorithm #

zstd’s level is a dial from lz4-like speed (level 1) to high ratio (level 15+). Pick the highest level that stays within your CPU budget.

# Python 3.10+
def pick_zstd_level(spare_cpu_fraction: float) -> int:
    match spare_cpu_fraction:
        case f if f > 0.5:
            return 9      # plenty of headroom: chase ratio
        case f if f > 0.2:
            return 3      # balanced default for busy primaries
        case _:
            return 1      # scarce CPU: near-lz4 speed, still better ratio

PITR relevance: tuning the level lets one algorithm (zstd) span the whole trade-off space, so you rarely need lz4 unless you are truly CPU-starved.

4. Confirm decompression speed for recovery #

Decompression happens on the RTO critical path; both are fast, but verify the recovery target can decompress faster than it can apply.

PITR relevance: a compressor that decompresses slower than the target applies would bottleneck replay, but in practice both zstd and lz4 decompress far faster than MySQL applies row events, so ratio and compression-CPU dominate the decision.

Configuration Snippet & Reference Table #

# Python 3.10+  — stream-compress a segment with zstd before encryption
import zstandard as zstd

def zstd_stream(src, dst, level: int = 3) -> None:
    zstd.ZstdCompressor(level=level).copy_stream(src, dst)   # chunked, bounded memory

Verification Checklist #

• Ratios are measured on real ROW binlog segments, not synthetic data.
• Compression throughput fits within the primary’s spare CPU at peak write load.
• The zstd level is chosen from measured headroom, not left at a default.
• Decompression speed on the recovery target exceeds its apply rate (not a bottleneck).
• Compression runs as a stream (bounded memory), never buffering whole segments.
• The compressed output feeds client-side encryption before upload.

Gotchas & Version-Specific Caveats #

Compress before encrypting, never after. Encrypted bytes are effectively random and do not compress; the pipeline must compress the plaintext segment then encrypt the compressed output, per Compression & Encryption Workflows.

lz4’s ratio advantage is speed, not size. If storage or egress is the constraint, lz4 rarely wins; zstd at a low level is nearly as fast with a better ratio. Reach for lz4 only when the primary genuinely cannot spare the CPU.

zstd dictionaries help many small objects, not big segments. Trained dictionaries boost ratio on lots of tiny similar payloads; binlog segments are large, so a dictionary adds complexity for little gain.

Match the library to the CLI format. A segment compressed with the zstandard Python binding is readable by the zstd CLI and vice versa, but pin versions so a recovery host can always decompress what an archiver produced.

Frequently Asked Questions #

Related #

• Compression & Encryption Workflows — where compression sits ahead of encryption in the pipeline.
• Implementing AES-256 Encryption for Archived MySQL Binary Logs — the encryption stage the compressed output feeds.
• Object Storage Lifecycle & Cost Management — the other lever on archive storage cost.

Back to Compression & Encryption Workflows.