Zstandard compression and decompression in safe Rust

A round trip

[dependencies]
mzstd = "0.2.13"

use mzstd::{Compressor, Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = "hello, smaller world! ".repeat(1000);
    let mut encoder = Compressor::new(EncoderConfig::default().with_level(Level::try_from(3)?))?;
    let compressed = encoder.compress(input.as_bytes())?;

    let mut decoder = Decompressor::default();
    assert_eq!(decoder.decompress(&compressed)?, input.as_bytes());
    Ok(())
}

Requires Rust 1.89 or later. Getting started continues from here with memory reuse, streaming and feature selection.

Pick a level

The level trades speed for size; the default is 3. Each bar is the compressed size of the 211 MB Silesia corpus against its original size.

mzstd 0.2.12 in lzbench 2.4.1 on one thread of an AMD Ryzen 9 7950X VDS, fastest of three rounds. Your data and machine will differ.

See the lzbench run for the command and raw output, and benchmark results for every measured level against zstd 1.5.7, including the cases where mzstd is slower.

Find your path

What you can rely on

• Byte-identical encoder. For the same input and parameters, serial compression writes exactly the bytes of zstd 1.5.7. See what is claimed and how it is checked.
• Safe implementation, not a C wrapper. Production code forbids unsafe; SIMD goes through fearless_simd. The reference C library is only a test and benchmark oracle.
• No hidden I/O or threads. Sessions touch only the buffers you pass in, and parallel compression hands tasks to your scheduler. The library does not create a thread pool.
• Reusable state. A codec keeps its workspace between payloads, under a retention policy you choose.

Framing requires the opt-in framed feature; its API may change in a patch release.

Getting started

Build a complete round trip, then choose the API that fits your application. Requires Rust 1.89+.

Install

cargo new zstd-demo
cd zstd-demo
cargo add mzstd@0.2.13

Or add this to Cargo.toml:

[dependencies]
mzstd = "0.2.13"

Compress and decompress

Replace src/main.rs with this complete program:

use mzstd::{Compressor, Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = "hello, smaller world! ".repeat(1000);
    let config = EncoderConfig::default().with_level(Level::try_from(3)?);
    let mut encoder = Compressor::new(config)?;
    let compressed = encoder.compress(input.as_bytes())?;

    let mut decoder = Decompressor::default();
    let restored = decoder.decompress(&compressed)?;
    assert_eq!(restored, input.as_bytes());

    println!(
        "{} input bytes → {} compressed bytes",
        input.len(),
        compressed.len()
    );
    Ok(())
}

cargo run --release

Choose the level explicitly: the default is level 3. Higher positive levels spend more CPU searching for compression opportunities. Compare speed and output size on your data; the benchmark pages show how results vary with input and level.

Reuse the workspace and output

A codec retains working memory between calls. compress_into appends a stream and returns its range; clear the destination when you want to reuse it for a new payload.

use mzstd::{Compressor, Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut encoder = Compressor::new(EncoderConfig::default().with_level(Level::try_from(3)?))?;
    let mut decoder = Decompressor::default();
    let mut output = Vec::new();

    for payload in [b"first message".as_slice(), b"second message".as_slice()] {
        output.clear();
        let stream = encoder.compress_into(payload, &mut output)?;
        assert_eq!(decoder.decompress(&output[stream])?, payload);
    }
    Ok(())
}

See retention policies to control which workspace allocations remain after an operation. Caller output capacity is separate.

Stream through Write

Write plain input into the adapter, then call finish to obtain a complete Zstandard stream. Dropping the writer does not finalize it.

use mzstd::{Compressor, Decompressor, EncoderConfig, InputSize, Level};
use std::io::Write;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"a payload arriving in two writes";
    let mut encoder = Compressor::new(EncoderConfig::default().with_level(Level::try_from(3)?))?;
    let stream = InputSize::Exact(input.len() as u64).into();
    let mut writer = encoder.writer(Vec::new(), stream);
    writer.write_all(&input[..10])?;
    writer.write_all(&input[10..])?;
    let compressed = writer
        .finish()
        .map_err(mzstd::io::FinishError::into_error)?;

    let mut decoder = Decompressor::default();
    assert_eq!(decoder.decompress(&compressed)?, input);
    Ok(())
}

For a retryable sink failure, FinishError::into_parts returns the error and writer. into_error discards the writer, as this in-memory example does. The streaming guide explains completion, backpressure, reader read-ahead, and direct sessions.

Choose features

The default features enable std and both codecs. To build only one codec, drop std, or enable framing containers, see Cargo features.

Next steps

• Sans I/O compression and decompression: caller buffers, progress counts, and completion.
• Read / Write I/O: readers and writers for raw, parallel, and framed streams.
• Parallel compression: scoped threads, Rayon, files, and streaming.
• Framing containers: multiple resources, metadata, and indexed reads.
• Choose your API: every entry point for both codecs, with its guide.
• Configuration and features: level, parameters, and Cargo features.
• Decompression and limits: budgets for untrusted input.
• Examples: pick a recipe and run the repository examples.

Choose your API

Compression and decompression expose the same I/O shapes. Pick the row that matches how your application already moves bytes, then follow its guide. New to the crate? Start with getting started.

Both codecs

Compressor and Decompressor own reusable working state. Keep one alive across payloads when allocation reuse matters.

Parallel compression

ParallelCompressor (in mzstd::compressor::parallel) is a separate execution strategy for compression: it splits one frame into independent segments and hands them to your executor as tasks. The library does not create a thread pool. The output is a valid Zstandard frame, decoded by the ordinary Decompressor. Requires std.

See parallel compression for scoped threads, Rayon, sources, staging, and streaming.

Guides by task

Errors

ConfigError covers encoder configuration, DecodeConfigError decoder limits, DictionaryError dictionary preparation, and EncodeError / DecodeError codec failures. Failed decoder calls report exact progress in DecodeFailure; output delivered before final validation can be partial. I/O adapters return std::io::Error, with the codec error as its source.

Reference

• API reference on docs.rs
• Compatibility and validation: what is byte-identical to zstd 1.5.7, and how it is checked
• Benchmark results for every measured level, with raw data
• Architecture: how the implementation works
• Development and benchmarking for contributors

Examples

The recipes below include complete programs and round-trip assertions. Examples use the default std, compression, and decompression features unless stated otherwise. The documentation workflow compiles and runs their Rust code against the current checkout.

Run from the repository

The runnable user recipes are complete programs embedded in the guides. To compile and run all of them against this checkout:

python3 scripts/docs.py test-examples

The repository also provides a profiling driver:

git submodule update --init --recursive
cargo run --release --example profile_compressor -- testdata/alice29.txt

See benchmarking for profiling options and the C comparison.

For a parallel batch, all taken tasks must run or be dropped before finalization. For a streaming adapter, keep workers running while the adapter blocks. For writers, finish each resource and the enclosing stream or container explicitly.

Verify the documented recipes

python3 scripts/docs.py test-examples

This creates a temporary Cargo consumer under target/, depends on the local crate with framed enabled and on Rayon, and runs each fenced Rust program from the recipe guides: getting started, configuration, buffers, decompression, streaming, parallel, frames, Sans I/O compression/decompression, the I/O guide, and dictionaries. It does not exercise the codec with --all-features, because that would also enable no_std and remove the std APIs these recipes demonstrate.

Configuration and features

Three things are configured separately: the encoder (level and parameters), the stream (what you know about one input), and the Cargo features (which codecs and adapters are compiled in). Decoder settings are covered in decompression and limits.

Choose a level first

EncoderConfig::default() uses level 3. Levels run from -131072 to 22; negative levels trade size for speed, and level 0 is an alias for 3.

use mzstd::{Compressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = EncoderConfig::default().with_level(Level::try_from(19)?);
    let mut encoder = Compressor::new(config)?;
    let compressed = encoder.compress(b"configured payload")?;
    assert!(!compressed.is_empty());
    Ok(())
}

Speed and size depend on your data. The benchmark pages show both for every measured level. Serial output is byte-identical to zstd 1.5.7 at every level, so compressed sizes match the reference.

Encoder settings

EncoderConfig holds settings shared by successive frames:

Explicit parameters override the level as ZSTD_CCtx_setParameter does. Compressor::new validates parameter combinations. CParameter provides a validated parameter-by-ID interface for integrations; see OpenZL integration.

StreamConfig carries an InputSize: unknown by default, or InputSize::Exact(input.len() as u64) when the full length is known. Declared sizes are checked. Session scheduling is part of the byte-identity contract.

Explicit parameters, a checksum and long distance matching on top of a level:

use mzstd::{CompressionParams, Compressor, EncoderConfig, LdmParams, Level, LongDistanceMatching, Strategy};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = EncoderConfig::default()
        .with_level(Level::try_from(9)?)
        .with_checksum(true)
        .with_params(CompressionParams::default().with_window_log(24).with_strategy(Strategy::Lazy2))
        .with_long_distance_matching(LongDistanceMatching::Enabled(LdmParams::default()));
    let mut encoder = Compressor::new(config)?;
    assert!(!encoder.compress(b"tuned payload")?.is_empty());
    Ok(())
}

See dictionaries and extended formats for dictionaries, large windows, long distance matching and magicless frames.

Cargo features

The default feature set is std, compression, and decompression.

See codec features and no_std support for the gates and the checked combinations.

Not supported

• External sequence producers
• Legacy (v0.x) frames
• Byte identity with zstd’s multithreaded mode
• Dictionaries in parallel compression and the C ABI

Buffers and memory reuse

Every call involves two kinds of memory: the destination you hand to the codec, and the workspace the codec keeps for itself. You control both.

Output buffers

compress returns a new Vec. compress_into appends to a supplied vector and returns the range containing the new frame. Existing bytes remain intact and its original length is restored on failure.

compress_to_slice returns the number of bytes written. Allocate a conservative buffer using Compressor::max_compressed_size:

use mzstd::Compressor;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"data to compress";
    let mut encoder = Compressor::default();
    let mut output = vec![0; Compressor::max_compressed_size(input.len())?];
    let written = encoder.compress_to_slice(input, &mut output)?;
    output.truncate(written);
    assert_eq!(output, encoder.compress(input)?);
    Ok(())
}

An undersized slice returns an error and can contain a partial prefix. Retry the complete operation with sufficient storage. compress_to_uninit and decompress_to_uninit accept caller-owned MaybeUninit<u8> destinations; read only the prefix reported as initialized. See decoder output contracts.

Reusing memory

Keep a compressor or decompressor across requests to reuse workspace. Caller output capacity is managed separately. A retention policy controls storage kept between operations:

Use Compressor::builder(config).with_retention(policy).build() or the decoder builder, and trim(policy) for immediate cleanup. retained_bytes() excludes caller output and allocator overhead. A retention ceiling is not a peak memory limit. Use a separate codec per simultaneous independent operation, or parallel compression to split one input across workers.

Decompression and limits

Decompressor owns reusable state. decompress returns a Vec, decompress_into appends a result, and decompress_to_slice writes into caller storage. Sessions report consumed/produced counts; reader and writer adapt synchronous I/O. Construction is infallible: Decompressor::new(config).

Set limits for untrusted input

The default accepts concatenated Zstandard frames and skips skippable frames. Content checksums are always verified. The default window limit is 128 MiB; input and output byte budgets are unlimited. Set limits for your application:

use mzstd::{DecodeLimits, DecoderConfig, Decompressor};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let limits = DecodeLimits::default()
        .with_max_window_log(23)?
        .with_max_input_bytes(Some(1 << 20))
        .with_max_output_bytes(Some(8 << 20));
    let mut decoder = Decompressor::new(DecoderConfig::default().with_limits(limits));
    let frame = mzstd::Compressor::default().compress(b"bounded payload")?;
    assert_eq!(decoder.decompress(&frame)?, b"bounded payload");
    Ok(())
}

The window log accepts 10–31. Input/output budgets count the whole operation, including concatenated frames; they are not process-memory ceilings. FrameMode::Single stops a session after one Zstandard or skippable frame. One-shot decoding rejects trailing bytes in that mode. FrameHeaderInfo inspects headers without decoding the payload. See decoder mechanics.

Where to go next

Streaming and completion

Use streaming when the payload does not fit in memory, arrives in pieces, or must leave before the input ends. There are three shapes, and they drive the same state machine:

Readers and writers require std. Sessions fit blocking code, async runtimes and event loops.

Readers and writers

encoder.reader(source, stream) pulls plain input and yields compressed bytes. encoder.writer(sink, stream) accepts plain input and delivers compressed bytes. Decoder adapters take just the source or sink: decoder.reader(source) and decoder.writer(sink). Their frame mode and limits come from DecoderConfig.

Read a reader to EOF. Call finish on a writer: dropping it performs no finalization. Encoder flush creates an intermediate decoding boundary and can change output size; decoder flush drains output without declaring final input. try_finish supports retrying a sink failure. Consuming finish returns the sink or a FinishError; into_parts recovers the error and writer, while into_error discards the writer. Reader into_parts preserves source read-ahead.

Sessions

For direct control, encoder.start(stream) creates a borrowed EncoderSession. decoder.start() creates a DecoderSession. Neither performs I/O.

Advance input by Progress::consumed and deliver only output[..produced]. Finish can take several calls. Decoder sessions use DecodeOperation and DecoderStatus; use Finish when no more compressed input can arrive. Owned session variants move the codec into a session without a borrowed-owner lifetime. See Sans I/O compression and decompression for complete loops.

Read / Write I/O

Use std adapters to connect the codecs to files, network streams, in-memory buffers, and other implementations of Read or Write. Adapters drive the same sessions as the Sans I/O APIs, adding buffering and synchronous I/O conventions.

API map

The library has no dedicated framed decoder Write adapter. Its event API keeps resource boundaries and metadata visible; forward each payload event to an appropriate Write destination. It also has no parallel decoder: parallel compression produces one ordinary raw Zstandard stream.

All adapters on this page require std. Enable framed for framing. no_std removes the std surface even when std is also enabled.

Raw compression and decompression through Read

An encoder reader consumes plain bytes and yields compressed bytes. A decoder reader consumes compressed bytes and yields restored bytes. std::io::copy drives each one to EOF. Use a File or socket in place of a slice.

use mzstd::{Compressor, Decompressor, EncoderConfig, InputSize, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"pull plain bytes through the encoder\n".repeat(100);
    let mut encoder = Compressor::new(EncoderConfig::default().with_level(Level::try_from(3)?))?;
    let stream = InputSize::Exact(input.len() as u64).into();
    let mut reader = encoder.reader(input.as_slice(), stream);
    let mut compressed = Vec::new();
    std::io::copy(&mut reader, &mut compressed)?;
    drop(reader);

    let mut decoder = Decompressor::default();
    let mut reader = decoder.reader(compressed.as_slice());
    let mut restored = Vec::new();
    std::io::copy(&mut reader, &mut restored)?;
    assert_eq!(restored, input);
    Ok(())
}

When the total input size is unknown, use the default stream configuration for the encoder. Reading a decoder to EOF is necessary to observe final validation errors. A raw decoder reader can read past a frame boundary in FrameMode::Single; recover its source and buffered suffix with into_parts when another protocol message follows.

Raw compression and decompression through Write

An encoder writer accepts plain bytes; a decoder writer accepts compressed bytes. Each owns its sink and borrows its codec owner. Finalize both writers.

use mzstd::{Compressor, Decompressor, EncoderConfig, Level};
use std::io::Write;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"push input into a writer\n".repeat(100);
    let mut encoder = Compressor::new(EncoderConfig::default().with_level(Level::try_from(3)?))?;
    let mut writer = encoder.writer(Vec::new(), Default::default());
    for chunk in input.chunks(13) {
        writer.write_all(chunk)?;
    }
    let compressed = writer
        .finish()
        .map_err(mzstd::io::FinishError::into_error)?;

    let mut decoder = Decompressor::default();
    let mut writer = decoder.writer(Vec::new());
    for chunk in compressed.chunks(3) {
        writer.write_all(chunk)?;
    }
    let restored = writer
        .finish()
        .map_err(mzstd::io::FinishError::into_error)?;
    assert_eq!(restored, input);
    Ok(())
}

A decoder writer’s finish checks that input ends at a valid stream boundary; flush only drains output and flushes the sink. Encoder flush also creates an intermediate decoding boundary and can change compressed size. Dropping a writer does not finalize or flush it.

Parallel compression through Read and Write

Both constructors return an adapter and a task queue. Run that queue on independent workers while using the adapter; reads, writes, and finish can block waiting for tasks. A queue with no running executor cannot make progress.

This reader example compresses across scoped threads, then uses an ordinary decoder reader to restore the assembled raw stream:

use mzstd::compressor::parallel::{ParallelCompressor, ParallelConfig, TaskCount};
use mzstd::{Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"parallel reader with caller-owned workers\n".repeat(220_000);
    let mut encoder = ParallelCompressor::new(
        EncoderConfig::default().with_level(Level::try_from(3)?),
        ParallelConfig::default(),
    )?;
    let mut compressed = Vec::new();
    std::thread::scope(|scope| -> Result<(), Box<dyn std::error::Error>> {
        let (mut reader, queue) = encoder.reader(
            input.as_slice(),
            Default::default(),
            TaskCount::try_from(4)?,
        );
        for _ in 0..4 {
            let worker_queue = queue.clone();
            scope.spawn(move || worker_queue.into_iter().for_each(|task| task.run()));
        }
        drop(queue);
        std::io::copy(&mut reader, &mut compressed)?;
        Ok(())
    })?;

    let mut decoder = Decompressor::default();
    let mut reader = decoder.reader(compressed.as_slice());
    let mut restored = Vec::new();
    std::io::copy(&mut reader, &mut restored)?;
    assert_eq!(restored, input);
    Ok(())
}

For the writer direction, the complete parallel writer recipe copies a source into the returned writer, then finishes it while workers keep running. The task window bounds outstanding segments; the parallel writer’s flush ends the current segment early. Decode its output with either raw adapter shown above. See parallel output and backpressure.

Framed compression through readers and resource writers

framed_reader encodes a complete borrowed FramedInput into a Read stream. It does not accept resources that arrive later. Use framed_writer when input arrives gradually: start a resource, write its payload, finish it, release its borrow, then start another resource. Finish the enclosing container last.

The framed writer recipe demonstrates compressed and uncompressed resource writers and metadata. Here is the reader direction, paired with sequential event decoding:

use mzstd::framing::{
    FramedCompressor, FramedDecompressor, FramedEvent, FramedInput, FramedItem, FramedResource,
};
use std::io::BufReader;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"a resource exposed as an encoded reader";
    let items = [FramedItem::Resource(FramedResource::from(input.as_slice()))];
    let mut encoder = FramedCompressor::new(Default::default())?;
    let mut encoded =
        encoder.framed_reader(FramedInput::from(items.as_slice()), Default::default())?;
    let mut container = Vec::new();
    std::io::copy(&mut encoded, &mut container)?;
    drop(encoded);

    let mut decoder = FramedDecompressor::new(Default::default())?;
    let source = BufReader::new(container.as_slice());
    let mut reader = decoder.framed_reader(source, Default::default())?;
    let mut restored = Vec::new();
    let mut resources = 0;
    while let Some(event) = reader.next_event()? {
        match event {
            FramedEvent::ResourceStart(_) => resources += 1,
            FramedEvent::ResourceData(data) => restored.extend_from_slice(data.bytes),
            _ => {}
        }
    }
    assert!(reader.is_finished());
    assert_eq!(resources, 1);
    assert_eq!(restored, input);
    Ok(())
}

Framed decompression into a Write sink

FramedReader lends events over a BufRead source. Wrap an ordinary Read in BufReader. For each ResourceData event, send data.bytes to your sink with write_all; use ResourceStart to select the destination for that resource. Continue until next_event returns None, after complete validation.

use mzstd::framing::{
    FramedCompressor, FramedDecompressor, FramedEvent, FramedInput, FramedItem, FramedResource,
};
use std::io::{BufReader, Write};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let items = [
        FramedItem::Resource(FramedResource::from(b"first".as_slice())),
        FramedItem::Resource(FramedResource::from(b"second".as_slice())),
    ];
    let mut encoder = FramedCompressor::new(Default::default())?;
    let container = encoder.compress(FramedInput::from(items.as_slice()))?;
    let mut decoder = FramedDecompressor::new(Default::default())?;
    let mut reader =
        decoder.framed_reader(BufReader::new(container.as_slice()), Default::default())?;
    let mut sinks: Vec<Vec<u8>> = Vec::new();
    while let Some(event) = reader.next_event()? {
        match event {
            FramedEvent::ResourceStart(_) => sinks.push(Vec::new()),
            FramedEvent::ResourceData(data) => {
                let sink = sinks.last_mut().ok_or("payload without a resource")?;
                sink.write_all(data.bytes)?;
            }
            _ => {}
        }
    }
    assert!(reader.is_finished());
    assert_eq!(sinks, vec![b"first".to_vec(), b"second".to_vec()]);
    Ok(())
}

This recipe uses in-memory sinks; a resource router can instead choose files or other writers. If a sink fails after an event was read, retain its unwritten suffix before advancing to the next event. FramedReader::partial_output exposes payload produced on a failing reader call. Keep protocol bytes after the container by recovering the buffered source with into_inner.

For Read + Seek, use indexed resource reads: framed_seek_reader opens the directory and resource(index) returns a ResourceReader implementing Read, suitable for std::io::copy. For push-driven compressed input, use the native framed decoder and route the same resource events to sinks.

Completion, retries, and ownership

Short sink writes and errors must not cause accepted input to be replayed. The writer adapters retain pending output, and a later call may report an error after an earlier call already accepted input. WouldBlock is not EOF. If you need explicit readiness handling, use native sessions and drive them when your runtime says buffers are ready. See streaming completion and framed retry mechanics.

Sans I/O compression

Use a session when your application owns the input/output buffers and decides when more bytes are available. Sessions perform no I/O. This works in a blocking loop, an async task, or a custom event loop without an I/O adapter.

For transport integration, see Read / Write I/O. For the inverse operation, see Sans I/O decompression.

Drive a raw encoder

Supply chunks with Operation::Continue. After the last chunk, call End until EncoderStatus::Finished. An empty input passed with Continue is not EOF. The deliberately small output buffer below forces repeated output drains.

use mzstd::{Compressor, Decompressor, EncoderConfig, EncoderStatus, InputSize, Level, Operation};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"incremental encoding with tiny buffers\n".repeat(100);
    let config = EncoderConfig::default().with_level(Level::try_from(3)?);
    let mut encoder = Compressor::new(config)?;
    let mut compressed = Vec::new();
    let mut output = [0; 11];
    {
        let mut session = encoder.start(InputSize::Exact(input.len() as u64).into());
        for chunk in input.chunks(7) {
            let mut remaining = chunk;
            loop {
                let progress = session.process(remaining, &mut output, Operation::Continue)?;
                remaining = &remaining[progress.consumed..];
                compressed.extend_from_slice(&output[..progress.produced]);
                if progress.status == EncoderStatus::NeedsInput {
                    assert!(remaining.is_empty());
                    break;
                }
            }
        }
        loop {
            let progress = session.process(&[], &mut output, Operation::End)?;
            compressed.extend_from_slice(&output[..progress.produced]);
            if progress.status == EncoderStatus::Finished {
                break;
            }
        }
    }
    let mut decoder = Decompressor::default();
    assert_eq!(decoder.decompress(&compressed)?, input);
    Ok(())
}

Always advance input by consumed and publish only output[..produced] before handling status. NeedsOutput requires output space and the unconsumed input suffix; reaching the end of the current input slice alone does not finish the stream.

Flush and finish are different

Operation::Flush makes accepted bytes decodable without ending the stream. Drain it to NeedsInput before changing operation or supplying new input. Flush boundaries can change output size and bytes. End ends the stream; while retrying it, keep the same operation and exact unconsumed input suffix.

Use InputSize::Exact when the total input size is known. Otherwise use the default unknown size; the encoder uses streaming parameter resolution. See stream scheduling and identity.

Drive parallel input

ParallelCompressor::start(stream, window) returns (session, queue). Keep workers running the queue while you feed input and drain output. The session’s process does not block: ParallelEncoderStatus::Pending means progress depends on tasks. Call session.wait() with independently runnable workers, or execute queued tasks yourself before waiting on a single thread.

The complete Sans I/O parallel example handles all statuses and source chunks. Its window bounds queued, running, and not-yet-delivered segments. With the exact input size declared and no flushes, output matches a batch over the same input and segment configuration. See parallel session mechanics.

Drive a framed encoder

Enable framed. Container process takes an output buffer and a FramedEncodeOperation; resource process takes payload, output, and an ordinary Operation. Finish the resource before finishing the container.

use mzstd::framing::{
    FramedCompressor, FramedDecompressor, FramedEncodeConfig, FramedEncodeOperation,
    FramedEncoderStatus, FramingConfig,
};
use mzstd::{EncoderConfig, Level, Operation};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"framed incremental payload\n".repeat(20);
    let config = FramedEncodeConfig::default()
        .with_encoder_config(EncoderConfig::default().with_level(Level::try_from(3)?))
        .with_framing_config(FramingConfig {
            chunk_bytes: 32,
            ..Default::default()
        });
    let mut encoder = FramedCompressor::new(config)?;
    let mut session = encoder.start(Default::default())?;
    let mut buffer = [0; 13];
    let mut container = Vec::new();
    // Drain the main header before accepting a resource command.
    loop {
        let progress = session.process(&mut buffer, FramedEncodeOperation::Process)?;
        container.extend_from_slice(&buffer[..progress.produced]);
        if progress.status == FramedEncoderStatus::NeedsInput {
            break;
        }
    }
    {
        let mut resource = session.resource(Default::default(), Default::default())?;
        let mut chunks = input.chunks(9).peekable();
        while let Some(chunk) = chunks.next() {
            let operation = if chunks.peek().is_none() {
                Operation::End
            } else {
                Operation::Continue
            };
            let mut remaining = chunk;
            loop {
                let progress = resource.process(remaining, &mut buffer, operation)?;
                remaining = &remaining[progress.consumed..];
                container.extend_from_slice(&buffer[..progress.produced]);
                if progress.status == FramedEncoderStatus::Finished
                    || progress.status == FramedEncoderStatus::NeedsInput
                {
                    assert!(remaining.is_empty());
                    break;
                }
            }
        }
        assert!(resource.is_finished());
    }
    loop {
        let progress = session.process(&mut buffer, FramedEncodeOperation::Finish)?;
        container.extend_from_slice(&buffer[..progress.produced]);
        if progress.status == FramedEncoderStatus::Finished {
            break;
        }
    }
    let mut decoder = FramedDecompressor::new(Default::default())?;
    assert_eq!(decoder.decompress(&container)?.resources[0].data, input);
    Ok(())
}

The example marks the last nonempty chunk with End; for an empty resource, drive resource.process(&[], output, Operation::End) to Finished too. Metadata and new resource commands require previous container output to be drained; otherwise they return OutputPending without committing the command. See resource lifecycles and metadata.

Ownership and cancellation

Borrowed sessions exclusively borrow their codec owner. Dropping an unfinished session cancels it; dropping an unfinished framed resource abandons its container. A forgotten framed session requires owner recover(). No drop performs transport I/O. Owned session variants move the codec into the session when a lifetime-free holder fits your application better. See owned sessions.

Sans I/O decompression

Feed compressed input and supply writable output space as it becomes available. Sessions own decoding state and retain no caller input/output slices between calls. DecodeOperation::Process allows more input; Finish declares the final input boundary and enables truncation checks.

There is no separate parallel decoder. Parallel compression assembles a standard raw stream, decoded through the ordinary decoder APIs. For transport adapters, see Read / Write I/O.

Decode a raw stream in chunks

This example supplies three compressed bytes at a time, uses a tiny output buffer, and declares Finish only for the last input chunk.

use mzstd::{Compressor, DecodeOperation, DecoderStatus, Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"incremental decoding preserves every byte\n".repeat(100);
    let mut encoder = Compressor::new(EncoderConfig::default().with_level(Level::try_from(3)?))?;
    let compressed = encoder.compress(&input)?;
    let mut decoder = Decompressor::default();
    let mut session = decoder.start();
    let mut chunks = compressed.chunks(3).peekable();
    let mut output = [0; 11];
    let mut restored = Vec::new();
    let mut finished = false;
    while let Some(chunk) = chunks.next() {
        let operation = if chunks.peek().is_none() {
            DecodeOperation::Finish
        } else {
            DecodeOperation::Process
        };
        let mut remaining = chunk;
        loop {
            let progress = session.process(remaining, &mut output, operation)?;
            remaining = &remaining[progress.consumed..];
            restored.extend_from_slice(&output[..progress.produced]);
            match progress.status {
                DecoderStatus::NeedsInput => {
                    assert!(remaining.is_empty());
                    break;
                }
                DecoderStatus::NeedsOutput => {}
                DecoderStatus::Finished => {
                    assert!(remaining.is_empty());
                    finished = true;
                    break;
                }
            }
        }
        if finished {
            break;
        }
    }
    assert!(finished);
    assert_eq!(restored, input);
    Ok(())
}

In an application where EOF is learned from a later read, offer chunks with Process and then call process(&[], output, Finish) until Finished. After the first Finish, continue with Finish and exactly the remaining input suffix. Do not add new bytes or switch back to Process.

NeedsOutput can occur after every input byte has been consumed: deliver the output and call again with an empty input slice and fresh output space. The default FrameMode::Concatenated waits for final input. For protocols with bytes after one frame, select FrameMode::Single and preserve unconsumed input when Finished arrives. The example asserts an empty suffix because its input contains exactly one complete member.

Decode a framing container as events

With framed, a framed session reports resource boundaries, metadata, payload fragments, directory entries, and a validated footer. Handle events before mutating the session again: events can borrow the session and output.

use mzstd::DecodeOperation;
use mzstd::framing::{
    FramedCompressor, FramedDecoderStatus, FramedDecompressor, FramedEvent, FramedInput,
    FramedItem, FramedResource,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"resource decoded one fragment at a time";
    let items = [FramedItem::Resource(FramedResource::from(input.as_slice()))];
    let mut encoder = FramedCompressor::new(Default::default())?;
    let container = encoder.compress(FramedInput::from(items.as_slice()))?;
    let mut decoder = FramedDecompressor::new(Default::default())?;
    let mut session = decoder.start(Default::default())?;
    let mut chunks = container.chunks(3).peekable();
    let mut buffer = [0; 5];
    let mut restored = Vec::new();
    let mut resources = 0;
    let mut finished = false;
    while let Some(chunk) = chunks.next() {
        let operation = if chunks.peek().is_none() {
            DecodeOperation::Finish
        } else {
            DecodeOperation::Process
        };
        let mut remaining = chunk;
        loop {
            let progress = session.process(remaining, &mut buffer, operation)?;
            remaining = &remaining[progress.consumed..];
            match progress.status {
                FramedDecoderStatus::Event(FramedEvent::ResourceStart(_)) => resources += 1,
                FramedDecoderStatus::Event(FramedEvent::ResourceData(data)) => {
                    restored.extend_from_slice(data.bytes);
                }
                FramedDecoderStatus::NeedsInput => {
                    assert!(remaining.is_empty());
                    break;
                }
                FramedDecoderStatus::Finished => {
                    assert!(remaining.is_empty());
                    finished = true;
                    break;
                }
                _ => {}
            }
        }
        if finished {
            break;
        }
    }
    assert!(finished);
    assert_eq!(resources, 1);
    assert_eq!(restored, input);
    Ok(())
}

ResourceDataEnd ends one resource’s payload; it does not validate the rest of the container. Continue through Finished to validate remaining structure. The recipe has one resource; for multiple resources, route ResourceData to the destination selected by ResourceStart rather than concatenating payloads. See framed event semantics.

Limits and errors

Set input, output, and window limits in DecoderConfig before starting a raw decoder. For framing, use FramedDecodeConfig and its framing limits in addition to raw decoder limits. See resource-limit configuration.

Output delivered before a later error is partial and has not passed complete validation. Failure values carry consumed/produced counts for that failing call; account for them if your transport needs exact progress. Drop a failed session before starting a new operation. Finish reports truncated streams; an empty Process call does not declare EOF.

Parallel compression

mzstd::compressor::parallel divides a known input into fixed independent segments and assembles them into one standard Zstandard frame. It supports levels -131072–22. Dictionaries, magicless frames, and framing containers are unsupported by this API. Output is deterministic for fixed input and segment settings, but differs from serial compression and zstd’s multithreaded mode.

The caller runs the tasks using scoped threads, Rayon, or another executor. The library does not create a thread pool.

Compressing a slice

use mzstd::compressor::parallel::{ParallelCompressor, ParallelConfig, TaskCount};
use mzstd::{Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = vec![b'a'; 8 << 20];
    let mut encoder = ParallelCompressor::new(
        EncoderConfig::default().with_level(Level::try_from(3)?),
        ParallelConfig::default(),
    )?;
    let mut batch = encoder.prepare_slice(&input, TaskCount::try_from(4)?);
    let tasks = batch.take_tasks();
    std::thread::scope(|scope| {
        for task in tasks {
            scope.spawn(move || task.run());
        }
    });
    let mut output = Vec::new();
    batch.finish_into(&mut output)?;
    let mut decoder = Decompressor::default();
    assert_eq!(decoder.decompress(&output)?, input);
    Ok(())
}

take_tasks transfers each task once. Every taken task must run or be dropped so its completion can be reported. Join workers before finalization: it collects already reported results, checks the source, and assembles the frame in input order. A missing completion returns InvalidState.

Running tasks with Rayon

Add rayon = "1.12" alongside mzstd in your dependencies. Create a pool once and reuse it across requests. Task completion order does not affect assembly order or the resulting bytes for fixed segment settings.

use mzstd::compressor::parallel::{ParallelCompressor, ParallelConfig, TaskCount};
use mzstd::{Decompressor, EncoderConfig, Level};
use rayon::prelude::*;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let pool = rayon::ThreadPoolBuilder::new().num_threads(4).build()?;
    let mut encoder = ParallelCompressor::new(
        EncoderConfig::default().with_level(Level::try_from(3)?),
        ParallelConfig::default(),
    )?;
    let input = b"parallel compression with a reusable pool\n".repeat(220_000);
    let mut batch = encoder.prepare_slice(&input, TaskCount::try_from(4)?);
    let tasks = batch.take_tasks();
    pool.install(|| tasks.into_par_iter().for_each(|task| task.run()));

    let mut compressed = Vec::new();
    batch.finish_into(&mut compressed)?;
    let mut decoder = Decompressor::default();
    assert_eq!(decoder.decompress(&compressed)?, input);
    Ok(())
}

For many unrelated small inputs, another option is one ordinary Compressor per worker. That preserves serial compression decisions for each input. The batch API above splits one input into a single assembled stream.

Streaming input

When input arrives in chunks, start(stream, window) returns a session and a task queue. Each full segment the session accepts becomes a task on that queue; clone the queue for each executor and run for task in queue { task.run() }. The session delivers finished parts in input order while later segments are still encoding, and closes the queue when the stream finishes, fails, or is dropped.

use mzstd::compressor::parallel::{
    ParallelCompressor, ParallelConfig, ParallelEncoderStatus, TaskCount,
};
use mzstd::{EncoderConfig, Level, Operation};
use std::io::{Read, Write};

fn compress(
    mut reader: impl Read,
    mut writer: impl Write,
) -> Result<(), Box<dyn std::error::Error>> {
    let mut encoder = ParallelCompressor::new(
        EncoderConfig::default().with_level(Level::try_from(3)?),
        ParallelConfig::default(),
    )?;
    std::thread::scope(|scope| {
        let (mut session, queue) = encoder.start(Default::default(), TaskCount::try_from(4)?);
        for _ in 0..4 {
            let queue = queue.clone();
            scope.spawn(move || queue.into_iter().for_each(|task| task.run()));
        }
        drop(queue);
        let mut chunk = vec![0; 1 << 16];
        let mut output = vec![0; 1 << 16];
        loop {
            let read = reader.read(&mut chunk)?;
            let operation = if read == 0 {
                Operation::End
            } else {
                Operation::Continue
            };
            let mut input = &chunk[..read];
            loop {
                let progress = session.process(input, &mut output, operation)?;
                input = &input[progress.consumed..];
                writer.write_all(&output[..progress.produced])?;
                match progress.status {
                    ParallelEncoderStatus::Finished => return Ok(()),
                    ParallelEncoderStatus::Pending => session.wait()?,
                    ParallelEncoderStatus::NeedsInput if input.is_empty() => break,
                    _ => {}
                }
            }
        }
    })
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"incremental parallel input\n".repeat(350_000);
    let mut compressed = Vec::new();
    compress(input.as_slice(), &mut compressed)?;
    let mut decoder = mzstd::Decompressor::default();
    assert_eq!(decoder.decompress(&compressed)?, input);
    Ok(())
}

• process never blocks. Pending means the window is full or a flush or finish waits for queued tasks; call wait, which blocks until one completes.
• window bounds the segments queued, running or awaiting delivery, and so the session’s memory. A full window stops input from being consumed.
• With the exact input size declared and no flushes, the output is byte-identical to prepare_slice over the same bytes, however the input was chunked. Operation::Flush ends the current segment early, which moves later segment boundaries.
• On a single thread, run queue.try_recv() tasks before wait; nothing else will. Dropping every queue handle fails the stream instead of hanging.
• into_session(stream, window) returns an owned session without a lifetime; into_compressor returns the compressor.

Reader and writer adapters

writer(sink, stream, window) and reader(source, stream, window) wrap the same session in std::io::Write and std::io::Read and return the task queue beside the adapter, so the loop above becomes std::io::copy:

use mzstd::compressor::parallel::{ParallelCompressor, ParallelConfig, TaskCount};
use mzstd::io::FinishError;
use mzstd::{EncoderConfig, Level};
use std::io::{Read, Write};

fn compress<W: Write>(mut reader: impl Read, writer: W) -> Result<W, Box<dyn std::error::Error>> {
    let mut encoder = ParallelCompressor::new(
        EncoderConfig::default().with_level(Level::try_from(3)?),
        ParallelConfig::default(),
    )?;
    std::thread::scope(|scope| {
        let (mut sink, queue) = encoder.writer(writer, Default::default(), TaskCount::try_from(4)?);
        for _ in 0..4 {
            let queue = queue.clone();
            scope.spawn(move || queue.into_iter().for_each(|task| task.run()));
        }
        drop(queue);
        std::io::copy(&mut reader, &mut sink)?;
        Ok(sink.finish().map_err(FinishError::into_error)?)
    })
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"stream through caller-owned workers\n".repeat(250_000);
    let compressed = compress(input.as_slice(), Vec::new())?;
    let mut decoder = mzstd::Decompressor::default();
    assert_eq!(decoder.decompress(&compressed)?, input);
    Ok(())
}

• Unlike process, the adapters block: a write waits while the window is full, a read waits for the next part, and finish waits for every part. Run the queue on threads other than the one using the adapter.
• With the exact input size declared and no flushes, output matches prepare_slice. The writer’s flush is Operation::Flush: it makes everything written decodable and moves later segment boundaries.
• The writer keeps every compressed byte until the sink takes it, so short writes, WouldBlock and sink errors lose nothing, and finish hands the adapter back inside FinishError for a retry, as EncoderWriter does. reader.into_parts() returns source bytes read ahead of the session.
• Session failures arrive as std::io::Error with the ParallelEncodeError as their source. Dropping an adapter abandons the stream and closes the queue.

Segments and compressed size

Segment size defaults to 4 MiB and accepts 64 KiB–256 MiB. Effective task count is at most the number of segments. Task grouping and completion order do not change output bytes.

Segments are primed with preceding input: an eighth of the window for most strategies, or the full window for BtUltra and BtUltra2. The overlap is not capped by segment size. Long distance matching runs within each segment. Compression can be worse than serial compression on data whose useful history exceeds that overlap; compare compressed size as well as elapsed time. An input that fits in one segment uses serial encoding and matches Compressor.

Sources and staging

tasks is a validated TaskCount. FileSource uses positional reads on Unix and Windows. SeekSource accepts Read + Seek + Send, serializes seek/read operations, and lets compression run concurrently. Sources must remain immutable for the batch lifetime; identity checks cannot detect every in-place mutation.

Owned sources and shared handles use the same generic entry point. For an existing Arc<FileSource>, type inference can require prepare_source::<FileSource, _>(shared_file, tasks).

Batch output is staged in memory. Use streaming sessions and their bounded window when the complete input or compressed output should not be retained.

Output and cancellation

finish returns a new vector. finish_into appends a frame and restores the vector’s original length on failure. Write the completed frame to a sink or use the streaming writer; the caller controls flushing, durability, and publication.

Every taken task must run or be dropped, and workers must finish before batch finalization. The batch does not wait for detached tasks. Dropping a batch abandons assembly without joining detached workers. Worker storage can be reused by the parallel owner. See parallel mechanics for lifecycle, source checks, memory, cancellation, and error propagation.

Framing containers

Package multiple resources, metadata, and optional dictionary references in an opt-in Zstandard container. This is useful when an application needs resource boundaries or indexed access in addition to compression.

Enable framing

[dependencies]
mzstd = { version = "0.2.13", features = ["framed"] }

The examples use std, compression, and decompression. Framing APIs may change in a patch release. Use FramedDecompressor to preserve resources and metadata. An ordinary Zstandard decoder extracts the concatenated visible resources and skips the container metadata, hidden resources, directory, and footer.

The library’s parallel compressor emits one raw Zstandard stream. It does not emit framing containers. See parallel compression when the goal is compressing one input across cores.

One container, two resources

Use borrowed inputs when all resource bytes are available. The first resource below is Zstandard-compressed; the second is explicitly stored without compression.

use mzstd::framing::{
    FramedCompressor, FramedDecompressor, FramedEncodeConfig, FramedInput, FramedItem,
    FramedResource, ResourceEncoding,
};
use mzstd::{EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let text = b"a document with repeated content\n".repeat(1000);
    let binary = [0x91, 0x02, 0x7f, 0x00];
    let mut stored = FramedResource::from(binary.as_slice());
    stored.encoding = ResourceEncoding::Uncompressed;
    let items = [
        FramedItem::Resource(FramedResource::from(text.as_slice())),
        FramedItem::Resource(stored),
    ];

    let config = FramedEncodeConfig::default()
        .with_encoder_config(EncoderConfig::default().with_level(Level::try_from(3)?));
    let mut encoder = FramedCompressor::new(config)?;
    let container = encoder.compress(FramedInput::from(items.as_slice()))?;

    let mut decoder = FramedDecompressor::new(Default::default())?;
    let decoded = decoder.decompress(&container)?;
    assert_eq!(decoded.resources.len(), 2);
    assert_eq!(decoded.resources[0].data, text);
    assert_eq!(decoded.resources[1].data, binary);
    Ok(())
}

compress_into appends a container to a reusable Vec; compress_to_slice writes into caller storage. The owner can be reused for later containers. The framed compressor specification describes size contracts, retention, and native incremental output APIs.

Write resources and metadata

Use a writer when resource input arrives gradually. Resource metadata precedes the resource it describes. Finish each resource before starting the next one, then finish the container.

use mzstd::framing::{
    FramedCompressor, FramedDecompressor, FramedEncodeConfig, FramingConfig, MetadataField,
    MetadataKind,
};
use mzstd::{EncoderConfig, Level};
use std::io::Write;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = FramedEncodeConfig::default()
        .with_encoder_config(EncoderConfig::default().with_level(Level::try_from(3)?))
        .with_framing_config(FramingConfig {
            chunk_bytes: 4096,
            ..Default::default()
        });
    let mut encoder = FramedCompressor::new(config)?;
    let mut writer = encoder.framed_writer(Vec::new(), Default::default())?;
    writer.metadata(
        MetadataKind::Resource,
        &[MetadataField {
            code: *b"id",
            value: b"notes.txt",
        }],
    )?;
    {
        let mut resource = writer.resource(Default::default(), Default::default())?;
        resource.write_all(b"first part\n")?;
        resource.write_all(b"second part\n")?;
        resource.try_finish()?;
    }
    {
        let mut resource = writer.uncompressed_resource(Default::default())?;
        resource.write_all(&[1, 2, 3, 4])?;
        resource.try_finish()?;
    }
    let container = writer.finish().map_err(|failure| failure.error)?;

    let mut decoder = FramedDecompressor::new(Default::default())?;
    let decoded = decoder.decompress(&container)?;
    assert_eq!(decoded.resources[0].data, b"first part\nsecond part\n");
    assert_eq!(
        decoded.resources[0]
            .metadata
            .as_ref()
            .and_then(|m| m.name()),
        Some("notes.txt")
    );
    assert_eq!(decoded.resources[1].data, [1, 2, 3, 4]);
    Ok(())
}

Neither resource drop nor container drop performs I/O. Dropping an unfinished resource abandons its container. try_finish supports retrying resource finalization; a failed consuming container finish retains the writer in FramingFinishError for recovery through its writer and error fields. The in-memory example above keeps the error and discards the writer. See the retry state machine.

Metadata id is a caller-provided UTF-8 name, not an extraction path or a unique lookup key. Checksums are supplied by the caller; the writer does not compute them and the decoder does not verify them.

Indexed resource reads

For a container with a central directory on a Read + Seek source, open the footer and directory, then read a selected resource. You can use a File in place of the in-memory Cursor below.

use mzstd::framing::{
    FramedCompressor, FramedDecompressor, FramedInput, FramedItem, FramedResource, ResourceIndex,
};
use std::io::{Cursor, Read};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let items = [
        FramedItem::Resource(FramedResource::from(b"first".as_slice())),
        FramedItem::Resource(FramedResource::from(b"second".as_slice())),
    ];
    let mut encoder = FramedCompressor::new(Default::default())?;
    let container = encoder.compress(FramedInput::from(items.as_slice()))?;

    let mut decoder = FramedDecompressor::new(Default::default())?;
    let mut reader = decoder.framed_seek_reader(Cursor::new(container))?;
    assert_eq!(reader.resources().len(), 2);
    let mut selected = Vec::new();
    reader
        .resource(ResourceIndex(1))?
        .read_to_end(&mut selected)?;
    assert_eq!(selected, b"second");
    Ok(())
}

Resources are identified by zero-based ResourceIndex, independent of metadata names. A resource that references earlier dictionary content can require decoding its dependencies. A selected resource reader implements Read; the enclosing reader uses Seek to locate container content. The final frame is a seek table in zstd’s contrib/seekable_format layout. See indexed decoder mechanics.

Limits and streaming decoding

FramingConfig bounds encoder chunks, metadata, resource/chunk counts, and framing storage. FramedDecodeConfig / FramedDecodeLimits control decoder budgets; raw DecoderConfig limits apply within compressed resources. Choose limits for the workload before decoding untrusted containers.

For arriving input, FramedDecompressor::start exposes incremental events; framed_reader adapts a BufRead source. InputMode::Auto also accepts ordinary Zstandard frames; the default is FramedOnly. External dictionaries need a caller-supplied resolver. See decoder APIs, limits, and events and dictionaries.

Dictionaries and extended formats

Prepared dictionaries

A PreparedDictionary owns immutable dictionary bytes and compression indexes. Prepare it once for the encoder configuration and borrow it across operations or threads. Both raw content and trained Zstandard-format dictionaries work with the ordinary codec features; framing is not required.

use mzstd::dictionary::DictionaryBuilder;
use mzstd::{Compressor, Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = EncoderConfig::default().with_level(Level::try_from(3)?);
    let dictionary = DictionaryBuilder::new(b"HTTP/1.1 200 OK\r\nContent-Type: ")
        .with_config(config)
        .build()?;
    let input = b"HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n";
    let frame = Compressor::new(config)?.compress_with_dictionary(&dictionary, input)?;
    assert_eq!(
        Decompressor::default().decompress_with_dictionary(&dictionary, &frame)?,
        input
    );
    Ok(())
}

DictionaryContent::Auto recognizes Zstandard dictionary magic, Raw forces raw content, and Zstd requires a trained dictionary including valid entropy tables. The library consumes trained dictionaries; it does not train them. A trained dictionary carries a nonzero ID checked against the frame. Raw content has no ID, so applications must supply the correct bytes themselves.

DictionaryBuilder::with_config prepares for matching compressor parameters; frame checksum/content-size options can differ. Incompatible configurations return EncodeError::DictionaryMismatch. with_level_parameters(level) instead prepares a standalone CDict with ZSTD_createCDict_advanced / ZSTD_CCtx_refCDict semantics; its level replaces the context level during parameter resolution. See dictionary mechanics.

Decoder dictionaries

For a decoder-only application, prepare DecodeDictionary::new(bytes, content). This avoids encoder search indexes and works with only decompression enabled. Decoder entry points borrow a prepared encoder or decoder dictionary through DictionaryRef. Reuse it for one-shot, slice, session, reader, and writer APIs.

Dictionary IDs identify content; the decoder does not fetch missing dictionaries. See decoder dictionaries.

Large windows and long distance matching

Set a window through CompressionParams::with_window_log and enable long distance matching through LongDistanceMatching::Enabled(LdmParams::default()). These remain standard Zstandard frames. Explicit parameter validation follows the reference; decoder window limits must permit the resulting declaration. Larger windows can require more retained history and memory. See encoder core and configuration.

Magicless frames

EncoderConfig::with_format(FrameFormat::Magicless) omits the four-byte Zstandard magic. Select the same format through DecoderConfig::with_format: it cannot be autodetected. This supports integrations such as OpenZL; it does not apply to parallel compression, framed containers, or the C ABI. See OpenZL integration.

Feature flags

See codec features for checked combinations.

Framing containers

With framed, resources and compressed metadata can refer to external or earlier internal dictionaries. The caller supplies dictionary identifiers, references, and a resolver for external content. Ordinary Zstandard decoding concatenates visible resources and skips the container’s other frames; resource-aware decoding uses FramedDecompressor.

Each compressed chunk is a Zstandard frame. Metadata, hidden resources, directory, and footer use skippable frames; the last frame is a contrib-compatible seek table. Optional resource IDs/checksums are caller-supplied and not computed or verified by the container layer; ordinary Zstandard frame checksums are verified by the codec. See framing recipes, wire format, and decoder limits and events.

Compatibility and validation

This page describes the contracts checked by the repository. Tests, fuzzing, coverage, and memory tools provide bounded evidence. Lean proofs cover selected format primitives, rather than the entire codec. For commands, see development.

What is claimed

The development oracle is the vendored submodule at zstd-ffi/vendor/zstd. It is not a production dependency of the main crate. The library forbids unsafe in production code; SIMD and uninitialized-output dependencies still need memory-tool checks. The separate mzstd-ffi crate provides a C ABI.

Output compatibility

Single-threaded one-shot compression is byte-identical to zstd 1.5.7 ZSTD_compress2 with equivalent input, parameters, and dictionary. Sessions match ZSTD_compressStream2 driven with the same chunks, output capacities, flushes, and pledged size. Different schedules are not interchangeable byte oracles. SIMD selection and workspace reuse preserve observable behavior.

Parallel compression writes valid, deterministic Zstandard frames, but its bytes differ from serial compression and zstd’s multithreaded mode. Framing is this crate’s container format built from Zstandard and skippable frames.

Validation layers

• Encoder differential tests cover levels, random advanced parameters, long distance matching, dictionaries, short destinations, pledged sizes, target compressed block sizes, and streaming schedules with flushes and window wrap.
• The overflow-correction build forces index rescaling often on both sides.
• Decoder tests cover reference frames, golden vectors, truncation, corruption, limits, checksums, and all public input/output shapes.
• AFL++ and ClusterFuzzLite exercise byte-consuming and stateful boundaries; minimized findings become committed regressions.
• Miri and AddressSanitizer exercise memory-sensitive paths, including SIMD and uninitialized destinations. Function coverage is checked separately.
• Lean 4 compares selected translated Rust functions with a transcription of RFC 8878. See formal verification for the trusted base and coverage checklist.

Scheduling and partial output

Serial session byte identity requires equivalent C and Rust call schedules. A different chunk or flush schedule can produce different valid bytes. Parallel output is not a serial or multithreaded C byte oracle. Performance comparisons must record size as well as speed and validate decoded bytes first.

Streaming output delivered before an error is partial and may not have passed checksum or structural validation. Account for a failure’s consumed/produced counts and do not replay input already accepted by a writer. Complete readers to EOF and explicitly finish writers and sessions.

Scope and known gaps

The codecs do not support legacy zstd v0.x frames or external sequence producers. Parallel compression and the C ABI do not accept dictionaries; magicless framing is limited to ordinary codec APIs. Container resource checksum fields are recorded without verification. See the current architecture gaps, encoder verification, decoder verification, and CI checks.

Benchmark results

Two kinds of runs, both on an AMD Ryzen 9 7950X (Linux KVM guest with two vCPUs, its host shared with other tenants): the cold library comparison of mzstd 0.2.12 against zstd 1.5.7, recorded on 2026-11-09, and lzbench on silesia.tar. Every report identifies its measured sources, sampling settings and limitations. Other tenants loaded the host during both: the medians are stable, single cells are good to about ±10%.

Compression

Each speed bar is the median of eight per-dataset ratios to C, timed in the same process. Higher speed and lower size are better. Every mzstd frame is byte-identical to zstd’s, so the size ratio is 1× at every level. The median over all 192 dataset/level pairs is 1.248×, with lower mean latency in 174 pairs (2026-10-30: 1.256×, 179 pairs). The per-dataset medians agree with that run to within a few percent: Alice 1.008×, cyclic Alice 1.088×, structured binary 1.241×, repeated bytes 2.437×. The 18 slower pairs are Alice at ten levels, cyclic Alice at four and random data at four, between 0.84× and 0.998×. Report · exact results and confidence bounds.

Decompression

Both decoders restore the same C-written frames; level is the source encoder’s setting. The median over all 192 pairs is 1.517× (2026-10-30: 1.502×), faster in 151 pairs: tens of times faster on empty and tiny frames, 3.2× on random 64 KiB, 1.53× on repeated bytes and 1.45× on structured binary, but 0.948× on Alice and 0.937× on cyclic Alice, the datasets of all 41 slower pairs. Report · exact results.

lzbench

lzbench master a6fa8aa (2.4.1), gcc 15.2 with -march=native, silesia.tar as one frame on one vCPU, the fastest of three rounds: mzstd 0.2.12 writes byte-identical output to zstd 1.5.7 at every level, compresses 6.6-17.5% faster at levels 15, 18 and 22, at zstd’s speed at level 3 and 2.4-6.6% slower at the other levels, and decompresses 4.7-7.2% slower at every level. Against 0.2.11 in the same rounds, relative to zstd, decompression gained 0.8-5 points at levels 1 to 22 and level 3 compression went from 0.973 to 1.011. Compression at level 2 and above moves by 3-21% between rounds on this host. Command, environment, raw output of every round.

How to interpret the results

The cold suites time construction, allocation, the call and disposal. Empty and tiny inputs have equal weight with large inputs; medians describe this corpus, not a production workload. C decoding knows the output capacity from the frame header. The lowest mean is not necessarily a statistically significant lead, and on this run’s loaded host a single cell’s ratio can move by 10% between sweeps.

The datasets, generators and seeds are mbrotli’s, so the two projects’ comparisons describe one corpus.

Reproduce

Use benchmarking and profiling for validation, fresh timings, the main branch’s series, the Silesia sweep and the report-generation commands.

Encoder comparison

Benchmark index

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

All 8 datasets contribute equally to each median, including empty and tiny input. For each dataset, speed / C = C mean latency / mzstd mean latency; output / C = mzstd bytes / C bytes. Medians are taken over those ratios. Levels are ordered by mzstd median speed / C, highest first. Frames are byte-identical, so output / C is 1× at every level.

Level -5

Level -1

Level 2

Level 1

Level 16

Level 19

Level 17

Level 18

Level 20

Level 8

Level 3

Level 9

Level 10

Level 5

Level 7

Level 6

Level 21

Level 15

Level 22

Level 12

Level 13

Level 11

Level 4

Level 14

Compression level -5

Benchmark index · Median overview · Level -1 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · tiny-text · repeated-1m · binary-64k · random-64k · random-1m · alice29 · text-1m

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 10.763×; output: 9 / 9 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 8.134×; output: 53 / 53 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 2.799×; output: 55 / 55 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.801×; output: 3,189 / 3,189 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.638×; output: 65,546 / 65,546 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.138×; output: 1,048,610 / 1,048,610 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 1.109×; output: 85,658 / 85,658 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 1.091×; output: 96,657 / 96,657 bytes.

Compression level -1

Benchmark index · Median overview · ← Level -5 · Level 1 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · tiny-text · repeated-1m · binary-64k · random-64k · random-1m · text-1m · alice29

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 11.441×; output: 9 / 9 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 9.111×; output: 53 / 53 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 2.735×; output: 51 / 51 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.738×; output: 2,442 / 2,442 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.528×; output: 65,546 / 65,546 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.122×; output: 1,048,610 / 1,048,610 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 1.085×; output: 80,894 / 80,894 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 0.953×; output: 71,645 / 71,645 bytes.

Compression level 1

Benchmark index · Median overview · ← Level -1 · Level 2 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · tiny-text · repeated-1m · binary-64k · random-64k · text-1m · random-1m · alice29

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 11.256×; output: 9 / 9 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 7.708×; output: 53 / 53 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 2.871×; output: 51 / 51 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.501×; output: 2,537 / 2,537 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.259×; output: 65,546 / 65,546 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 1.153×; output: 61,206 / 61,206 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.094×; output: 1,048,610 / 1,048,610 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 0.993×; output: 59,657 / 59,657 bytes.

Compression level 2

Benchmark index · Median overview · ← Level 1 · Level 3 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · tiny-text · repeated-1m · binary-64k · random-64k · text-1m · random-1m · alice29

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 11.177×; output: 9 / 9 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 7.214×; output: 53 / 53 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 2.772×; output: 50 / 50 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.484×; output: 2,425 / 2,425 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.426×; output: 65,546 / 65,546 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 1.198×; output: 58,340 / 58,340 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.099×; output: 1,048,609 / 1,048,609 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 0.841×; output: 57,348 / 57,348 bytes.

Compression level 3

Benchmark index · Median overview · ← Level 2 · Level 4 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · tiny-text · repeated-1m · binary-64k · text-1m · random-64k · random-1m · alice29

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 10.353×; output: 9 / 9 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 6.912×; output: 53 / 53 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 3.340×; output: 50 / 50 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.391×; output: 2,324 / 2,324 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 1.145×; output: 55,878 / 55,878 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.143×; output: 65,546 / 65,546 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.020×; output: 1,048,609 / 1,048,609 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 1.011×; output: 56,970 / 56,970 bytes.

Compression level 4

Benchmark index · Median overview · ← Level 3 · Level 5 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · tiny-text · repeated-1m · text-1m · random-64k · binary-64k · alice29 · random-1m

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 10.888×; output: 9 / 9 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 5.462×; output: 53 / 53 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 3.084×; output: 50 / 50 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 1.180×; output: 55,760 / 55,760 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.145×; output: 65,546 / 65,546 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.078×; output: 3,909 / 3,909 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 0.979×; output: 54,282 / 54,282 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 0.940×; output: 1,048,609 / 1,048,609 bytes.

Compression level 5

Benchmark index · Median overview · ← Level 4 · Level 6 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · tiny-text · repeated-1m · binary-64k · random-1m · random-64k · text-1m · alice29

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 11.276×; output: 9 / 9 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 5.223×; output: 53 / 53 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.975×; output: 50 / 50 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.275×; output: 1,270 / 1,270 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.236×; output: 1,048,609 / 1,048,609 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.178×; output: 65,546 / 65,546 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 1.052×; output: 54,185 / 54,185 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 0.998×; output: 53,372 / 53,372 bytes.

Compression level 6

Benchmark index · Median overview · ← Level 5 · Level 7 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · tiny-text · repeated-1m · binary-64k · random-64k · random-1m · text-1m · alice29

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 10.385×; output: 9 / 9 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 5.446×; output: 53 / 53 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 2.910×; output: 49 / 49 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.240×; output: 1,270 / 1,270 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.204×; output: 65,546 / 65,546 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.119×; output: 1,048,609 / 1,048,609 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 1.106×; output: 53,196 / 53,196 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 0.926×; output: 53,111 / 53,111 bytes.

Compression level 7

Benchmark index · Median overview · ← Level 6 · Level 8 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · tiny-text · repeated-1m · binary-64k · random-64k · random-1m · text-1m · alice29

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 10.730×; output: 9 / 9 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 5.314×; output: 53 / 53 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 2.427×; output: 49 / 49 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.334×; output: 1,270 / 1,270 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.154×; output: 65,546 / 65,546 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.042×; output: 1,048,609 / 1,048,609 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 0.986×; output: 52,804 / 52,804 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 0.929×; output: 53,195 / 53,195 bytes.

Compression level 8

Benchmark index · Median overview · ← Level 7 · Level 9 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · tiny-text · repeated-1m · binary-64k · random-64k · random-1m · text-1m · alice29

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 10.060×; output: 9 / 9 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 5.643×; output: 53 / 53 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 2.535×; output: 49 / 49 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.375×; output: 1,270 / 1,270 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.161×; output: 65,546 / 65,546 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.070×; output: 1,048,609 / 1,048,609 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 0.985×; output: 52,393 / 52,393 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 0.885×; output: 52,710 / 52,710 bytes.

Compression level 9

Benchmark index · Median overview · ← Level 8 · Level 10 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · tiny-text · repeated-1m · binary-64k · random-64k · random-1m · text-1m · alice29

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 10.021×; output: 9 / 9 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 5.109×; output: 53 / 53 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 2.345×; output: 49 / 49 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.279×; output: 1,270 / 1,270 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.253×; output: 65,546 / 65,546 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.102×; output: 1,048,609 / 1,048,609 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 1.090×; output: 52,381 / 52,381 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 1.006×; output: 52,188 / 52,188 bytes.

Compression level 10

Benchmark index · Median overview · ← Level 9 · Level 11 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · tiny-text · repeated-1m · binary-64k · random-64k · random-1m · alice29 · text-1m

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 10.012×; output: 9 / 9 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 5.047×; output: 53 / 53 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 2.447×; output: 49 / 49 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.270×; output: 1,270 / 1,270 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.251×; output: 65,546 / 65,546 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.169×; output: 1,048,609 / 1,048,609 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 0.973×; output: 51,894 / 51,894 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 0.936×; output: 52,148 / 52,148 bytes.

Compression level 11

Benchmark index · Median overview · ← Level 10 · Level 12 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · repeated-1m · tiny-text · binary-64k · random-1m · alice29 · text-1m · random-64k

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 6.068×; output: 9 / 9 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 2.231×; output: 49 / 49 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 2.015×; output: 53 / 53 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.189×; output: 1,276 / 1,276 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.140×; output: 1,048,609 / 1,048,609 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 1.072×; output: 51,491 / 51,491 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 1.041×; output: 52,034 / 52,034 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.011×; output: 65,546 / 65,546 bytes.

Compression level 12

Benchmark index · Median overview · ← Level 11 · Level 13 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · repeated-1m · tiny-text · binary-64k · random-64k · random-1m · text-1m · alice29

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 6.807×; output: 9 / 9 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 2.238×; output: 49 / 49 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 1.893×; output: 53 / 53 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.271×; output: 1,276 / 1,276 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.097×; output: 65,546 / 65,546 bytes.

random-1m

1 MiB of deterministic xorshift64 pseudorandom bytes. Input: 1,048,576 bytes.

mzstd speed / C: 1.091×; output: 1,048,609 / 1,048,609 bytes.

text-1m

Alice text repeated cyclically to 1 MiB. Input: 1,048,576 bytes.

mzstd speed / C: 1.080×; output: 52,034 / 52,034 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 0.880×; output: 51,492 / 51,492 bytes.

Compression level 13

Benchmark index · Median overview · ← Level 12 · Level 14 →

Recorded run: long-20261109 (2026-11-09). Raw results · Environment · Run analysis.

Median across all datasets

Each of the 8 datasets has equal weight, including empty and tiny input. Speed / C = C mean latency / implementation mean latency; output / C = implementation bytes / C bytes. We take the median of these 8 per-dataset ratios separately for speed and size. 1× matches C; higher speed and lower output are better. These are medians across datasets, not median sample latencies, and do not imply statistical significance.

Dataset details

Ordered by mzstd speed / C, highest first; all 8 datasets are shown. This order uses recorded mean latency, not statistical significance. Bars start at zero.

empty · repeated-1m · tiny-text · binary-64k · alice29 · random-64k · text-1m · random-1m

empty

Empty input; measures API overhead. Input: 0 bytes.

mzstd speed / C: 6.491×; output: 9 / 9 bytes.

repeated-1m

1 MiB of repeated a bytes. Input: 1,048,576 bytes.

mzstd speed / C: 3.155×; output: 50 / 50 bytes.

tiny-text

44-byte quick-brown-fox sentence. Input: 44 bytes.

mzstd speed / C: 1.570×; output: 53 / 53 bytes.

binary-64k

64 KiB of structured binary bytes: ((i / 16) XOR i) modulo 256. Input: 65,536 bytes.

mzstd speed / C: 1.201×; output: 1,259 / 1,259 bytes.

alice29

Alice in Wonderland text from the Canterbury corpus (testdata/alice29.txt). Input: 152,089 bytes.

mzstd speed / C: 1.144×; output: 49,705 / 49,705 bytes.

random-64k

64 KiB prefix of the deterministic pseudorandom corpus. Input: 65,536 bytes.

mzstd speed / C: 1.050×; output: 65,546 / 65,546 bytes.