OpenZL compression and decompression in safe Rust

A round trip

[dependencies]
mopenzl = "0.2.3"

use mopenzl::{Compressor, Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = "hello, smaller world! ".repeat(1000);
    let mut encoder = Compressor::new(EncoderConfig::default().with_level(Level::try_from(6)?))?;
    let frame = encoder.compress(input.as_bytes())?;

    let mut decoder = Decompressor::default();
    assert_eq!(decoder.decompress(&frame)?, input.as_bytes());
    Ok(())
}

Requires Rust 1.89 or later. Without a graph the compressor runs the default, compress_generic, which uses Zstandard for serial bytes. Getting started continues from here.

Tell it the shape of your data

OpenZL’s advantage over a general-purpose compressor is typed input. Say that a buffer holds 32-bit integers, and a numeric graph transforms them before an entropy coder sees them:

use mopenzl::graph::{CompressorGraph, StandardGraph};
use mopenzl::input::{Input, Numbers};
use mopenzl::{Compressor, DataType, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 10 000 slowly growing 32-bit timestamps, little-endian.
    let values: Vec<u8> = (0..10_000u32)
        .flat_map(|i| (1_700_000_000 + i * 3).to_le_bytes())
        .collect();
    let numbers = Numbers::try_from((&values[..], 4))?;

    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(CompressorGraph::standard(StandardGraph::SelectNumeric))
        .build()?;
    let frame = compressor.compress_inputs(&[Input::from(numbers)])?;
    println!("{} -> {} bytes", values.len(), frame.len());

    let outputs = Decompressor::default().decompress_typed(&frame)?;
    assert_eq!(outputs[0].data_type(), DataType::Numeric);
    assert_eq!(outputs[0].width(), 4);
    assert_eq!(outputs[0].content(), values);
    Ok(())
}

The decoder needs no graph: the frame records the codecs that produced it. Structured data covers numbers, records and strings.

Pick a graph

Graphs has the full table, and codecs lists every node and the Cargo feature that enables it. You can also compose nodes yourself with GraphBuilder or register a custom codec.

Find your path

What you can rely on

• Byte-identical serial encoder. For the same input, graph and global parameters, frames are exactly those of OpenZL v0.3.0 at format versions 21 to 27. See what is claimed and how it is checked.
• Safe implementation, not a C wrapper. Production code forbids unsafe. Zstandard comes from the safe-Rust mzstd crate; every other codec is ported here. The C library is only a test and benchmark oracle.
• No hidden I/O or threads. Sessions touch only the buffers you pass in, and parallel compression hands tasks to your scheduler. The library does not create a thread pool.
• Measured against C. The benchmarks show where mopenzl is faster than C and where it is still behind.

Framing requires the opt-in framed feature. The book tracks main, including unreleased changes; documented recipes are checked against the local checkout.

Getting started

Build a complete round trip, then choose the API that fits your application. Requires Rust 1.89+.

Install

cargo new openzl-demo
cd openzl-demo
cargo add mopenzl@0.2.3

Or add this to Cargo.toml:

[dependencies]
mopenzl = "0.2.3"

Compress and decompress

Replace src/main.rs with this complete program:

use mopenzl::{Compressor, Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = "hello, smaller world! ".repeat(1000);
    let config = EncoderConfig::default().with_level(Level::try_from(5)?);
    let mut encoder = Compressor::new(config)?;
    let compressed = encoder.compress(input.as_bytes())?;

    let mut decoder = Decompressor::new(Default::default());
    let restored = decoder.decompress(&compressed)?;
    assert_eq!(restored, input.as_bytes());

    println!("{} input bytes → {} compressed bytes", input.len(), compressed.len());
    Ok(())
}

cargo run --release

Choose the level explicitly: the default is 6 and accepted values are -131072 through 22. Graphs interpret the level according to their codecs; numeric selection changes at level 7. Compare speed and output size on your data; the benchmark pages show results by graph.

Choose a graph or codec

OpenZL describes compression as a graph of codec nodes. The default serial path uses Zstandard; typed data can use numeric, record and string graphs. Install a standard graph with Compressor::builder(config).with_graph(...), or compose nodes through GraphBuilder. The decoder reads the codec layout from the frame. Start with graphs, codecs, and typed inputs before tuning the compression level.

Reuse the workspace and output

A codec retains working memory between calls. compress_into appends a stream and returns its range; clear the destination when you want to reuse it for a new payload.

use mopenzl::{Compressor, Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut encoder = Compressor::new(
        EncoderConfig::default().with_level(Level::try_from(5)?),
    )?;
    let mut decoder = Decompressor::new(Default::default());
    let mut output = Vec::new();

    for payload in [b"first message".as_slice(), b"second message".as_slice()] {
        output.clear();
        let stream = encoder.compress_into(payload, &mut output)?;
        assert_eq!(decoder.decompress(&output[stream])?, payload);
    }
    Ok(())
}

See retention policies to control which workspace allocations remain after an operation. Caller output capacity is separate.

Stream through Write

Write plain input into the adapter, then call finish to obtain a complete OpenZL stream. Dropping the writer does not finalize it.

use mopenzl::{Compressor, Decompressor, EncoderConfig, InputSize, Level};
use std::io::Write;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"a payload arriving in two writes";
    let mut encoder = Compressor::new(
        EncoderConfig::default().with_level(Level::try_from(5)?),
    )?;
    let stream = InputSize::Exact(input.len() as u64).into();
    let mut writer = encoder.writer(Vec::new(), stream);
    writer.write_all(&input[..10])?;
    writer.write_all(&input[10..])?;
    let compressed = writer.finish().map_err(mopenzl::io::FinishError::into_error)?;

    let mut decoder = Decompressor::new(Default::default());
    assert_eq!(decoder.decompress(&compressed)?, input);
    Ok(())
}

For a retryable sink failure, FinishError::into_parts returns the error and writer. into_error discards the writer, as this in-memory example does. The streaming guide explains completion, backpressure, reader read-ahead, and direct sessions.

Choose features

The default features enable std, both directions and all-codecs. To build one direction, select individual codecs, drop std, or enable framed containers, see Cargo features. The book tracks main; use a local or Git dependency for APIs not yet released.

Next steps

• Sans I/O compression and decompression: caller buffers, progress counts, and completion.
• Read / Write I/O: readers and writers for raw, parallel, and framed streams.
• Parallel compression: scoped threads, Rayon, files, and streaming.
• Framing containers: multiple resources, metadata, and indexed reads.
• Graphs, codecs and structured data: choose what runs inside a frame.
• Choose your API: every entry point for both directions, with its guide.
• Configuration and features: level, format version, checksums, and Cargo features.
• Decompression and limits: budgets for untrusted input.
• Examples: pick a recipe and run the repository examples.

Choose your API

Compression and decompression expose the same I/O shapes. Pick the row that matches how your application already moves bytes, then follow its guide. New to the crate? Start with getting started; to choose what runs inside a frame, read graphs.

Both codecs

The decoder needs no graph: a frame records the codecs that produced it. Compressor and Decompressor own reusable working state. Keep one alive across payloads when allocation reuse matters.

Parallel compression

ParallelCompressor lives in mopenzl::compressor::parallel and is unavailable with no_std. It splits one input into segments and hands them to your executor as tasks; the library does not create a thread pool. Streaming buffers compressed chunks until finish, when the total input size is known.

See parallel compression for scoped threads, Rayon, sources, staging, and streaming.

Guides by task

Errors

ConfigError covers configuration, EncodeError covers compression (including EncodeError::Graph(GraphError) for graph failures), DecodeError covers decompression, graph::GraphError covers graph construction, and dict::DictError covers dictionaries. I/O adapters return std::io::Error; writer finalization preserves the writer in FinishError.

Reference

• API reference on docs.rs
• Compatibility and validation: what is byte-identical to OpenZL v0.3.0, and how it is checked
• Benchmark results per graph on Silesia, with raw data
• Architecture: how the implementation works
• Development and benchmarking for contributors

Examples

The recipes below include complete programs and round-trip assertions. Examples use the default std, compression, and decompression features unless stated otherwise. The documentation workflow compiles and runs their Rust code against the current checkout.

Run from the repository

Initialize the C reference submodule used by development builds:

git submodule update --init --recursive
cargo run --release --example compress

The parallel example uses caller-owned scoped threads, reuses its compressor and verifies every assembled frame:

cargo run --release --example parallel

For a parallel batch, all taken tasks must run or be dropped before finalization. For a streaming adapter, keep workers running while the adapter blocks. Finish ordinary/parallel writers and the enclosing framed container explicitly. Framed resource commands accept complete input rather than resource writer guards.

Verify the documented recipes

python3 scripts/docs.py test-examples

This creates a temporary Cargo consumer under target/, depends on the local crate with framed enabled and on Rayon, and runs each fenced Rust program from all user-guide chapters, including graphs, codecs, typed inputs, dictionaries, parallel, framing and Sans I/O. It does not exercise the codec with --all-features, because that would also enable no_std and remove the std APIs these recipes demonstrate.

Graphs

OpenZL compresses by routing typed streams through a graph of codecs. This guide explains how the graphs of this crate work, which standard graphs exist, and how to build your own with mopenzl::graph. Frame-level mechanics live in compression graphs.

Types · Standard graphs · GraphBuilder · Parameters · Selectors, function graphs, segmenters · Chunking · try_graph · Permissive mode · Sessions

Typed inputs and outputs

Every stream has a DataType:

The constructors validate the layout (widths, whole elements, lengths summing to the content size) and return InputError otherwise. Numbers are always little-endian bytes. Input::with_metadata(&[(id, value)]) attaches integer metadata that graphs can read, such as the tag the clustering graph groups by; metadata is never written to the frame.

Compressor::compress_inputs(&[Input]) writes one frame from one or more inputs and Decompressor::decompress_typed returns one TypedOutput per input, in order. Compressor::compress is compress_inputs with a single serial input.

A node is one codec invocation: it takes typed streams and produces typed streams (delta_int: numeric to numeric; float32_deconstruct: numeric to a struct plus a serial; tokenize: any to an alphabet plus numeric indices). A graph is anything that accepts streams and decides what happens to them: a node plus successor graphs, a selector, a function graph, a segmenter, or store. Nodes do no implicit type conversion when you run them yourself; the engine converts implicitly only when a stream reaches store (numeric and struct data are converted to serial bytes first, which emits a conversion codec).

Standard graphs

StandardGraph names the graphs of the reference’s registry with their numeric IDs. CompressorGraph::standard(graph) (or CompressorGraph::from) makes a ready compressor graph, and Compressor::builder(config).with_graph(..) installs it. Without a graph the compressor runs the default: compress_generic.

use mopenzl::graph::{CompressorGraph, StandardGraph};
use mopenzl::input::{Input, Records, Strings};
use mopenzl::{Compressor, DataType, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let text = b"alpha beta gamma delta alpha beta gamma delta ".repeat(20);
    let lens: Vec<u32> = vec![46; 20];
    let records: Vec<u8> = (0..4000u32)
        .flat_map(|i| [(i % 7) as u8, 1, 2, 3])
        .collect();
    let inputs = [
        Input::from(&text[..]),
        Input::from(Records::try_from((&records[..], 4))?),
        Input::from(Strings::try_from((&text[..], &lens[..]))?),
    ];

    // `compress_generic` takes any number of inputs and routes each by type.
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(CompressorGraph::standard(StandardGraph::CompressGeneric))
        .build()?;
    let frame = compressor.compress_inputs(&inputs)?;

    let outputs = Decompressor::default().decompress_typed(&frame)?;
    let types: Vec<DataType> = outputs.iter().map(|o| o.data_type()).collect();
    assert_eq!(
        types,
        [DataType::Serial, DataType::Struct, DataType::String]
    );
    assert_eq!(outputs[0].content(), text);
    assert_eq!(outputs[1].width(), 4);
    assert_eq!(outputs[2].string_lens(), lens);
    Ok(())
}

Which graph when

Start with CompressGeneric (the default) and move to a specific graph when you know your data. S, St, N and T below mean serial, struct, numeric and string inputs. Graphs that take “any” accept several inputs.

Public graphs

IDs from 32 on are private in the reference but exist here with stable numbers; they are useful as successors and for reading the registry:

Graphs that need successors

Some standard graphs route to graphs you must supply with GraphBuilder::parameterize_graph(graph, GraphParams::default().with_graphs([..])); run bare they fail (“needs exactly three custom graphs”). Each variant’s rustdoc says what it needs; in short:

FieldLz and Lz need none; their custom graphs are optional overrides picked by index through params::field_lz and params::lz. SegmentNumeric needs none. Every other standard graph is self-contained.

StandardGraph and StandardNode are #[non_exhaustive]; TryFrom<u32> and From<..> for u32 convert to and from the reference’s IDs. Every standard graph needs the codecs it uses: a graph whose codec feature is off fails like a failing codec (see permissive mode and codecs).

Building graphs

GraphBuilder starts with only the standard graphs and nodes and derives new ones. Each call returns a GraphId or NodeId that is meaningful to that builder only; build(start) freezes everything into a cheap-to-clone, thread-shareable CompressorGraph rooted at start.

use mopenzl::graph::{GraphBuilder, StandardGraph, StandardNode};
use mopenzl::input::{Input, Numbers};
use mopenzl::{Compressor, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut builder = GraphBuilder::new();
    // delta -> zigzag -> bit-pack: three nodes in a chain, then store.
    let packed = builder.pipeline(
        &[
            StandardNode::DeltaInt.into(),
            StandardNode::Zigzag.into(),
            StandardNode::BitpackInt.into(),
        ],
        StandardGraph::Store,
    )?;
    let graph = builder.build(packed)?;

    let values: Vec<u8> = (0..2000u32)
        .flat_map(|i| (500 + i % 17).to_le_bytes())
        .collect();
    let numbers = Numbers::try_from((&values[..], 4))?;
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let frame = compressor.compress_inputs(&[Input::from(numbers)])?;
    assert!(frame.len() < values.len() / 2);

    let outputs = Decompressor::default().decompress_typed(&frame)?;
    assert_eq!(outputs[0].content(), values);
    Ok(())
}

Builders report mistakes early: a successor count that does not match a node’s outputs, an unknown or foreign ID, an empty pipeline or an unknown start graph is a GraphError (Invalid, UnknownId). Failures while a graph runs surface from the compressor as EncodeError::Graph(GraphError::Execution(..)), or GraphError::User(..) when your own graph returned the error.

Parameterized nodes and graphs

Nodes and some graphs take local parameters: integer, byte-blob or object parameters keyed by the reference’s IDs. LocalParams::default().with_int(id, v) and .with_bytes(id, bytes) build them; giving a node or graph parameters replaces the ones it had. The global level reaches Zstandard unchanged, and the Zstandard node applies its own integer parameters after it, as raw ZSTD_c_* IDs (params::zstd::COMPRESSION_LEVEL is 100), in list order.

The IDs are the reference’s, and mopenzl::graph::params names them: one module per codec or graph (zstd, lz, lz4, field_lz, entropy, divide_by, bitunpack, partition, sparse_num, sentinel, tokenize, split, dispatch, mux_lengths, segmenters, sddl, sddl2, clustering, …). Each constant cites the C macro or enum it mirrors (ZL_ENTROPY_MIN_GAIN_BYTES_PID, ZL_LzParam_windowLog, …) and its doc says whether the parameter is an integer or a copy (bytes) parameter.

use mopenzl::graph::params::zstd;
use mopenzl::graph::{GraphBuilder, GraphParams, LocalParams, StandardGraph, StandardNode};
use mopenzl::{Compressor, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut builder = GraphBuilder::new();
    // The Zstandard node with its compression level pinned to 19.
    let zstd19 = builder.parameterize_node(
        StandardNode::Zstd,
        LocalParams::default().with_int(zstd::COMPRESSION_LEVEL, 19),
    )?;
    let strong = builder.static_graph(zstd19, &[StandardGraph::Store.into()])?;
    // A copy of `brute_force`, with its candidates replaced.
    let race = builder.parameterize_graph(
        StandardGraph::BruteForce,
        GraphParams::default()
            .with_graphs([
                strong,
                StandardGraph::Lz4.into(),
                StandardGraph::Store.into(),
            ])
            .with_name("zstd19-or-lz4"),
    )?;
    let graph = builder.build(race)?;

    let input = b"parameters replace, they do not merge ".repeat(500);
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let frame = compressor.compress(&input)?;
    assert!(frame.len() < input.len() / 10);
    assert_eq!(Decompressor::default().decompress(&frame)?, input);
    Ok(())
}

GraphBuilder::brute_force(&[..]) is the shorthand for the parameterize_graph call above. LocalParams::with_string_lengths supplies the lengths of the set_string_lens node. The key local parameters of every codec are in codecs.

Custom graphs in Rust

Three traits mirror the reference’s callbacks. All are Send + Sync + 'static so a built graph is shareable between threads.

Selector

A selector looks at a stream and returns one of its successors (or any other graph of the compressor). SelectorContext::try_graph compresses the stream with a candidate and returns the resulting frame size, so a selector can measure instead of guess.

use mopenzl::graph::{
    GraphBuilder, GraphId, InputView, Selector, SelectorContext, StandardGraph, TypeSet,
};
use mopenzl::{Compressor, DataType, Decompressor, EncoderConfig};

/// Picks the successor that produces the smallest frame.
struct Smallest;

impl Selector for Smallest {
    fn select(
        &self,
        cx: &mut SelectorContext<'_>,
        input: InputView<'_>,
        successors: &[GraphId],
    ) -> GraphId {
        let mut best = successors[0];
        let mut best_size = usize::MAX;
        for &graph in successors {
            let size = cx.try_graph(&input, graph).unwrap_or(usize::MAX);
            if size < best_size {
                (best, best_size) = (graph, size);
            }
        }
        best
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut builder = GraphBuilder::new();
    let selector = builder.selector(
        "smallest",
        Smallest,
        TypeSet::from(DataType::Serial),
        &[
            StandardGraph::Zstd.into(),
            StandardGraph::Lz4.into(),
            StandardGraph::Store.into(),
        ],
    )?;
    let graph = builder.build(selector)?;

    let input = b"selectors measure instead of guessing ".repeat(300);
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let frame = compressor.compress(&input)?;
    assert_eq!(Decompressor::default().decompress(&frame)?, input);
    Ok(())
}

A selector cannot fail: return a graph that always works (such as Store) as the fallback. SelectorContext also exposes level(), format_version(), depth(), is_node_supported(node), local_int(id) and set_successor_params(..).

Function graph

A function graph gets its inputs as edges, runs nodes on them and assigns every resulting edge a destination graph. Nodes run at the moment you ask; successors run after the function returns, in the order of the edge handles (the inputs first, then node outputs in creation order), not in the order you call set_destination. An edge left without a destination is an error (or triggers the permissive fallback). GraphContext::data(edge) reads a stream, custom_graphs() and custom_nodes() return the lists you registered, and run_node_with and set_destination_with override parameters at run time. run_split_node, run_dispatch_node and run_dispatch_string_node run the split and dispatch nodes with their runtime instructions.

use mopenzl::graph::{
    Edge, FunctionGraph, GraphBuilder, GraphContext, GraphError, StandardGraph, StandardNode,
};
use mopenzl::input::{Input, Strings};
use mopenzl::{Compressor, DataType, Decompressor, EncoderConfig};

/// Separates strings into lengths and content, and compresses each its own way.
struct SplitStrings;

impl FunctionGraph for SplitStrings {
    fn compress(&self, cx: &mut GraphContext<'_>, inputs: &[Edge]) -> Result<(), GraphError> {
        let parts = cx.run_node(StandardNode::SeparateStringComponents, inputs)?;
        let graphs = cx.custom_graphs();
        cx.set_destination(&parts[0..1], graphs[0])?; // content
        cx.set_destination(&parts[1..2], graphs[1])?; // lengths
        Ok(())
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut builder = GraphBuilder::new();
    let function = builder.function_graph(
        "split_strings",
        SplitStrings,
        &[DataType::String.into()],
        false,
        &[
            StandardGraph::Zstd.into(),
            StandardGraph::NumericCompress.into(),
        ],
        &[],
    )?;
    let graph = builder.build(function)?;

    let text = b"red green blue red green blue ".repeat(100);
    let lens: Vec<u32> = [3, 5, 4, 3, 5, 4]
        .iter()
        .cycle()
        .take(600)
        .copied()
        .collect();
    let content_len: u32 = lens.iter().sum();
    let strings = Strings::try_from((&text[..content_len as usize], &lens[..]))?;

    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let frame = compressor.compress_inputs(&[Input::from(strings)])?;
    let outputs = Decompressor::default().decompress_typed(&frame)?;
    assert_eq!(outputs[0].string_lens(), lens);
    assert_eq!(outputs[0].content(), &text[..content_len as usize]);
    Ok(())
}

Segmenter

A segmenter cuts the whole input into chunks, each compressed independently through a successor, and writes one frame. See chunking.

use mopenzl::graph::{GraphBuilder, GraphError, Segmenter, SegmenterContext, StandardGraph};
use mopenzl::{Compressor, DataType, Decompressor, EncoderConfig};

/// Cuts the input in thirds.
struct Thirds;

impl Segmenter for Thirds {
    fn segment(&self, cx: &mut SegmenterContext<'_>) -> Result<(), GraphError> {
        let total = cx.remaining()[0];
        let successor = cx.custom_graphs()[0];
        let third = total.div_ceil(3).max(1);
        let mut left = total;
        while left > 0 {
            let take = third.min(left);
            cx.process_chunk(&[take], successor)?;
            left -= take;
        }
        if total == 0 {
            cx.process_chunk(&[0], successor)?;
        }
        Ok(())
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut builder = GraphBuilder::new();
    let thirds = builder.segmenter(
        "thirds",
        Thirds,
        &[DataType::Serial.into()],
        &[StandardGraph::CompressGeneric.into()],
    )?;
    let graph = builder.build(thirds)?;

    let input = b"three chunks, three independent graphs ".repeat(400);
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let frame = compressor.compress(&input)?;
    assert_eq!(Decompressor::default().decompress(&frame)?, input);
    Ok(())
}

Chunking and segmenters

A frame is a sequence of chunks, each with its own codec layout. The default graph writes the whole input as one chunk (Chunking::Whole, which still splits every 2 GiB). A segmenter cuts several. The standard ones:

• EncoderConfig::with_chunking(Chunking::Fixed(ChunkSize)) is the serial segmenter of the reference CLI: chunks of ChunkSize input bytes (32 KiB to i32::MAX, default 16 MiB), the last one shorter. It wraps whatever graph you installed.
• GraphBuilder::serial_segmenter(chunk, successor) is the same as a start graph of your own, with a chosen successor per chunk. A start graph that is already a segmenter cannot be combined with Chunking::Fixed (an EncodeError::Graph).
• GraphBuilder::num_from_serial_segmenter(width, chunk, successor) cuts serial input aligned to width-byte integers and compresses each chunk as numeric data (the default successor, InterpretNum8Compress to InterpretNum64Compress, reads each serial chunk as numbers and compresses them; a successor you pass receives the chunk as serial data, so it must accept serial input and convert it itself, for example a static graph that starts with ConvertSerialToNumLe32).
• SegmentNumeric cuts numeric input every 16 MiB; the SDDL2 graph is a segmenter that cuts along the segments of its program.

Chunks cap memory: with a declared InputSize::Exact both the encoder session and the decoder work one chunk at a time. Chunking needs format version 21 or newer, which every supported version has.

use mopenzl::graph::GraphBuilder;
use mopenzl::{ChunkSize, Compressor, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut builder = GraphBuilder::new();
    // 8-byte integers, 64 KiB chunks; each chunk is read as numbers and compressed.
    let segmenter = builder.num_from_serial_segmenter(8, ChunkSize::try_from(64 << 10)?, None)?;
    let graph = builder.build(segmenter)?;

    let input: Vec<u8> = (0..40_000u64)
        .flat_map(|i| (i * 1000).to_le_bytes())
        .collect();
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let frame = compressor.compress(&input)?;
    assert_eq!(Decompressor::default().decompress(&frame)?, input);
    Ok(())
}

Trial compression: try_graph

SelectorContext::try_graph(&input, graph) and GraphContext::try_graph(edge, graph) compress the stream with graph in a throwaway context (checksums off, no comment) and return the size of the complete frame it would produce, or an error when the graph fails. brute_force is built on it. Trials cost a full compression each; use them on the successors that matter. The selector example shows the idiom.

Permissive mode

Normally a failing node, graph or successor fails the compression. With EncoderConfig::with_permissive(true), a failing successor outside a segmenter is rolled back and its input recompressed with compress_generic, as in the reference. This is the net under graphs that cannot always apply: Constant on data that is not constant, an MlSelector whose model does not match its successors, a codec whose Cargo feature is off.

use mopenzl::graph::{CompressorGraph, StandardGraph};
use mopenzl::{Compressor, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let not_constant = b"this is certainly not one repeated byte".repeat(10);
    let graph = CompressorGraph::standard(StandardGraph::Constant);

    let mut strict = Compressor::builder(EncoderConfig::default())
        .with_graph(graph.clone())
        .build()?;
    assert!(strict.compress(&not_constant).is_err());

    let config = EncoderConfig::default().with_permissive(true);
    let mut permissive = Compressor::builder(config).with_graph(graph).build()?;
    let frame = permissive.compress(&not_constant)?;
    assert_eq!(Decompressor::default().decompress(&frame)?, not_constant);
    Ok(())
}

Two more nets always apply: a lone stream below min_stream_size (10) is stored, and, with store_on_expansion (the default), a chunk that would not shrink is rebuilt as store.

Sessions and graphs

Compressor::start and the std::io writer work with any graph, with these rules:

• The default graph and a serial segmenter (including Chunking::Fixed) compress incrementally, chunk by chunk, and match the one-shot bytes.
• Every other graph is buffered: the session holds its input until Operation::End, runs the graph once and then drains the output. A declared InputSize::Exact is checked at the end.
• Typed and multiple inputs go through compress_inputs; sessions, readers and writers take serial bytes.

See streaming.

Codecs

A codec is the decoder half of a node: every standard OpenZL transform or coder has a node (the encoder side, a StandardNode) and a decoder in this crate, and both live behind one Cargo feature. This page lists them. Wire formats, invariants and the deviations on malformed data are in decoder codecs; how nodes are wired into graphs is in graphs.

Types are serial (bytes), struct (fixed-width records), numeric (little-endian integers) and string. “Since” is the first format version that has the node. The crate writes versions 21 to 27, so a node with a “since” of 21 or less exists at every version you can select; nodes introduced at 23, 24, 26 or 27 (lz4, lz, the bit-split family, partition, sentinel, sparse_num, pivco_huffman) are not available when you pick an older format version.

Features

all-codecs (in default) enables every feature below. Without a codec’s feature the encoder never writes it (a chunk that would use it is stored; a graph that runs its node fails, falling back only in permissive mode) and the decoder reports frames that use it as DecodeError::Unsupported. The decoder parses the chunk layout in full either way, so the error names the missing codec before any stream is decoded.

Feature names with a hyphen are Cargo features (field-lz); the library selects them independently, so features = ["std", "decompression", "zstd", "delta"] decodes only those codecs. Several standard graphs need more than one feature (the entropy graph uses most coders); a missing one makes the graph fail at run time. The scripts/check_codec_features.py script builds every combination.

Nodes

StandardNode::X below is the variant name for the registry ID; run a node from a function graph with GraphContext::run_node, or in a chain with GraphBuilder::pipeline. “Key parameters” are local parameters (see parameterized nodes) whose IDs are the reference’s; nodes not listed take none that matter.

Numeric transforms

Dictionary-style transforms

Splitting, merging and routing

Type conversions

The conversion feature covers IDs 10-26 and 57. They change the type of a stream, not its bytes, and appear in frames whenever typed data meets a serial-only codec.

Entropy coders

These come from OpenZL’s own FSE and Huffman fork, not from mzstd.

LZ and general-purpose coders

use mopenzl::graph::params::{divide_by, zstd};
use mopenzl::graph::{GraphBuilder, LocalParams, StandardGraph, StandardNode};
use mopenzl::input::{Input, Numbers};
use mopenzl::{Compressor, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut builder = GraphBuilder::new();
    // divide_by with an explicit divisor of 1000, then zstd at level 19.
    let divide = builder.parameterize_node(
        StandardNode::DivideBy,
        LocalParams::default().with_bytes(divide_by::DIVIDE_BY, 1000u64.to_le_bytes().to_vec()),
    )?;
    let zstd19 = builder.parameterize_node(
        StandardNode::Zstd,
        LocalParams::default().with_int(zstd::COMPRESSION_LEVEL, 19),
    )?;
    let to_bytes = builder.static_graph(zstd19, &[StandardGraph::Store.into()])?;
    let start = builder.pipeline(
        &[divide, StandardNode::ConvertNumToSerialLe.into()],
        to_bytes,
    )?;
    let graph = builder.build(start)?;

    let values: Vec<u8> = (0..4096u32)
        .flat_map(|i| (i * 1000).to_le_bytes())
        .collect();
    let numbers = Numbers::try_from((&values[..], 4))?;
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let frame = compressor.compress_inputs(&[Input::from(numbers)])?;
    let outputs = Decompressor::default().decompress_typed(&frame)?;
    assert_eq!(outputs[0].content(), values);
    Ok(())
}

Custom codecs

Application-defined codecs are supported for compression and decompression at format versions 21 to 27. The custom-codec feature is included in all-codecs and enabled by default; it also works with no_std and alloc.

Implement custom_codec::CustomCodec:

• signature() declares the encoder’s input types, mandatory outputs and variable ports. Streams can be serial bytes, fixed-width records, numbers or strings. Numeric content is little-endian.
• encode() reads borrowed CodecInput views, commits outputs through EncoderContext::emit or emit_strings, and can send a private header with send_header. Every mandatory output must be committed exactly once.
• decode() receives the encoder’s output streams and private header, then restores the original inputs through DecoderContext::emit or emit_strings. The context validates widths, string lengths and the per-stream output budget.

Register the encoder with GraphBuilder::custom_codec(id, codec) and use the returned NodeId in a static graph, pipeline or function graph. Register the matching decoder with Decompressor::builder(config).with_custom_codec(id, codec)?. Both sides must agree on the wire ID, signature and encoded representation. Custom u32 IDs occupy a separate namespace from standard codec IDs; duplicate custom decoder IDs are rejected.

Custom nodes support parameterization and the regular one-shot and streaming APIs. Minimum-stream-size and store-on-expansion rules still apply: a small or expanding chunk may be stored instead of using the custom codec. Callbacks and signatures remain registered across operations and buffer trimming.

A frame carries the custom ID and its data, so the receiving application must register the decoder implementation. An unregistered ID, or a build without custom-codec, reports DecodeError::Unsupported { feature: Unsupported::CustomCodec { id } }. Custom dictionary materializers are not implemented.

See the API example for a complete implementation and registration example, and the custom codec specification for ownership, wire format, validation and error propagation.

Legacy codecs

Legacy codecs whose newest format version is below 21 (rolz, fastlz, zstd_fixed and the deprecated FSE and Huffman nodes) are not registered and decode as Unsupported::Codec.

Structured data

OpenZL’s advantage over a general-purpose compressor is knowing the shape of the data. Tell it that a buffer is an array of 32-bit timestamps, 16-byte records or a column of strings, and a graph can split, transform and entropy-code each part with the codec that fits. This guide covers the typed inputs, the standard graphs that fit each shape, clustering of related inputs, ML-selected successors and the SDDL description languages. Graphs explains the graph machinery.

Always decode typed frames with Decompressor::decompress_typed: the byte-oriented methods read one serial output.

Numeric data

Numbers::try_from((bytes, width)) wraps little-endian unsigned integers of width 1, 2, 4 or 8. For signed or floating-point data pass the bit patterns; the numeric codecs treat them as unsigned (zigzag, float32_deconstruct and friends are the bridges). Pick a graph by effort:

Higher levels cost more time: the transformer is only used from level 7, and it is a model’s best guess, not a guarantee: measure on your data (the example below prints the sizes).

use mopenzl::graph::{CompressorGraph, StandardGraph};
use mopenzl::input::{Input, Numbers};
use mopenzl::{Compressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // A slowly growing counter with a little noise, 64-bit.
    let values: Vec<u8> = (0..20_000u64)
        .flat_map(|i| (1_000_000 + i * 16 + (i * 2654435761) % 5).to_le_bytes())
        .collect();
    let input = Input::from(Numbers::try_from((&values[..], 8))?);

    for (name, graph, level) in [
        ("zstd", StandardGraph::Zstd, 6),
        ("field_lz", StandardGraph::FieldLz, 6),
        ("select_numeric, level 6", StandardGraph::SelectNumeric, 6),
        ("select_numeric, level 10", StandardGraph::SelectNumeric, 10),
    ] {
        let config = EncoderConfig::default().with_level(Level::try_from(level)?);
        let mut compressor = Compressor::builder(config)
            .with_graph(CompressorGraph::standard(graph))
            .build()?;
        // `Zstd` takes serial input and nodes do no implicit conversion, so it is
        // shown on the raw bytes.
        let frame = if graph == StandardGraph::Zstd {
            compressor.compress(&values)?
        } else {
            compressor.compress_inputs(&[input])?
        };
        println!("{name}: {} -> {} bytes", values.len(), frame.len());
    }
    Ok(())
}

Fixed-width records

Records::try_from((bytes, width)) wraps records of any width. Record-aware graphs treat each byte position of a record as its own column:

To compress each byte column of the records on its own, run the TransposeSplit node (one serial stream per byte of the record) from a function graph and send every column to a successor. GraphContext::run_split_node and the split_by_struct node cut records into fields of different widths, and SDDL describes whole layouts.

use mopenzl::graph::{
    Edge, FunctionGraph, GraphBuilder, GraphContext, GraphError, StandardGraph, StandardNode,
};
use mopenzl::input::{Input, Records};
use mopenzl::{Compressor, DataType, Decompressor, EncoderConfig};

/// One serial stream per byte of the record, each compressed on its own.
struct Columns;

impl FunctionGraph for Columns {
    fn compress(&self, cx: &mut GraphContext<'_>, inputs: &[Edge]) -> Result<(), GraphError> {
        let columns = cx.run_node(StandardNode::TransposeSplit, inputs)?;
        let successor = cx.custom_graphs()[0];
        for column in columns {
            cx.set_destination(&[column], successor)?;
        }
        Ok(())
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 16-byte records: a counter, a small enum, a constant tail.
    let mut bytes = Vec::new();
    for i in 0..5_000u32 {
        bytes.extend_from_slice(&i.to_le_bytes());
        bytes.extend_from_slice(&(i % 3).to_le_bytes());
        bytes.extend_from_slice(&[0xAB; 8]);
    }
    let records = Records::try_from((&bytes[..], 16))?;

    let mut builder = GraphBuilder::new();
    let columns = builder.function_graph(
        "columns",
        Columns,
        &[DataType::Struct.into()],
        false,
        &[StandardGraph::Zstd.into()],
        &[],
    )?;
    let graph = builder.build(columns)?;
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let frame = compressor.compress_inputs(&[Input::from(records)])?;
    println!("{} -> {} bytes", bytes.len(), frame.len());

    let outputs = Decompressor::default().decompress_typed(&frame)?;
    assert_eq!(outputs[0].data_type(), DataType::Struct);
    assert_eq!(outputs[0].width(), 16);
    assert_eq!(outputs[0].content(), bytes);
    Ok(())
}

Strings

Strings::try_from((content, &lengths)) pairs concatenated bytes with one u32 length per string; the lengths must sum to the content size. StringCompress (what CompressGeneric uses) separates the two components and compresses the content as serial and the lengths as numeric. TryParseInt parses columns of decimal numbers as integers when they are canonical decimals; it takes three successors you provide (GraphParams::with_graphs): the integers, the strings that did not parse, and the indices that tell the two apart. The tokenize_string nodes build an alphabet for columns of repeated values.

use mopenzl::graph::{GraphBuilder, GraphParams, StandardGraph};
use mopenzl::input::{Input, Strings};
use mopenzl::{Compressor, DataType, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // A column of numbers written as text.
    let column: Vec<String> = (0..3_000u32)
        .map(|i| (10_000 + i * 7).to_string())
        .collect();
    let content: Vec<u8> = column.concat().into_bytes();
    let lens: Vec<u32> = column.iter().map(|s| s.len() as u32).collect();
    let strings = Strings::try_from((&content[..], &lens[..]))?;

    // successors: [integers, strings that did not parse, indices]
    let mut builder = GraphBuilder::new();
    let parse = builder.parameterize_graph(
        StandardGraph::TryParseInt,
        GraphParams::default().with_graphs([
            StandardGraph::NumericCompress.into(),
            StandardGraph::StringCompress.into(),
            StandardGraph::NumericCompress.into(),
        ]),
    )?;
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(builder.build(parse)?)
        .build()?;
    let frame = compressor.compress_inputs(&[Input::from(strings)])?;
    println!("{} -> {} bytes", content.len(), frame.len());

    let outputs = Decompressor::default().decompress_typed(&frame)?;
    assert_eq!(outputs[0].data_type(), DataType::String);
    assert_eq!(outputs[0].string_lens(), lens);
    assert_eq!(outputs[0].content(), content);
    Ok(())
}

Clustering related inputs with tags

A frame may carry several inputs of different types. The Clustering graph groups them by (type, element width, tag), concatenates each group with a concat node and compresses it with the successor its cluster names, so inputs that look alike compress together. Tags are integer metadata with id CLUSTERING_TAG_METADATA_ID attached with Input::with_metadata; metadata never reaches the frame.

ClusteringConfig describes the clusters:

• with_cluster(type, width, successor, codec, tags) makes a cluster of the inputs of that type and width whose tag is in tags. successor and codec index the successor graphs and clustering codecs given to the builder (clustering uses the four standard concat nodes: serial 0, struct 1, numeric 2, string 3).
• with_type_default(type, width, successor, codec) handles inputs of that type whose tag no cluster lists.
• Inputs matching neither form their own cluster, compressed by CompressGeneric.
• The width of a serial input is 1 and that of a string input 0.

The configuration serializes to the reference’s CBOR blob (Vec::from(&config), ClusteringConfig::try_from(&bytes[..])), so a configuration trained elsewhere loads here.

use mopenzl::graph::{CLUSTERING_TAG_METADATA_ID, ClusteringConfig, GraphBuilder, StandardGraph};
use mopenzl::input::{Input, Numbers};
use mopenzl::{Compressor, DataType, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let latency: Vec<u8> = (0..4000u32)
        .flat_map(|i| (20 + i % 9).to_le_bytes())
        .collect();
    let latency_p99: Vec<u8> = (0..4000u32)
        .flat_map(|i| (90 + i % 13).to_le_bytes())
        .collect();
    let ids: Vec<u8> = (0..4000u32)
        .flat_map(|i| (1_000_000 + i).to_le_bytes())
        .collect();
    let tags = [
        [(CLUSTERING_TAG_METADATA_ID, 1)],
        [(CLUSTERING_TAG_METADATA_ID, 2)],
        [(CLUSTERING_TAG_METADATA_ID, 3)],
    ];
    let inputs = [
        Input::from(Numbers::try_from((&latency[..], 4))?).with_metadata(&tags[0]),
        Input::from(Numbers::try_from((&latency_p99[..], 4))?).with_metadata(&tags[1]),
        Input::from(Numbers::try_from((&ids[..], 4))?).with_metadata(&tags[2]),
    ];

    // Tags 1 and 2 share a cluster (successor 0); tag 3 falls to the type default.
    let config = ClusteringConfig::default()
        .with_cluster(DataType::Numeric, 4, 0, 2, [1, 2])
        .with_type_default(DataType::Numeric, 4, 1, 2);
    let mut builder = GraphBuilder::new();
    let clustering = builder.clustering(
        &config,
        &[
            StandardGraph::SelectNumeric.into(),
            StandardGraph::FieldLz.into(),
        ],
    )?;
    let graph = builder.build(clustering)?;

    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let frame = compressor.compress_inputs(&inputs)?;

    let outputs = Decompressor::default().decompress_typed(&frame)?;
    assert_eq!(outputs.len(), 3);
    assert_eq!(outputs[0].content(), latency);
    assert_eq!(outputs[1].content(), latency_p99);
    assert_eq!(outputs[2].content(), ids);
    Ok(())
}

ML selectors

MlSelector sends a numeric input to the successor a gradient-boosted-tree model picks from statistics of the data. The statistics (features) are nbElts, eltWidth, cardinality, cardinality_upper, cardinality_lower, range_size, mean, variance, stddev, skewness and kurtosis; a name outside this list reads as missing.

MlSelectorConfig wraps the model:

• MlSelectorConfig::try_from(&GbtModel) validates a model as the reference does and serializes it; MlSelectorConfig::try_from(&blob[..]) loads a trained CBOR blob (the model is not validated, so a bad one may fail when the graph runs).
• A model with one forest is binary (two successors; class 1 when the summed leaves are not negative); otherwise there is one forest per successor and the highest score wins. GbtNode::split(feature, threshold, left, right, missing) and GbtNode::leaf(score) build trees; a feature strictly below its threshold goes left, and children must come after their parent.
• GraphBuilder::untrained_ml_selector(&successors) always picks the first successor. The number of successors must match the model when the graph runs; otherwise it fails (and a permissive compression falls back to compress_generic).

use mopenzl::graph::{GbtModel, GbtNode, GraphBuilder, MlSelectorConfig, StandardGraph};
use mopenzl::input::{Input, Numbers};
use mopenzl::{Compressor, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // One tree on `nbElts`: fewer than 64 elements -> class 0 (store), else class 1.
    let tree = vec![
        GbtNode::split(0, 64.0, 1, 2, 1),
        GbtNode::leaf(-1.0),
        GbtNode::leaf(1.0),
    ];
    let model = GbtModel {
        forests: vec![vec![tree]],
        successors: 2,
        features: vec!["nbElts".to_string()],
    };
    let config = MlSelectorConfig::try_from(&model)?;

    let mut builder = GraphBuilder::new();
    let selector = builder.ml_selector(
        &config,
        &[StandardGraph::Store.into(), StandardGraph::FieldLz.into()],
    )?;
    let graph = builder.build(selector)?;
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;

    let mut decoder = Decompressor::default();
    for count in [16u32, 4096] {
        let values: Vec<u8> = (0..count).flat_map(|i| (i % 5).to_le_bytes()).collect();
        let numbers = Numbers::try_from((&values[..], 4))?;
        let frame = compressor.compress_inputs(&[Input::from(numbers)])?;
        println!("{count} values -> {} bytes", frame.len());
        assert_eq!(decoder.decompress_typed(&frame)?[0].content(), values);
    }
    Ok(())
}

SDDL and SDDL2

The Simple Data Description Language describes the layout of a binary format so a graph can cut a buffer into typed fields: the integers of a header, the records of an array, the floats of a column. The reference compiles descriptions with C++ tools that are not part of this crate; mopenzl takes the compiled program.

• SDDL (GraphBuilder::sddl(&description, successor)): the description is the CBOR program the SDDL compiler writes ({"exprs": [...]}). The graph runs it, dispatches the fields it cuts out to destinations, converts them to numeric where the description says so, and hands every stream (tags, segment sizes, then one per destination) to successor as one multi-input destination. An invalid program is a GraphError::Invalid at build time; a description that does not match the data fails when compressing (and falls back only in permissive mode).
• SDDL2 (GraphBuilder::sddl2(&bytecode, chunk_size, successor)): the segmenter runs bytecode (32-bit little-endian words) once over the input, cuts chunks of about chunk_size bytes (0 = the 16 MiB default) along the segments the program defines, and compresses each chunk’s typed fields with successor (default CompressGeneric). The bytecode is validated when the data is compressed.

The example below builds a tiny SDDL description by hand — “an array of 1000 little-endian u32 fields, sent to destination d0” — and a minimal SDDL2 program.

use mopenzl::graph::{GraphBuilder, StandardGraph};
use mopenzl::{Compressor, Decompressor, EncoderConfig};

// A minimal CBOR writer: just the shapes the program needs.
fn head(major: u8, n: usize) -> Vec<u8> {
    assert!(n < 24);
    vec![major << 5 | n as u8]
}
fn text(s: &str) -> Vec<u8> {
    [head(3, s.len()), s.as_bytes().to_vec()].concat()
}
fn map1(key: &str, value: Vec<u8>) -> Vec<u8> {
    [head(5, 1), text(key), value].concat()
}
fn array(items: &[Vec<u8>]) -> Vec<u8> {
    [head(4, items.len()), items.concat()].concat()
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let records = 1000u32;
    // d0 = dest; consume(array(send(atom("u4l"), d0), int(records)))
    let var_d0 = map1("var", text("d0"));
    let assign = map1("assign", array(&[var_d0.clone(), map1("dest", vec![0xF6])]));
    let count = map1(
        "int",
        [vec![0x19], (records as u16).to_be_bytes().to_vec()].concat(),
    );
    let field = map1("send", array(&[map1("atom", text("u4l")), var_d0]));
    let consume = map1("consume", map1("array", array(&[field, count])));
    let description = map1("exprs", array(&[assign, consume]));

    let mut builder = GraphBuilder::new();
    let sddl = builder.sddl(&description, StandardGraph::CompressGeneric)?;
    let graph = builder.build(sddl)?;
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let data: Vec<u8> = (0..records).flat_map(|i| (i * 3).to_le_bytes()).collect();
    let frame = compressor.compress(&data)?;
    assert_eq!(Decompressor::default().decompress(&frame)?, data);

    // SDDL2: push.i32 4000, segment.create_unspecified, halt (a 4000-byte segment).
    let words: [u32; 4] = [0x0001_0003, 4000, 0x000C_0001, 0x0005_0001];
    let bytecode: Vec<u8> = words.iter().flat_map(|word| word.to_le_bytes()).collect();
    let mut builder = GraphBuilder::new();
    let sddl2 = builder.sddl2(&bytecode, 0, None)?;
    let graph = builder.build(sddl2)?;
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let frame = compressor.compress(&data)?;
    assert_eq!(Decompressor::default().decompress(&frame)?, data);
    Ok(())
}

Dictionaries

OpenZL names external dictionaries through bundles and references them from codec nodes in compression graphs. Start with graphs and codecs to choose which nodes reference a dictionary.

A codec may need side information a frame does not carry. In OpenZL v0.3.0 that is a trained Zstandard dictionary: small records that share structure compress far better against a dictionary built from similar records. This guide covers the mopenzl::dict types, attaching a dictionary to a graph, and resolving it on the decoder.

Concepts · Building dictionaries · Compressing · Decoding · Errors

Concepts

flowchart LR
    D1["Dict (zstd content, ID = SHA-256 of content)"] --> B["DictBundle (ordered Dicts, ID = SHA-256 of the Dict IDs)"]
    B --> L["GraphBuilder::load_dictionaries"]
    N["zstd node + NodeParams::with_dictionary(id)"] --> L
    L --> F["frame: bundle ID in the header, dictionary lane in every chunk"]
    F --> S["BundleStore::insert(bundle)"]
    S --> R["Decompressor resolves the bundle by its ID"]

• A Dict is opaque content for one codec with a DictId, by default the SHA-256 of its content.
• A DictBundle is an ordered list of dictionaries with a BundleId, the SHA-256 of the dictionary IDs in order. Codecs address a dictionary by its position in the bundle, so the order is part of the bundle’s identity.
• A frame compressed with a bundle records the BundleId in its header (format version 25 and newer) and each chunk carries a lane naming the dictionary of every codec. The dictionary content itself is not in the frame: the decoder must hold the same bundle.
• Only standard Zstandard dictionaries can be materialized, and only with the zstd feature. Other codecs, custom codecs and a build without zstd get DictError::NoMaterializer.

Building dictionaries and bundles

This crate does not train dictionaries. Train with the reference’s tools or with zstd --train, and load the result here.

use mopenzl::dict::{Dict, DictBundle, DictCodec};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let users = Dict::zstd(&br#"{"user":"alice","plan":"pro"}"#.repeat(8), 3)?;
    let events = Dict::new(DictCodec::ZSTD, b"raw dictionary content".to_vec())?;

    // Order matters: the bundle's ID, and the index frames record, follow it.
    let bundle = DictBundle::try_from(vec![users.clone(), events.clone()])?;
    let reversed = DictBundle::try_from(vec![events, users.clone()])?;
    assert_ne!(bundle.id(), reversed.id());

    // Fat bundles round-trip through bytes.
    let fat = Vec::from(&bundle);
    assert_eq!(DictBundle::try_from(&fat[..])?, bundle);
    assert_eq!(bundle.dicts()[0].id(), users.id());
    println!("bundle {}", bundle.id()); // 64 hexadecimal digits
    Ok(())
}

Compressing with a dictionary

The order of the calls matters. A dictionary can only be materialized for a node that references it, so:

1. Parameterize the node that will use it: parameterize_node_with(node, NodeParams::default().with_dictionary(dict_id)). Only the Zstandard node uses a dictionary.
2. Load the bundle: load_dictionaries(&bundle). Every dictionary in the bundle must already be referenced by some node, and a compressor holds one bundle.
3. Build the graph: build(start) resolves each node’s dictionary to its index in the bundle.

Format version 25 or newer is required: below it, a node that references a dictionary fails when it runs (as in the reference), and a bundle that no node uses leaves the frame unchanged. The default format version is 27.

use mopenzl::dict::{BundleStore, Dict, DictBundle};
use mopenzl::graph::{GraphBuilder, NodeParams, StandardGraph, StandardNode};
use mopenzl::{Compressor, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Dictionary content: 40 representative records of the data to compress.
    let record = |i: usize| {
        let users = ["alice", "bob", "carol", "dave", "erin"];
        let cities = ["Oslo", "Lima", "Kyiv", "Pune"];
        format!(
            r#"{{"user":"{}","city":"{}","plan":"{}","seen":{},"flags":["beta","email"],"locale":"en-GB"}}"#,
            users[i % 5],
            cities[i % 4],
            if i.is_multiple_of(2) { "pro" } else { "free" },
            i * 7
        )
    };
    let history = (0..40).map(record).collect::<Vec<_>>().join("\n");
    let dict = Dict::zstd(history.as_bytes(), 3)?;
    let bundle = DictBundle::try_from(vec![dict.clone()])?;

    let mut builder = GraphBuilder::new();
    let node = builder.parameterize_node_with(
        StandardNode::Zstd,
        NodeParams::default().with_dictionary(dict.id()),
    )?;
    builder.load_dictionaries(&bundle)?;
    let start = builder.static_graph(node, &[StandardGraph::Store.into()])?;
    let graph = builder.build(start)?;

    let record = record(1234).into_bytes();
    let mut with_dict = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let small = with_dict.compress(&record)?;
    let plain = Compressor::default().compress(&record)?;
    println!(
        "with dictionary {} bytes, without {}",
        small.len(),
        plain.len()
    );

    let mut store = BundleStore::default();
    store.insert(&bundle)?;
    let mut decoder = Decompressor::builder(Default::default())
        .with_dictionaries(store)
        .build();
    assert_eq!(decoder.decompress(&small)?, record);
    Ok(())
}

Notes:

• A graph with a loaded bundle writes the bundle ID and a dictionary lane in every chunk, including stored and empty ones, as the reference does.
• Block splitting inside the Zstandard node ignores the dictionary, as in the reference.
• The dictionary is prepared at the level stored in its content: pass the level you compress with to Dict::zstd.
• A BundleStore is shared: put it in an Arc and give the same store to many decoders (with_dictionaries(impl Into<Arc<BundleStore>>)); a built CompressorGraph is cheap to clone for many compressors.

Decoding with a BundleStore

BundleStore holds the bundles frames may refer to. insert(&bundle) materializes the dictionaries once (a Zstandard DDict), returns the BundleId, and shares them with every decoder given the store. A dictionary listed by several bundles is materialized once; inserting the same bundle twice is a no-op. contains(&id) and len() inspect it. A store holds at most 128 bundles and 65 535 dictionaries.

DecompressorBuilder::with_dictionaries(store) attaches the store (the builder is no longer Copy for that reason). A frame that names a bundle the decoder does not hold is DecodeError::MissingBundle { bundle }, and a codec that names a dictionary index past the end of the bundle is DecodeError::InvalidData of kind DictionaryIndex.

Errors

use mopenzl::dict::{BundleStore, Dict, DictBundle};
use mopenzl::graph::{GraphBuilder, GraphError, NodeParams, StandardGraph, StandardNode};
use mopenzl::{Compressor, DecodeError, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let dict = Dict::zstd(&b"some shared history ".repeat(10), 3)?;
    let bundle = DictBundle::try_from(vec![dict.clone()])?;

    // The order rule: loading before any node references the dictionary fails.
    let mut early = GraphBuilder::new();
    assert!(matches!(
        early.load_dictionaries(&bundle),
        Err(GraphError::Dictionary(_))
    ));

    // A node referencing a dictionary with no bundle loaded fails at build time.
    let mut unloaded = GraphBuilder::new();
    let node = unloaded.parameterize_node_with(
        StandardNode::Zstd,
        NodeParams::default().with_dictionary(dict.id()),
    )?;
    let start = unloaded.static_graph(node, &[StandardGraph::Store.into()])?;
    assert!(matches!(
        unloaded.build(start),
        Err(GraphError::DictionaryNotFound(_))
    ));

    // The right order, and a second load is refused.
    let mut builder = GraphBuilder::new();
    let node = builder.parameterize_node_with(
        StandardNode::Zstd,
        NodeParams::default().with_dictionary(dict.id()),
    )?;
    builder.load_dictionaries(&bundle)?;
    assert_eq!(
        builder.load_dictionaries(&bundle),
        Err(GraphError::BundleAlreadyLoaded)
    );
    let start = builder.static_graph(node, &[StandardGraph::Store.into()])?;
    let graph = builder.build(start)?;

    let frame = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?
        .compress(b"some shared history, again")?;

    // A decoder without the bundle says which one it is missing.
    assert!(matches!(
        Decompressor::default().decompress(&frame),
        Err(DecodeError::MissingBundle { bundle: id }) if id == bundle.id()
    ));
    let mut store = BundleStore::default();
    store.insert(&bundle)?;
    let mut decoder = Decompressor::builder(Default::default())
        .with_dictionaries(store)
        .build();
    assert_eq!(decoder.decompress(&frame)?, b"some shared history, again");
    Ok(())
}

Feature flags

OpenZL frame versions are selected through EncoderConfig::with_format_version: this implementation reads/writes 21–27, comments start at 22, and dictionary bundle IDs/lanes require 25 or newer. Codec availability is separate from the frame version: all-codecs enables the default node set, while direction and individual codec features can be selected independently. See codec features.

Brotli-style Large Window headers and arbitrary headerless continuations are not part of this API. Use an independent OpenZL frame, a segmenter inside one frame, or parallel assembly according to the required boundary. Decoder input/output caps live in DecodeLimits; graph execution retains the existing codec workspace budgets described in the decoder specification.

Serialized and custom static dictionaries

DictBundle packs/unpacks the OpenZL dictionary bundle wire format and BundleStore materializes matching decoder dictionaries. Only Zstandard has a materializer today. Application codec registration through custom_codec is a separate facility, not a universal custom dictionary loader. Parsing, duplicate identities and unavailable materializers produce typed DictError values. Dictionary training is not provided by the safe Rust API.

Framing containers

The opt-in framed container can select a different dictionary-bearing graph per resource. Convert a configured ordinary decoder with FramedDecompressor::from to retain its BundleStore and custom codec registrations. Each inner frame records its bundle ID. No automatic resolver fetches external bytes and no resource implicitly borrows dictionary content from earlier resources. See framed resources and bundle ownership.

Prefix dictionaries

A raw-content Zstandard dictionary is prepared through Dict::zstd, placed in a DictBundle and referenced by a Zstandard graph node. Decode through a matching BundleStore. There is no Brotli-style DictionaryBuilder::add_prefix or headerless raw-prefix attachment overload; see dictionary construction.

Large Window

The public OpenZL EncoderConfig has no Large Window toggle. Frame versions, graph/node parameters and chunking determine the format and codec behavior. Zstandard-specific controls belong to its parameterized node, while decoder limits live in DecoderConfig. No Brotli extended window declaration is emitted.

Headerless continuations

Each ordinary API call emits a complete OpenZL frame. There is no nonzero stream offset for appending a headerless continuation. Use frame concatenation for independent byte payloads, a segmenter for chunks within one frame, or a framed container for typed resources and whole-resource seek.

Configuration and features

Four things are configured separately: the graph (which codecs run), the encoder (global parameters shared by successive frames), the stream (what you know about one input), and the Cargo features (which codecs and adapters are compiled in). Graphs have their own guide: start with graphs. Decoder settings are covered in decompression and limits.

Encoder settings

EncoderConfig holds the global parameters shared by successive frames. They are the reference’s global parameters and shape the bytes of a frame.

Compressor::set_comment(Some(Comment)) stores 1 to 10 000 bytes in the frame header (format version 22 or newer). Unlike the reference, which forgets the comment after each compression, it stays until you change it. Read the comment with Decompressor::frame_info; the decompressed payload contains only the original input.

StreamConfig carries the declared input size of a session; its default is InputSize::Unknown. For a complete known input use StreamConfig::from(InputSize::Exact(len)).

use mopenzl::graph::Comment;
use mopenzl::{Compressor, Decompressor, EncoderConfig, FormatVersion, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = EncoderConfig::default()
        .with_level(Level::try_from(19)?)
        .with_format_version(FormatVersion::try_from(24)?)
        .with_content_checksum(false)
        .with_min_stream_size(32)
        .with_store_on_expansion(true);
    let mut encoder = Compressor::new(config)?;
    encoder.set_comment(Some(Comment::try_from(&b"produced by the docs"[..])?))?;

    let frame = encoder.compress(&b"configured payload ".repeat(100))?;
    assert_eq!(
        Decompressor::default().decompress(&frame)?,
        b"configured payload ".repeat(100)
    );

    // A comment needs format version 22; version 21 refuses it.
    let mut old = Compressor::new(config.with_format_version(FormatVersion::try_from(21)?))?;
    assert!(
        old.set_comment(Some(Comment::try_from(&b"too old"[..])?))
            .is_err()
    );
    Ok(())
}

Validated values (Level, FormatVersion, ChunkSize, Comment) reject out-of-range inputs when built. Compressor::new and reconfigure check combinations.

Cargo features

The default feature set is std, compression, decompression and all-codecs. all-codecs enables one Cargo feature per OpenZL codec (zstd, entropy, lz4, field-lz, delta, tokenize, custom-codec, …); the codec guide lists them all.

Add the codec features you need, such as zstd or all-codecs, when disabling defaults. Without a codec’s feature the encoder never writes it (a chunk that would use it is stored, so frames stop matching the reference’s default graph) and the decoder reports frames that use it as DecodeError::Unsupported. Frames of the default graph need only zstd.

no_std

no_std takes precedence when enabled. It requires a global allocator, removes the std::io adapters, parallel APIs and runtime SIMD detection (the compile-time baseline is used), and pulls in libm for the estimators the numeric selectors rank candidates with.

Caveat: libm’s expf differs from the platform’s by one ULP for some inputs, so the numeric transformer selector’s scores can differ slightly from the reference’s; decisions change only on exact ties (none were observed in 4000+ frames), but byte identity of encoder decisions that depend on floating point is not guaranteed under no_std. Cargo features are additive: another dependency can re-enable std.

See codec features for the gates and what each build exposes.

Buffers and memory reuse

Every call involves two kinds of memory: the destination you hand to the codec, and the workspace the codec keeps for itself. You control both.

Output buffers

compress returns a new Vec<u8>. compress_into appends to the vector you pass and returns the range that holds the new frame; existing bytes stay intact. compress_to_slice returns the number of bytes written. Size the slice with Compressor::max_compressed_size:

use mopenzl::{Compressor, Decompressor, EncodeError};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"data to compress";
    let mut encoder = Compressor::default();
    let bound = Compressor::max_compressed_size(input.len())?;
    let mut output = vec![0; bound];
    let written = encoder.compress_to_slice(input, &mut output)?;
    output.truncate(written);
    assert_eq!(output, encoder.compress(input)?);

    // A slice that is too small is an error, not a truncated frame.
    let mut cramped = [0u8; 4];
    assert!(matches!(
        encoder.compress_to_slice(input, &mut cramped),
        Err(EncodeError::OutputTooSmall { provided: 4 })
    ));

    // Decompression into a slice reports how much it wrote.
    let mut restored = [0u8; 64];
    let n = Decompressor::default().decompress_to_slice(&output, &mut restored)?;
    assert_eq!(&restored[..n], input);
    Ok(())
}

Uninitialized memory

compress_to_uninit, decompress_to_uninit and the sessions’ process_uninit write into &mut [MaybeUninit<u8>], such as Vec::spare_capacity_mut, without zeroing it first. They produce the same bytes, counts and errors as the initialized APIs; on success exactly the reported prefix is initialized and nothing after it is written. Reading the prefix back is the caller’s unsafe step. See uninitialized output.

use mopenzl::{Compressor, Decompressor};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"uninitialized destination ".repeat(50);
    let mut encoder = Compressor::default();
    let mut frame: Vec<u8> = Vec::with_capacity(Compressor::max_compressed_size(input.len())?);
    let written = encoder.compress_to_uninit(&input, frame.spare_capacity_mut())?;
    // SAFETY: `compress_to_uninit` initialized the first `written` bytes.
    unsafe { frame.set_len(written) };
    assert_eq!(frame, encoder.compress(&input)?);

    let mut restored: Vec<u8> = Vec::with_capacity(input.len());
    let n = Decompressor::default().decompress_to_uninit(&frame, restored.spare_capacity_mut())?;
    // SAFETY: `decompress_to_uninit` initialized the first `n` bytes.
    unsafe { restored.set_len(n) };
    assert_eq!(restored, input);
    Ok(())
}

Reusing memory

Keep the codec and your output buffer alive across operations when allocation reuse matters:

use mopenzl::{Compressor, Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut encoder = Compressor::new(EncoderConfig::default().with_level(Level::try_from(3)?))?;
    let mut decoder = Decompressor::default();
    let mut frame = Vec::new();

    for input in [b"first payload".as_slice(), b"second payload".as_slice()] {
        frame.clear(); // Keep the allocation; compress_into appends.
        let written = encoder.compress_into(input, &mut frame)?;
        assert_eq!(written, 0..frame.len());
        assert_eq!(decoder.decompress(&frame)?, input);
    }
    Ok(())
}

A compressor or decompressor retains working state between operations: hash tables, stream buffers and the Zstandard context. Compatible calls reuse the allocations. Destination capacity is yours to manage.

Set the policy with Compressor::builder(config).with_retention(policy).build() (Decompressor::builder likewise). Use trim(policy) for an immediate cleanup and retained_bytes() for the owned heap size; it excludes caller buffers and allocator overhead, and a ceiling is not a peak-memory limit. The policy never changes the bytes produced.

use mopenzl::{Compressor, EncoderConfig, RetentionPolicy};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut encoder = Compressor::builder(EncoderConfig::default())
        .with_retention(RetentionPolicy::Bounded { max_bytes: 1 << 20 })
        .build()?;
    encoder.compress(&b"warm the tables up ".repeat(1000))?;
    println!("retained {} bytes", encoder.retained_bytes());
    encoder.trim(RetentionPolicy::ReleaseAll);
    assert_eq!(encoder.retained_bytes(), 0);
    Ok(())
}

For simultaneous independent streams use one codec per worker; a CompressorGraph is cheap to clone and share between them. A compressor configured with a graph keeps it; Compressor::builder(config).with_graph(graph).

Decompression and limits

Decompressor owns reusable state. decompress returns a Vec, decompress_into appends, and decompress_to_slice writes into caller storage. Sessions report consumed and produced counts; reader and writer adapt synchronous I/O. The decoder needs no graph: a frame records the codecs that produced it, so one decoder reads frames from the default graph, from any standard graph and from your own.

The byte-oriented methods read frames with one serial output, as the reference’s ZL_decompress does. A frame built from typed or several inputs reports DecodeError::Unsupported there (TypedOutput or MultipleOutputs); read it with decompress_typed, which returns one TypedOutput per output with its data_type(), width(), content() and, for strings, string_lens().

By default one call decodes exactly one frame and rejects trailing bytes. FrameMode::Concatenated decodes every frame until the input ends. Both checksums are verified unless you switch them off with with_content_checksum_check(false) and with_compressed_checksum_check(false).

Set limits for untrusted input

Configure limits for untrusted input. Limits are unlimited by default. A frame declaring more output than allowed fails before any chunk is decoded.

use mopenzl::{DecodeError, DecodeLimits, DecoderConfig, Decompressor, FrameMode};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let limits = DecodeLimits::default()
        .with_max_input_bytes(Some(1 << 20))
        .with_max_output_bytes(Some(8 << 20));
    let config = DecoderConfig::default()
        .with_frame_mode(FrameMode::Concatenated)
        .with_limits(limits);
    let mut decoder = Decompressor::new(config);

    // Two frames back to back decode to their concatenation.
    let mut encoder = mopenzl::Compressor::default();
    let mut frames = encoder.compress(b"first frame, ")?;
    frames.extend(encoder.compress(b"second frame")?);
    assert_eq!(decoder.decompress(&frames)?, b"first frame, second frame");

    // Garbage and over-limit frames are errors, not panics.
    assert!(decoder.decompress(b"not a frame").is_err());
    let big = encoder.compress(&vec![7u8; 16 << 20])?;
    assert!(matches!(
        decoder.decompress(&big),
        Err(DecodeError::OutputLimitExceeded { .. })
    ));
    let _ = DecoderConfig::from(limits);
    Ok(())
}

A frame that needs a codec whose Cargo feature is off, an unregistered custom codec, or a format version outside 21 to 27 reports DecodeError::Unsupported. Custom codecs are supported through CustomCodec, GraphBuilder::custom_codec and DecompressorBuilder::with_custom_codec; see custom codecs. A frame compressed with a dictionary bundle the decoder does not hold reports DecodeError::MissingBundle; see dictionaries. See decoder mechanics and the codec catalogue.

Frame metadata

Decompressor::frame_info(&frame) (or FrameInfo::try_from(&frame[..])) reads the frame header without decoding any chunk: the format version, every output’s type, size and element count (ZL_FrameInfo_getOutputType, getDecompressedSize, getNumElts; struct and numeric outputs have no element count, as in the reference), the checksum flags, the header comment and the dictionary bundle ID.

use mopenzl::graph::Comment;
use mopenzl::{Compressor, DataType, Decompressor};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut encoder = Compressor::default();
    encoder.set_comment(Some(Comment::try_from(&b"nightly export"[..])?))?;
    let frame = encoder.compress(b"payload")?;

    let info = Decompressor::default().frame_info(&frame)?;
    assert_eq!(info.comment(), Some(&b"nightly export"[..]));
    assert_eq!(info.outputs()[0].data_type(), DataType::Serial);
    assert_eq!(info.decompressed_size(), 7);
    assert!(info.bundle_id().is_none());
    Ok(())
}

Where to go next

Streaming and completion

Use streaming when the payload does not fit in memory, arrives in pieces, or must leave before the input ends. There are three shapes, and they drive the same state machine:

Readers and writers require std. Sessions fit blocking code, async runtimes and event loops.

Declare the input size

A frame header records the frame’s total size, so declare it with InputSize::Exact when you know it: a session then writes the header at once and, with Chunking::Fixed, holds at most one chunk of input. With InputSize::Unknown it holds the whole input until the frame ends.

Readers and writers

reader(source, stream) consumes source bytes and yields transformed bytes through Read; writer(sink, stream) accepts source bytes through Write and delivers transformed bytes to the sink. Compression takes plain input, decompression takes compressed input (and no stream argument). Both adapter families require std.

A writer borrows the codec and owns its sink. Use write_all for input and finish to end the frame: consuming finish returns the sink, and FinishError retains the writer on failure (try_finish retries, into_error discards the writer). Dropping a writer performs no finalization. A reader owns its source; read to EOF for the complete output. flush() makes accepted input decodable without ending the frame.

use mopenzl::{ChunkSize, Chunking, Compressor, Decompressor, EncoderConfig, InputSize};
use std::io::{Read, Write};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let payload = b"streamed payload ".repeat(10_000);
    let config = EncoderConfig::default().with_chunking(Chunking::Fixed(ChunkSize::MIN));

    // Push model: write plain bytes into a compressed sink.
    let mut encoder = Compressor::new(config)?;
    let mut writer = encoder.writer(Vec::new(), InputSize::Exact(payload.len() as u64).into());
    writer.write_all(&payload)?;
    let frame = writer
        .finish()
        .map_err(mopenzl::io::FinishError::into_error)?;

    // Pull model: read compressed bytes from a plain source.
    let mut reader = encoder.reader(&payload[..], InputSize::Exact(payload.len() as u64).into());
    let mut from_reader = Vec::new();
    reader.read_to_end(&mut from_reader)?;
    assert_eq!(from_reader, frame);

    // Decompression, pulling restored bytes from a compressed source.
    let mut decoder = Decompressor::default();
    let mut restored = Vec::new();
    decoder.reader(&frame[..]).read_to_end(&mut restored)?;
    assert_eq!(restored, payload);

    // ... or pushing compressed bytes into a restored sink.
    let mut writer = decoder.writer(Vec::new());
    writer.write_all(&frame)?;
    let restored = writer
        .finish()
        .map_err(mopenzl::io::FinishError::into_error)?;
    assert_eq!(restored, payload);
    Ok(())
}

The decoder adapters use bounded internal buffering, and decoder-writer finalization reports truncated or invalid input instead of silently accepting an incomplete frame.

Sessions

For direct buffer control start creates an EncoderSession; call process with input, output and an Operation:

Progress reports consumed, produced and status; advance by the counts. Finishing can take several calls when output buffers are small. The decoder’s DecoderSession::process mirrors it with DecodeOperation::{Process, Finish}, DecoderStatus and DecodeProgress, and total_in, total_out and frames_decoded counters.

Operation names. The encoder and decoder name the same idea differently, and both are kept for compatibility:

Adapters return no Result. Compressor::reader, Compressor::writer and the decoder’s reader and writer return the adapter directly. Unlike mbrotli, whose adapter constructors return a Result, nothing can fail when they are created; every error surfaces from the first read, write or finish call as an io::Error. FinishError lives in mopenzl::io, not at the crate root: use writer.finish().map_err(mopenzl::io::FinishError::into_error)? to turn a failed finish into the plain error and drop the writer.

Sessions and graphs. A session holds the input a frame needs. With the default graph and Chunking::Fixed, or a start graph that is a serial segmenter, it compresses chunk by chunk and holds at most one chunk when you declare InputSize::Exact. Any other graph buffers its input until the frame ends and then runs the graph once; the declared size is checked at the end. Session output is byte-identical to the one-shot call. See graphs.

use mopenzl::{
    ChunkSize, Chunking, Compressor, DecodeOperation, DecoderStatus, Decompressor, EncoderConfig,
    EncoderStatus, InputSize, Operation,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let payload = b"incremental payload ".repeat(5_000);
    let config = EncoderConfig::default().with_chunking(Chunking::Fixed(ChunkSize::MIN));
    let mut encoder = Compressor::new(config)?;
    let expected = encoder.compress(&payload)?;

    // Compress in 4 KiB pieces through a small 1 KiB output buffer.
    let mut session = encoder.start(InputSize::Exact(payload.len() as u64).into());
    let mut frame = Vec::new();
    let mut buffer = [0u8; 1024];
    let mut rest = &payload[..];
    loop {
        let take = rest.len().min(4096);
        let operation = if take == rest.len() {
            Operation::End
        } else {
            Operation::Continue
        };
        let progress = session.process(&rest[..take], &mut buffer, operation)?;
        frame.extend_from_slice(&buffer[..progress.produced]);
        rest = &rest[progress.consumed..];
        if progress.status == EncoderStatus::Finished {
            break;
        }
    }
    assert_eq!(frame, expected);

    // Decompress the same way.
    let mut decoder = Decompressor::default();
    let mut session = decoder.start();
    let mut restored = Vec::new();
    let mut position = 0;
    loop {
        let end = (position + 1000).min(frame.len());
        let operation = if end == frame.len() {
            DecodeOperation::Finish
        } else {
            DecodeOperation::Process
        };
        let progress = session.process(&frame[position..end], &mut buffer, operation)?;
        restored.extend_from_slice(&buffer[..progress.produced]);
        position += progress.consumed;
        if progress.status == DecoderStatus::Finished {
            break;
        }
    }
    assert_eq!(restored, payload);
    Ok(())
}

A dropped unfinished session abandons its frame; the codec is ready for the next one.

Read / Write I/O

Use std adapters to connect the codecs to files, network streams, in-memory buffers, and other implementations of Read or Write. Adapters drive the same sessions as the Sans I/O APIs, adding buffering and synchronous I/O conventions.

API map

The library has no dedicated framed decoder Write adapter. Its event API keeps resource boundaries and metadata visible; forward each payload event to an appropriate Write destination. It also has no parallel decoder: parallel compression produces one ordinary OpenZL frame.

All adapters on this page require std. Enable framed for framing. no_std removes the std surface even when std is also enabled.

Raw compression and decompression through Read

An encoder reader consumes plain bytes and yields compressed bytes. A decoder reader consumes compressed bytes and yields restored bytes. std::io::copy drives each one to EOF. Use a File or socket in place of a slice.

use mopenzl::{Compressor, Decompressor, EncoderConfig, InputSize, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"pull plain bytes through the encoder\n".repeat(100);
    let mut encoder = Compressor::new(
        EncoderConfig::default().with_level(Level::try_from(5)?),
    )?;
    let stream = InputSize::Exact(input.len() as u64).into();
    let mut reader = encoder.reader(input.as_slice(), stream);
    let mut compressed = Vec::new();
    std::io::copy(&mut reader, &mut compressed)?;
    drop(reader);

    let mut decoder = Decompressor::new(Default::default());
    let mut reader = decoder.reader(compressed.as_slice());
    let mut restored = Vec::new();
    std::io::copy(&mut reader, &mut restored)?;
    assert_eq!(restored, input);
    Ok(())
}

When the total input size is unknown, use the default stream configuration for the encoder. Reading a decoder to EOF is necessary to observe final validation errors. A raw decoder reader can read past a member boundary; recover its source and buffered suffix with into_parts when another protocol message follows.

Raw compression and decompression through Write

An encoder writer accepts plain bytes; a decoder writer accepts compressed bytes. Each owns its sink and borrows its codec owner. Finalize both writers.

use mopenzl::{Compressor, Decompressor, EncoderConfig, Level};
use std::io::Write;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"push input into a writer\n".repeat(100);
    let mut encoder = Compressor::new(
        EncoderConfig::default().with_level(Level::try_from(5)?),
    )?;
    let mut writer = encoder.writer(Vec::new(), Default::default());
    for chunk in input.chunks(13) {
        writer.write_all(chunk)?;
    }
    let compressed = writer.finish().map_err(mopenzl::io::FinishError::into_error)?;

    let mut decoder = Decompressor::new(Default::default());
    let mut writer = decoder.writer(Vec::new());
    for chunk in compressed.chunks(3) {
        writer.write_all(chunk)?;
    }
    let restored = writer.finish().map_err(mopenzl::io::FinishError::into_error)?;
    assert_eq!(restored, input);
    Ok(())
}

A decoder writer’s finish checks that input ends at a valid stream boundary; flush only drains output and flushes the sink. With a known input size, encoder flush creates an intermediate chunk boundary and can change compressed size; unknown-size frames wait until End. Dropping a writer does not finalize or flush it.

Parallel compression through Read and Write

Both constructors return an adapter and a task queue. Run that queue on independent workers while using the adapter; reads, writes, and finish can block waiting for tasks. A queue with no running executor cannot make progress.

This reader example compresses across scoped threads, then uses an ordinary decoder reader to restore the assembled raw stream:

use mopenzl::compressor::parallel::{ParallelCompressor, ParallelConfig, TaskCount};
use mopenzl::{Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"parallel reader with caller-owned workers\n".repeat(220_000);
    let mut encoder = ParallelCompressor::new(
        EncoderConfig::default().with_level(Level::try_from(5)?),
        ParallelConfig::default(),
    )?;
    let mut compressed = Vec::new();
    std::thread::scope(|scope| -> Result<(), Box<dyn std::error::Error>> {
        let (mut reader, queue) = encoder.reader(input.as_slice(), TaskCount::try_from(4)?)?;
        for _ in 0..4 {
            let worker_queue = queue.clone();
            scope.spawn(move || worker_queue.into_iter().for_each(|task| task.run()));
        }
        drop(queue);
        std::io::copy(&mut reader, &mut compressed)?;
        Ok(())
    })?;

    let mut decoder = Decompressor::new(Default::default());
    let mut reader = decoder.reader(compressed.as_slice());
    let mut restored = Vec::new();
    std::io::copy(&mut reader, &mut restored)?;
    assert_eq!(restored, input);
    Ok(())
}

For the writer direction, the complete parallel writer recipe copies a source into the returned writer, then finishes it while workers keep running. The task window bounds outstanding segments; the parallel writer’s flush ends the current segment early. Decode its output with either raw adapter shown above. See parallel output and backpressure.

Framed compression through readers and resource writers

Enable framed. A framed writer accepts complete resources as they arrive; each resource command queues one OpenZL frame and optional metadata. It is resource-oriented: there is no encoded framed_reader or byte-oriented resource writer on FramedCompressor. Use its native session to drain a container into caller-owned output buffers, or use framed_writer for a Write transport.

The framed writer recipe demonstrates graph selection and metadata. Here is a complete writer paired with sequential resource decoding:

use mopenzl::framing::{FramedCompressor, FramedDecompressor, FramedResource};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"a resource sent through an encoded writer";
    let mut encoder = FramedCompressor::new(Default::default())?;
    let mut writer = encoder.framed_writer(Vec::new())?;
    writer.resource(FramedResource::from(input.as_slice()))?;
    let container = writer.finish().map_err(|failure| failure.error)?;
    let mut decoder = FramedDecompressor::new(Default::default())?;
    let mut reader = decoder.framed_reader(container.as_slice());
    let mut restored = Vec::new();
    let mut resources = 0;
    while let Some(resource) = reader.read_resource()? {
        resources += 1;
        restored.extend_from_slice(resource.outputs[0].content());
    }
    assert_eq!(resources, 1);
    assert_eq!(restored, input);
    Ok(())
}

Framed decompression into a Write sink

FramedReader accepts any Read source and returns owned DecodedResource values. Route its outputs to a sink with write_all, using metadata to select a destination when needed. Continue through None to validate the directory, footer and EOF. Typed resources may have several outputs and string lengths; the example below contains one serial output per resource.

use mopenzl::framing::{FramedCompressor, FramedDecompressor, FramedResource};
use std::io::Write;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let resources = [
        FramedResource::from(&b"first"[..]),
        FramedResource::from(&b"second"[..]),
    ];
    let container = FramedCompressor::new(Default::default())?
        .compress(resources.as_slice().into())?;
    let mut decoder = FramedDecompressor::new(Default::default())?;
    let mut reader = decoder.framed_reader(container.as_slice());
    let mut sinks: Vec<Vec<u8>> = Vec::new();
    while let Some(resource) = reader.read_resource()? {
        let mut sink = Vec::new();
        sink.write_all(resource.outputs[0].content())?;
        sinks.push(sink);
    }
    assert_eq!(sinks, vec![b"first".to_vec(), b"second".to_vec()]);
    Ok(())
}

An application can route resources to files or other writers. Retain any unwritten suffix after a sink error; the owned resource remains available while the transport retries. A resource already delivered remains tentative until the container validates. This reader consumes through EOF; it does not expose a buffered suffix for another container or protocol message.

For Read + Seek, indexed reads use framed_seek_reader and read_resource(index) to return a whole selected resource. For push-driven compressed input use the native framed decoder and route its owned resource events to sinks.

Completion, retries, and ownership

Short sink writes and errors must not replay accepted input. Ordinary and parallel writers can accept input before delivery fails; respect returned byte counts. A framed writer’s resource command drains previous bytes before commit, so a command error means that new resource was not accepted. WouldBlock is not EOF. Use native sessions for explicit runtime readiness handling. See streaming completion and framed retry mechanics.

Sans I/O compression

Use a session when your application owns the input/output buffers and decides when more bytes are available. Sessions perform no I/O. This works in a blocking loop, an async task, or a custom event loop without an I/O adapter.

For transport integration, see Read / Write I/O. For the inverse operation, see Sans I/O decompression.

Drive a raw encoder

Supply chunks with Operation::Continue. After the last chunk, call Finish until EncoderStatus::Finished. An empty input passed with Process is not EOF. The deliberately small output buffer below forces repeated output drains.

use mopenzl::{
    Compressor, Decompressor, EncoderConfig, EncoderStatus, InputSize, Operation, Level,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"incremental encoding with tiny buffers\n".repeat(100);
    let config = EncoderConfig::default().with_level(Level::try_from(5)?);
    let mut encoder = Compressor::new(config)?;
    let mut compressed = Vec::new();
    let mut output = [0; 11];
    {
        let mut session = encoder.start(InputSize::Exact(input.len() as u64).into());
        for chunk in input.chunks(7) {
            let mut remaining = chunk;
            loop {
                let progress = session.process(remaining, &mut output, Operation::Continue)?;
                remaining = &remaining[progress.consumed..];
                compressed.extend_from_slice(&output[..progress.produced]);
                if progress.status == EncoderStatus::NeedsInput {
                    assert!(remaining.is_empty());
                    break;
                }
            }
        }
        loop {
            let progress = session.process(&[], &mut output, Operation::End)?;
            compressed.extend_from_slice(&output[..progress.produced]);
            if progress.status == EncoderStatus::Finished {
                break;
            }
        }
    }
    // Exact input size, offset zero, and no flushes preserve one-shot bytes.
    assert_eq!(compressed, encoder.compress(&input)?);
    let mut decoder = Decompressor::new(Default::default());
    assert_eq!(decoder.decompress(&compressed)?, input);
    Ok(())
}

Always advance input by consumed and publish only output[..produced] before handling status. NeedsOutput requires output space and the unconsumed input suffix; reaching the end of the current input slice alone does not finish the stream.

Flush and finish are different

Operation::Flush ends the current chunk without ending the frame, provided its total input size is known. With unknown size it has no effect. Drain it to NeedsInput before changing operation or supplying new input. Flush boundaries can change output size and bytes. Operation::End ends the frame; while retrying it, keep the same operation and exact unconsumed input suffix.

Use InputSize::Exact when the total input size is known. Otherwise use the default unknown size; the encoder buffers input until End because the frame header needs the total size. See stream scheduling and identity.

Drive parallel input

ParallelCompressor::start(window) returns (session, queue). Keep workers running the queue while you feed input and drain output. The session’s process does not block: ParallelEncoderStatus::Pending means progress depends on tasks. Call session.wait() with independently runnable workers, or execute queued tasks yourself before waiting on a single thread.

The complete Sans I/O parallel example handles all statuses and source chunks. Its window bounds active tasks; OpenZL streaming holds compressed results until the total size is known at End. Without flushes, output matches a batch over the same input and segment configuration. See parallel session mechanics.

Drive a framed encoder

Enable framed. Container process takes an output buffer and a FramedEncodeOperation. Submit each complete FramedResource with its optional graph, drain the queued frame, then finish the container. Resource inputs can be serial, typed or multiple; this API does not offer byte-oriented resource guards.

use mopenzl::framing::{
    FramedCompressor, FramedDecompressor, FramedEncodeOperation,
    FramedEncoderStatus, FramedResource,
};
use mopenzl::graph::{CompressorGraph, StandardGraph};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"framed incremental payload\n".repeat(20);
    let store = CompressorGraph::standard(StandardGraph::Store);
    let mut encoder = FramedCompressor::new(Default::default())?;
    let mut session = encoder.start()?;
    let mut buffer = [0; 13];
    let mut container = Vec::new();
    // Drain the main header before accepting a resource command.
    loop {
        let progress = session.process(&mut buffer, FramedEncodeOperation::Process)?;
        container.extend_from_slice(&buffer[..progress.produced]);
        if progress.status == FramedEncoderStatus::NeedsInput { break; }
    }
    session.resource(FramedResource {
        graph: Some(&store),
        ..FramedResource::from(input.as_slice())
    })?;
    loop {
        let progress = session.process(&mut buffer, FramedEncodeOperation::Finish)?;
        container.extend_from_slice(&buffer[..progress.produced]);
        if progress.status == FramedEncoderStatus::Finished { break; }
    }
    drop(session);
    let mut decoder = FramedDecompressor::new(Default::default())?;
    assert_eq!(decoder.decompress(&container)?.resources[0].outputs[0].content(), input);
    Ok(())
}

An empty resource is an ordinary FramedResource::from(&b""[..]) command and still produces one complete frame. A new resource requires previous container output to be drained; otherwise OutputPending rejects the command before commit. After a failed resource command the same session can accept a corrected resource. See resource lifecycles and metadata.

Ownership and cancellation

Borrowed sessions exclusively borrow their codec owner. Dropping an unfinished session cancels its frame/container and releases the owner; Drop performs no transport I/O. Raw codec into_session and parallel into_session provide owned sessions when a lifetime-free holder is useful. Framed native sessions are borrowed only. See encoder sessions, decoder sessions and framed lifecycles.

Sans I/O decompression

Feed compressed input and supply writable output space as it becomes available. Sessions own decoding state and retain no caller input/output slices between calls. DecodeOperation::Process allows more input; Finish declares the final input boundary and enables truncation checks.

There is no separate parallel decoder. Parallel compression assembles a standard OpenZL frame, decoded through the ordinary decoder APIs. For transport adapters, see Read / Write I/O.

Decode a raw stream in chunks

This example supplies three compressed bytes at a time, uses a tiny output buffer, and declares Finish only for the last input chunk.

use mopenzl::{
    Compressor, DecodeOperation, DecoderStatus, Decompressor, EncoderConfig, Level,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"incremental decoding preserves every byte\n".repeat(100);
    let mut encoder = Compressor::new(
        EncoderConfig::default().with_level(Level::try_from(5)?),
    )?;
    let compressed = encoder.compress(&input)?;
    let mut decoder = Decompressor::new(Default::default());
    let mut session = decoder.start();
    let mut chunks = compressed.chunks(3).peekable();
    let mut output = [0; 11];
    let mut restored = Vec::new();
    let mut finished = false;
    while let Some(chunk) = chunks.next() {
        let operation = if chunks.peek().is_none() {
            DecodeOperation::Finish
        } else {
            DecodeOperation::Process
        };
        let mut remaining = chunk;
        loop {
            let progress = session.process(remaining, &mut output, operation)?;
            remaining = &remaining[progress.consumed..];
            restored.extend_from_slice(&output[..progress.produced]);
            match progress.status {
                DecoderStatus::NeedsInput => {
                    assert!(remaining.is_empty());
                    break;
                }
                DecoderStatus::NeedsOutput => {}
                DecoderStatus::Finished => {
                    assert!(remaining.is_empty());
                    finished = true;
                    break;
                }
            }
        }
        if finished {
            break;
        }
    }
    assert!(finished);
    assert_eq!(restored, input);
    Ok(())
}

In an application where EOF is learned from a later read, offer chunks with Process and then call process(&[], output, Finish) until Finished. After the first Finish, continue with Finish and exactly the remaining input suffix. Do not add new bytes or switch back to Process.

NeedsOutput can occur after every input byte has been consumed: deliver the output and call again with an empty input slice and fresh output space. For protocols with bytes after the OpenZL member, preserve unconsumed input when Finished arrives. The example asserts an empty suffix because its input contains exactly one complete member.

Decode a framing container as events

With framed, a session yields one owned DecodedResource at each resource boundary. Metadata and every typed output are attached to that value. There are no borrowed payload-fragment events or caller output buffer in this API.

use mopenzl::framing::{
    FramedCompressor, FramedDecompressor, FramedDecoderStatus, FramedResource,
};
use mopenzl::DecodeOperation;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"resource decoded from tiny arriving chunks";
    let resources = [FramedResource::from(input.as_slice())];
    let container = FramedCompressor::new(Default::default())?
        .compress(resources.as_slice().into())?;
    let mut decoder = FramedDecompressor::new(Default::default())?;
    let mut session = decoder.start();
    let mut chunks = container.chunks(3).peekable();
    let mut restored = Vec::new();
    let mut resources = 0;
    let mut finished = false;
    while let Some(chunk) = chunks.next() {
        let operation = if chunks.peek().is_none() {
            DecodeOperation::Finish
        } else {
            DecodeOperation::Process
        };
        let mut remaining = chunk;
        loop {
            let progress = session.process(remaining, operation)?;
            remaining = &remaining[progress.consumed..];
            if let Some(resource) = progress.resource {
                resources += 1;
                restored.extend_from_slice(resource.outputs[0].content());
            }
            match progress.status {
                FramedDecoderStatus::NeedsInput => { assert!(remaining.is_empty()); break; }
                FramedDecoderStatus::Finished => { assert!(remaining.is_empty()); finished = true; break; }
                FramedDecoderStatus::Resource => {}
            }
        }
        if finished { break; }
    }
    assert!(finished);
    assert_eq!(resources, 1);
    assert_eq!(restored, input);
    Ok(())
}

A resource event does not validate the rest of the container. Continue through Finished to verify its directory, footer and EOF. For multiple resources, route each DecodedResource to a separate destination. Metadata does not select an extraction path. See framed event semantics.

Limits and errors

Set input and output budgets in DecoderConfig before starting an ordinary OpenZL decoder. Framing adds FramingConfig caps for encoded frame bytes, metadata, resource count, aggregate decoded content and total wire bytes. See resource-limit configuration.

Raw DecodeFailure carries consumed/produced counts for a failing call; FramedDecodeFailure carries consumed bytes and no partial owned resource. Previously returned resource events remain tentative until the container finishes validation. Failures are sticky: drop and start a new session. Finish reports truncation; an empty Process call does not declare EOF.

Parallel compression

mopenzl::compressor::parallel divides a known input into fixed independent segments and assembles them into one ordinary OpenZL frame. It uses the same encoder configuration, graphs and codec registry as serial compression. Typed/multiple top-level inputs and frame comments are not exposed here.

The caller runs the tasks using scoped threads, Rayon, or another executor. The library does not create a thread pool.

Compressing a slice

use mopenzl::compressor::parallel::{
    BatchConfig, ParallelCompressor, ParallelConfig, TaskCount,
};
use mopenzl::{Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = vec![b'a'; 8 << 20];
    let mut encoder = ParallelCompressor::new(
        EncoderConfig::default().with_level(Level::try_from(5)?),
        ParallelConfig::default(),
    )?;
    let mut batch = encoder.prepare_slice(
        &input,
        BatchConfig::auto(TaskCount::try_from(4)?),
    )?;
    let tasks = batch.take_tasks()?;
    std::thread::scope(|scope| {
        for task in tasks {
            scope.spawn(move || task.run());
        }
    });
    let mut output = Vec::new();
    batch.finish_into(&mut output)?;
    let mut decoder = Decompressor::new(Default::default());
    assert_eq!(decoder.decompress(&output)?, input);
    Ok(())
}

take_tasks transfers each task once. Every taken task must run or be dropped so its completion can be reported. Finalization collects task results, validates artifacts, and assembles them in input order. A blocking wait requires the caller’s executor to remain runnable.

Running tasks with Rayon

Add rayon = "1.12" alongside mopenzl in your dependencies. Create a pool once and reuse it across requests. Task completion order does not affect assembly order or the resulting bytes for fixed segment settings.

use mopenzl::compressor::parallel::{
    BatchConfig, ParallelCompressor, ParallelConfig, TaskCount,
};
use mopenzl::{Decompressor, EncoderConfig, Level};
use rayon::prelude::*;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let pool = rayon::ThreadPoolBuilder::new().num_threads(4).build()?;
    let mut encoder = ParallelCompressor::new(
        EncoderConfig::default().with_level(Level::try_from(5)?),
        ParallelConfig::default(),
    )?;
    let input = b"parallel compression with a reusable pool\n".repeat(220_000);
    let mut batch = encoder.prepare_slice(
        &input,
        BatchConfig::auto(TaskCount::try_from(4)?),
    )?;
    let tasks = batch.take_tasks()?;
    pool.install(|| tasks.into_par_iter().for_each(|task| task.run()));

    let mut compressed = Vec::new();
    batch.finish_into(&mut compressed)?;
    let mut decoder = Decompressor::new(Default::default());
    assert_eq!(decoder.decompress(&compressed)?, input);
    Ok(())
}

For many unrelated small inputs, another option is one ordinary Compressor per worker. That preserves serial compression decisions for each input. The batch API above splits one input into a single assembled stream.

Streaming input

When input arrives in chunks, start(window) returns a session and a task queue. Each full segment the session accepts becomes a task on that queue; clone the queue for each executor and run for task in queue { task.run() }. OpenZL requires the total size in its frame header, so the session stages compressed segments until End establishes that size. It then assembles results in input order. The queue closes when the session finishes, fails, or is dropped.

use mopenzl::compressor::parallel::{
    ParallelCompressor, ParallelConfig, ParallelEncoderStatus, TaskCount,
};
use mopenzl::{EncoderConfig, Operation, Level};
use std::io::{Read, Write};

fn compress(
    mut reader: impl Read,
    mut writer: impl Write,
) -> Result<(), Box<dyn std::error::Error>> {
    let mut encoder = ParallelCompressor::new(
        EncoderConfig::default().with_level(Level::try_from(5)?),
        ParallelConfig::default(),
    )?;
    std::thread::scope(|scope| {
        let (mut session, queue) = encoder.start(TaskCount::try_from(4)?)?;
        for _ in 0..4 {
            let queue = queue.clone();
            scope.spawn(move || queue.into_iter().for_each(|task| task.run()));
        }
        drop(queue);
        let mut chunk = vec![0; 1 << 16];
        let mut output = vec![0; 1 << 16];
        loop {
            let read = reader.read(&mut chunk)?;
            let operation = if read == 0 { Operation::End } else { Operation::Continue };
            let mut input = &chunk[..read];
            loop {
                let progress = session.process(input, &mut output, operation)?;
                input = &input[progress.consumed..];
                writer.write_all(&output[..progress.produced])?;
                match progress.status {
                    ParallelEncoderStatus::Finished => return Ok(()),
                    ParallelEncoderStatus::Pending => session.wait()?,
                    ParallelEncoderStatus::NeedsInput if input.is_empty() => break,
                    _ => {}
                }
            }
        }
    })
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"incremental parallel input\n".repeat(350_000);
    let mut compressed = Vec::new();
    compress(input.as_slice(), &mut compressed)?;
    let mut decoder = mopenzl::Decompressor::new(Default::default());
    assert_eq!(decoder.decompress(&compressed)?, input);
    Ok(())
}

• process never blocks. Pending means the window is full or a flush or finish waits for queued tasks; call wait, which blocks until one completes.
• window bounds the segments queued, running or awaiting delivery, and therefore active work. Compressed results accumulate until End; the optional aggregate memory estimate also accounts for buffered results.
• Without a flush, the output is byte-identical to prepare_slice over the same bytes, however the input was chunked. Operation::Flush ends the current segment early, which moves later segment boundaries.
• On a single thread, run queue.try_recv() tasks before wait; nothing else will. Dropping every queue handle fails the stream instead of hanging.
• into_session(window) returns an owned session without a lifetime; into_compressor returns the compressor.

Reader and writer adapters

writer(sink, window) and reader(source, window) wrap the same session in std::io::Write and std::io::Read and return the task queue beside the adapter, so the loop above becomes std::io::copy:

use mopenzl::compressor::parallel::{ParallelCompressor, ParallelConfig, TaskCount};
use mopenzl::io::FinishError;
use mopenzl::{EncoderConfig, Level};
use std::io::{Read, Write};

fn compress<W: Write>(
    mut reader: impl Read,
    writer: W,
) -> Result<W, Box<dyn std::error::Error>> {
    let mut encoder = ParallelCompressor::new(
        EncoderConfig::default().with_level(Level::try_from(5)?),
        ParallelConfig::default(),
    )?;
    std::thread::scope(|scope| {
        let (mut sink, queue) = encoder.writer(writer, TaskCount::try_from(4)?)?;
        for _ in 0..4 {
            let queue = queue.clone();
            scope.spawn(move || queue.into_iter().for_each(|task| task.run()));
        }
        drop(queue);
        std::io::copy(&mut reader, &mut sink)?;
        Ok(sink.finish().map_err(FinishError::into_error)?)
    })
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = b"stream through caller-owned workers\n".repeat(250_000);
    let compressed = compress(input.as_slice(), Vec::new())?;
    let mut decoder = mopenzl::Decompressor::new(Default::default());
    assert_eq!(decoder.decompress(&compressed)?, input);
    Ok(())
}

• Unlike process, the adapters block: a write waits while the window is full, a read waits for the next part, and finish waits for every part. Run the queue on threads other than the one using the adapter.
• Output is byte-identical to prepare_slice over the same bytes. The writer’s flush ends the partial segment early and moves later boundaries; it emits no decodable prefix until End establishes the total size.
• The writer keeps every compressed byte until the sink takes it, so short writes, WouldBlock and sink errors lose nothing, and finish hands the adapter back inside FinishError for a retry, as EncoderWriter does. reader.into_parts() returns source bytes read ahead of the session.
• Session failures arrive as std::io::Error with the ParallelEncodeError as their source. Dropping an adapter abandons the stream and closes the queue.

Segments and compressed size

Segment size defaults to 4 MiB and accepts 64 KiB–16 MiB. Effective task count is at most the number of segments. For fixed input and segment settings, task grouping and completion order do not change output bytes.

Segments are encoded independently, without cross-segment history. The frame is OpenZL-compatible, but byte identity with the serial reference is not promised. For fixed input, graph and segment settings, task grouping and completion order do not change bytes. Compare compressed size and throughput with serial output.

Empty input uses serial encoding without tasks. A nonempty input below the default 8 MiB parallel threshold uses a single serial task only when it also fits in one segment.

Sources and staging

FileSource uses positional reads on Unix and Windows. SeekSource accepts Read + Seek + Send, serializes seek/read operations, and lets compression run concurrently. Sources must remain immutable for the batch lifetime; metadata checks cannot detect every in-place mutation.

Owned sources and shared handles use the same generic entry point. For an existing Arc<FileSource>, type inference can require prepare_source::<FileSource, _>(shared_file, config).

BatchConfig::auto(tasks) stages in memory using bounds derived from source length and configured segment size. It caps effective tasks at the segment count and still enforces any configured aggregate memory limit. It does not query available RAM or switch to files.

BatchConfig::memory(tasks, max_bytes) sets an explicit payload-and-descriptor budget. To obtain its exact required preflight allowance, call estimate_source with BatchConfig::auto(tasks), then convert estimate.maximum_staging_memory_bytes()? using usize::try_from. maximum_staged_bytes remains the payload-only bound. Planning also accounts for worker and assembly storage. These estimates are not a process RSS measurement. BatchConfig::directory(tasks, directory) stages output in exclusive temporary files in an existing directory, keeping payload RAM proportional to active workers and segment size. File cleanup is best effort on drop.

Output and cancellation

finish_into appends to a vector and restores its length on failure. finish_to_writer consumes the batch and returns the destination with statistics on success. On failure it preserves destination ownership and the number of accepted bytes; the consumed batch cannot retry assembly. The caller controls flushing, file durability, and publication.

Cancellation is cooperative around source reads and segment encoding. A running codec call or blocking source read can delay it. Dropping a batch cancels work without waiting for detached tasks. Worker retention is configurable and defaults to retaining no workers.

The repository example runs scoped batches, reuses the compressor and checks each resulting frame:

cargo run --release --example parallel

See parallel mechanics for lifecycle, resource accounting, source checks, and error propagation.

Choose a graph or individual codecs

ParallelCompressor::builder(config, parallel).with_graph(graph).build() accepts the same CompressorGraph used by the serial API. Standard graphs, parameterized nodes, selectors, segmenters and loaded Zstandard dictionaries run for every segment, including tails and flush-cut segments. Choose aligned boundaries for a graph interpreting bytes as records or numbers. Callbacks may run concurrently and must be deterministic for schedule-independent output.

use mopenzl::compressor::parallel::{BatchConfig, ParallelCompressor, ParallelConfig, TaskCount};
use mopenzl::graph::{CompressorGraph, StandardGraph};
use mopenzl::Decompressor;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut encoder = ParallelCompressor::builder(Default::default(), ParallelConfig::default())
        .with_graph(CompressorGraph::standard(StandardGraph::Store))
        .build()?;
    let input = b"graph selected for every segment";
    let mut batch = encoder.prepare_slice(input, BatchConfig::auto(TaskCount::ONE))?;
    for task in batch.take_tasks()? { task.run(); }
    let mut frame = Vec::new();
    batch.finish_into(&mut frame)?;
    assert_eq!(Decompressor::default().decompress(&frame)?, input);
    Ok(())
}

For individual codecs, use GraphBuilder::pipeline, static_graph or parameterize_node, then build the selected root graph and pass it to with_graph. Graph selection also applies to serial fallback and survives reconfiguration. See graphs and codec nodes.

Framing containers

Package independent resources with metadata and a checked seek index. Each resource contains one ordinary OpenZL frame, with its own graph and serial, typed or multiple inputs. The outer format belongs to mopenzl; an ordinary OpenZL decoder reads an extracted frame rather than the complete container.

Enable framing

mopenzl = { version = "0.2", features = ["framed"] }

The book follows the repository’s main branch; use this checkout or a Git dependency for features not yet in a crates.io release. framed is opt-in, outside default and all-codecs. With compression it exposes the framed encoder; with decompression, the framed decoder. Owned results and borrowed native sessions work with alloc; std adds writer, sequential reader and seek reader adapters. no_std removes those transport adapters.

The container has one resource profile and a mandatory directory/footer. Resource commands take complete borrowed inputs; there are no byte-oriented resource writers, hidden resources, padding or repeated metadata. Split large input into resources explicitly. See format and lifecycle.

One container, two resources

Use the default graph for text and Store for already encoded binary data. Graph overrides apply only to their resource and are restored after failure.

use mopenzl::framing::{FramedCompressor, FramedDecompressor, FramedResource};
use mopenzl::graph::{CompressorGraph, StandardGraph};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let text = b"notes for a compressed resource\n".repeat(200);
    let binary = [1, 2, 3, 4, 5];
    let store = CompressorGraph::standard(StandardGraph::Store);
    let resources = [
        FramedResource::from(text.as_slice()),
        FramedResource { graph: Some(&store), ..FramedResource::from(&binary[..]) },
    ];
    let mut encoder = FramedCompressor::new(Default::default())?;
    let container = encoder.compress(resources.as_slice().into())?;
    let mut decoder = FramedDecompressor::new(Default::default())?;
    let decoded = decoder.decompress(&container)?;
    assert_eq!(decoded.resources.len(), 2);
    assert_eq!(decoded.resources[0].outputs[0].content(), text);
    assert_eq!(decoded.resources[1].outputs[0].content(), binary);
    Ok(())
}

compress_into appends transactionally to a reusable Vec and returns the container’s range. Raw slice/uninitialized output methods belong to the ordinary codec; framed output uses Vec, a native output session or a writer. FramedResource::from(&inputs[..]) accepts typed/multiple Input values. See graphs and typed resources.

Write resources and metadata

Use a writer when complete resources arrive one after another. Attach borrowed metadata directly to each resource command, then finish the enclosing container. Metadata codes are uppercase ASCII pairs, id (unique UTF-8 name within one resource’s fields), or mt (unique eight-byte signed little-endian timestamp).

use mopenzl::framing::{
    FramedCompressor, FramedDecompressor, FramedEncodeConfig, FramedResource,
    FramingConfig, MetadataField,
};
use mopenzl::graph::{CompressorGraph, StandardGraph};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = FramedEncodeConfig {
        framing: FramingConfig { max_resources: 16, ..Default::default() },
        ..Default::default()
    };
    let fields = [MetadataField { code: *b"id", value: b"notes.txt" }];
    let store = CompressorGraph::standard(StandardGraph::Store);
    let mut encoder = FramedCompressor::new(config)?;
    let mut writer = encoder.framed_writer(Vec::new())?;
    writer.resource(FramedResource {
        metadata: &fields,
        ..FramedResource::from(&b"first part\nsecond part\n"[..])
    })?;
    writer.resource(FramedResource {
        graph: Some(&store),
        ..FramedResource::from(&[1, 2, 3, 4][..])
    })?;
    let container = writer.finish().map_err(|failure| failure.error)?;
    let decoded = FramedDecompressor::new(Default::default())?.decompress(&container)?;
    assert_eq!(decoded.resources[0].outputs[0].content(), b"first part\nsecond part\n");
    assert_eq!(decoded.resources[0].metadata[0].value, b"notes.txt");
    assert_eq!(decoded.resources[1].outputs[0].content(), [1, 2, 3, 4]);
    Ok(())
}

No Drop performs I/O. resource drains older output before accepting its command; an error means this resource was not accepted, so retrying that command does not duplicate it. try_finish retains the writer after an I/O error; consuming finish returns FramedFinishError with writer and error fields. See the retry state machine.

id is a name, not an extraction path or a globally unique lookup key. Framing computes/verifies metadata and directory XXH3 checksums, while inner OpenZL checksums follow encoder/decoder configuration. These detect damage and do not authenticate content.

Indexed resource reads

Open the footer and directory on a Read + Seek source, then decode only a selected whole resource. A File can replace this example’s Cursor.

use mopenzl::framing::{FramedCompressor, FramedDecompressor, FramedResource};
use std::io::Cursor;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let resources = [
        FramedResource::from(&b"first"[..]),
        FramedResource::from(&b"second"[..]),
    ];
    let container = FramedCompressor::new(Default::default())?
        .compress(resources.as_slice().into())?;
    let mut decoder = FramedDecompressor::new(Default::default())?;
    let mut reader = decoder.framed_seek_reader(Cursor::new(container))?;
    assert_eq!(reader.resources().len(), 2);
    let selected = reader.read_resource(1)?;
    assert_eq!(selected.outputs[0].content(), b"second");
    Ok(())
}

Indices are zero-based usize values, independent of metadata names. Opening validates the directory; reading validates that resource’s header, metadata, size and inner frame. It returns owned typed outputs rather than a Read adapter. The container extends from the source’s current position to EOF; offsets remain relative for a container embedded after a prefix. External dictionary bundles come from the decoder’s configured BundleStore. See indexed decoder mechanics.

Limits and streaming decoding

FramingConfig bounds resources, metadata per resource, encoded frame bytes, aggregate decoded content and total wire bytes. FramedDecodeConfig::decoder adds raw decoder input/output limits and checksum policy for each frame.

For arriving input, FramedDecompressor::start yields owned whole-resource events; framed_reader adapts a Read source through read_resource. Continue until Finished or None to validate the directory, footer and EOF. Earlier resources remain tentative until then. There is no automatic raw-frame detection or dictionary reference to an earlier resource. See Sans I/O decoding and dictionary bundles.

Framed resources with independent graphs

Enable the opt-in feature alongside your codec directions:

mopenzl = { version = "0.2", features = ["framed"] }

framing::FramedCompressor writes an indexed container. Each resource is one ordinary OpenZL frame with its own graph and optional metadata. The outer container is mopenzl-specific: an ordinary OpenZL decoder accepts an extracted resource frame, not the complete container. The decoder reconstructs codecs from each frame; it needs no graph selection.

use mopenzl::framing::{FramedCompressor, FramedDecompressor, FramedResource, MetadataField};
use mopenzl::graph::{CompressorGraph, StandardGraph};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let store = CompressorGraph::standard(StandardGraph::Store);
    let name = [MetadataField { code: *b"id", value: b"image.bin" }];
    let text = b"text text text text text text";
    let image = [0x42; 1024];
    let resources = [
        FramedResource::from(&text[..]), // default graph
        FramedResource {
            graph: Some(&store),
            metadata: &name,
            ..FramedResource::from(&image[..])
        },
    ];
    let mut encoder = FramedCompressor::new(Default::default())?;
    let bytes = encoder.compress(resources.as_slice().into())?;
    let mut decoder = FramedDecompressor::new(Default::default())?;
    let output = decoder.decompress(&bytes)?;
    assert_eq!(output.resources[0].outputs[0].content(), text);
    assert_eq!(output.resources[1].outputs[0].content(), image);
    Ok(())
}

For typed or multiple inputs, construct FramedResource::from(&inputs[..]) where inputs is a slice of mopenzl::input::Input. Decoded resources retain the original output types, widths and string lengths. A graph override lasts only for its resource. Convert a configured Compressor or Decompressor with FramedCompressor::from / FramedDecompressor::from to retain its backend, graph, retention, dictionaries or custom-codec registrations.

Without no_std, use encoder.framed_writer(sink) to add complete resources as they arrive, then finish to write the seek index. resource accepts complete borrowed inputs and queues one encoded resource. flush delivers queued bytes; finish returns the sink or a failure containing the writer for retry. Nothing is written on Drop. Reader methods return whole typed resources:

use mopenzl::framing::{FramedCompressor, FramedDecompressor, FramedResource};
use std::io::Cursor;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut encoder = FramedCompressor::new(Default::default())?;
    let mut writer = encoder.framed_writer(Vec::new())?;
    writer.resource(FramedResource::from(&b"first"[..]))?;
    writer.resource(FramedResource::from(&b"second"[..]))?;
    let bytes = writer.finish().map_err(|failure| failure.error)?;

    let mut decoder = FramedDecompressor::new(Default::default())?;
    let mut reader = decoder.framed_seek_reader(Cursor::new(&bytes))?;
    assert_eq!(reader.resources().len(), 2);
    assert_eq!(reader.read_resource(1)?.outputs[0].content(), b"second");
    drop(reader);
    let mut sequential = decoder.framed_reader(&bytes[..]);
    while let Some(resource) = sequential.read_resource()? {
        println!("{} content bytes", resource.info.decoded_size);
    } // None validates the directory, footer and EOF.
    Ok(())
}

With alloc (including no_std), both codecs also offer borrowed native sessions. Drain an encoder session’s header with process(output, Process) until NeedsInput, submit resource, drain again, and finally use Finish until Finished. Decoder sessions consume wire bytes until a resource event; process the unconsumed suffix, and use DecodeOperation::Finish at EOF. Resource events are tentative until Finished validates the directory and footer.

FramingConfig supplies bounded resource count, metadata, frame, aggregate output and total wire sizes. Configure it through FramedEncodeConfig, FramedDecodeConfig, or reconfigure. Inputs are complete per resource, and each resource/frame is staged in memory. Choose resource boundaries explicitly for large streams. This first version supports resource metadata and whole resource seek; it does not mirror every mbrotli/mzstd container command.

See the wire format, lifecycle and limits.

Compatibility and validation

This page states the output contract and the evidence behind it. Rust/C frame identity, decoding interoperability, malformed-input behavior and memory safety are separate checks. See oracle mechanics.

What is claimed

• Serial compression is byte-identical to Meta’s pinned OpenZL v0.3.0 for the same graph, typed inputs, global/local parameters and chunk boundaries.
• Supported frame versions are 21–27. Every non-legacy standard node/graph is ported; deprecated nodes 76–85 and older formats remain unsupported.
• A frame describes its codec layout. Decoding needs matching enabled codecs, custom-codec registrations and external dictionary bundles, not its selector or compressor graph definition.
• Backend choice, workspace retention and destination shape preserve output. Sessions without flushes match one-shot bytes for arbitrary buffer splits.
• Parallel assembly produces compatible OpenZL frames, with a separate byte contract from serial compression. Framing adds a mopenzl-specific container around independent ordinary frames.

no_std estimators use libm; Transformer/estimator decisions have a C byte identity guarantee only with std. Missing codec features can produce stored fallbacks or explicit graph errors, and disabled decoder codecs report Unsupported. See codec features.

Output compatibility

Serial frames match OpenZL v0.3.0 for the same graph, inputs, global parameters, format version, checksum choices and chunk boundaries. Destination shape, workspace reuse and backend selection do not change bytes. Sessions without explicit flushes match one-shot output for any input/output split. Unknown-size frames wait until End; they do not infer an early header size.

Parallel segments form one compatible frame but are not promised byte-identical to serial encoding. Framed containers have a mopenzl-specific outer format; every inner resource is an ordinary frame from its selected graph. Custom codecs must be registered on both sides and external dictionary bundles must match.

Validation layers

graph TD
    API[Public APIs and examples] --> Tests[Unit and integration tests]
    Tests --> C[Pinned C OpenZL oracle]
    Tests --> Identity[Backend, chunking and output-destination identity]
    Tests --> Fuzz[AFL and libFuzzer shared target bodies]
    Tests --> Coverage[Function coverage report]
    Tests --> Memory[Miri and AddressSanitizer]

Differential tests

Internal identity tests

tests/streaming.rs, tests/io.rs, tests/uninit.rs and tests/simd_backends.rs cover arbitrary input/output splits, completion, buffer destinations and every available backend. Private core tests explicitly exercise scalar baselines. tests/parallel.rs covers task ordering, sources, staging and streaming assembly. tests/framed.rs covers independent graphs, typed resources, metadata, seek, transport retries and corruption/truncation.

Feature and fuzz coverage

scripts/check_codec_features.py checks public direction/codec isolation. scripts/check_simd_uninit.py selects focused Miri tests and full ASan codec modules; x86-64 Miri uses x86-64-v3, while ARM64 tests Neon paths supported by Miri and leaves unsupported Neon operations to ASan.

The isolated AFL package covers ordinary/typed decoding, one-shot and streaming compression, graph configuration, individual nodes, custom codecs, parallel assembly and framed containers. Every target has an engine-neutral body, small regression corpus and matching libFuzzer adapter. CI replays regressions and ClusterFuzzLite runs on pull requests and daily schedules. See fuzzing coverage and commands.

Framed-container smoke campaign — 2026-11-11

Apple M5 Pro, aarch64-apple-darwin, libFuzzer with AddressSanitizer under Rust 1.101.0-nightly (2026-10-23). The bounded arbitrary-container target completed 411,538 executions in 31 seconds; resource round-trip completed 67,489 in 31 seconds, with no findings. Inputs cap frame/output sizes and resource count; the round-trip oracle checks native/one-shot equality and C decoding of every embedded frame. These are short smoke campaigns, not an exhaustive audit.

The same AFL++ 4.40c/cargo-afl 0.18.2 targets built and replayed the committed corpus. Local campaigns failed before execution because macOS SysV shared-memory allocation was unavailable, also outside the sandbox. No host tuning was done. Linux CI contains bounded AFL smoke runs. See the recorded framing verification.

Campaign records and reproducibility

Record input corpus, target, revision, toolchain, feature set, engine version, limits, duration and findings together. Keep live queues/findings out of Git. Minimize every confirmed crash, commit a deterministic regression, then fix it. The AFL guide and libFuzzer guide describe replay and minimization. Every published campaign record should identify the mopenzl revision and its actual target bodies, corpus and oracle.

Limits of the evidence

Passing tests and short fuzz runs do not prove every input safe or every platform equivalent. Function coverage shows which functions ran, not exhaustive branch or state coverage. There are no Lean/Aeneas proofs in this repository. C compatibility intentionally differs where the reference reads uninitialized data on malformed input; see decoder codec deviations. Directory/resource checksums detect corruption and do not authenticate input.

Silesia comparison

AMD Ryzen 9 7950X (KVM, 2 vCPUs) · rustc 1.99.1 · C at -O3 · full Silesia files · 2026-11-10. Each standard graph compresses every listed Silesia file as one input with the library defaults (level 6, format version 27, both checksums). Every frame was verified byte-identical to OpenZL v0.3.0 C, and both decoders restored the input, before timing; compressed sizes are therefore equal. Throughput is MB/s (10⁶ bytes per second) of uncompressed bytes, from the median of alternating rounds; a graph’s figure is its total bytes over its total median time. “/ C” is mopenzl’s speed relative to C (above 1× is faster).

Raw measurements: silesia-2026-11-10/ · how to reproduce.

Per graph

zstd

lz4

lz

huffman

fse

entropy

huffman_pivco

flatpack

generic

field_lz_struct4

bitpack_num16

numeric16

numeric32

generic_num16

generic_num32

Encoder results

Published measurements compare mopenzl with the pinned OpenZL v0.3.0 C implementation on identical complete Silesia files. The matrix varies graphs and typed views at default level 6, format 27 and both checksums. Every inner frame was checked byte-identical before either implementation was timed.

AMD Ryzen 9 7950X (KVM, 2 vCPUs) · rustc 1.99.1 · C -O3 · 2026-11-10.

Throughput is uncompressed MB/s (10⁶ bytes/s), aggregated as total bytes over the sum of per-file median times. Compressed bytes are equal for both sides. The linked graph pages contain file-by-file numbers and charts. This dataset is a graph sweep, not a per-level benchmark; no unmeasured levels are implied.

Read the measurements

Use the full comparison for per-file charts and ratios, raw CSV for analysis, and commands to reproduce the API mode and corpus. Numeric/record views use only aligned compatible files, so compare each graph within its own corpus rather than ranking unrelated rows as one workload.

Other API modes

The parallel API records segment/task scheduling separately. The framed measurements record container overhead and independent-frame C timings. End-to-end results include different setup/allocation costs from reused per-graph benchmarks.

zstd · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

lz4 · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

lz · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

huffman · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

fse · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

entropy · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

huffman_pivco · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

flatpack · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

generic · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

field_lz_struct4 · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

bitpack_num16 · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

numeric16 · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

numeric32 · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

generic_num16 · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

generic_num32 · compress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

Decoder results

Published measurements compare mopenzl with the pinned OpenZL v0.3.0 C implementation on identical complete Silesia files. The matrix varies graphs and typed views at default level 6, format 27 and both checksums. Every inner frame was checked byte-identical before either implementation was timed.

AMD Ryzen 9 7950X (KVM, 2 vCPUs) · rustc 1.99.1 · C -O3 · 2026-11-10.

Throughput is uncompressed MB/s (10⁶ bytes/s), aggregated as total bytes over the sum of per-file median times. Compressed bytes are equal for both sides. The linked graph pages contain file-by-file numbers and charts. This dataset is a graph sweep, not a per-level benchmark; no unmeasured levels are implied.

Read the measurements

Use the full comparison for per-file charts and ratios, raw CSV for analysis, and commands to reproduce the API mode and corpus. Numeric/record views use only aligned compatible files, so compare each graph within its own corpus rather than ranking unrelated rows as one workload.

Other API modes

The parallel API records segment/task scheduling separately. The framed measurements record container overhead and independent-frame C timings. End-to-end results include different setup/allocation costs from reused per-graph benchmarks.

zstd · decompress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

lz4 · decompress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

lz · decompress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

huffman · decompress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.

All graphs · Encoder matrix · Decoder matrix · Download CSV · Reproduction

fse · decompress

Derived from the committed Silesia CSV; context and parameters are in the measurement record. C and Rust use identical frame bytes. Throughput is uncompressed MB/s (10⁶ bytes/s); ratios are uncompressed bytes / compressed bytes.