OpenZL compression and decompression in safe Rust

A round trip

[dependencies]
mopenzl = "0.2.3"

use mopenzl::{Compressor, Decompressor, EncoderConfig, Level};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = "hello, smaller world! ".repeat(1000);
    let mut encoder = Compressor::new(EncoderConfig::default().with_level(Level::try_from(6)?))?;
    let frame = encoder.compress(input.as_bytes())?;

    let mut decoder = Decompressor::default();
    assert_eq!(decoder.decompress(&frame)?, input.as_bytes());
    Ok(())
}

Requires Rust 1.89 or later. Without a graph the compressor runs the default, compress_generic, which uses Zstandard for serial bytes. Getting started continues from here.

Tell it the shape of your data

OpenZL’s advantage over a general-purpose compressor is typed input. Say that a buffer holds 32-bit integers, and a numeric graph transforms them before an entropy coder sees them:

use mopenzl::graph::{CompressorGraph, StandardGraph};
use mopenzl::input::{Input, Numbers};
use mopenzl::{Compressor, DataType, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 10 000 slowly growing 32-bit timestamps, little-endian.
    let values: Vec<u8> = (0..10_000u32)
        .flat_map(|i| (1_700_000_000 + i * 3).to_le_bytes())
        .collect();
    let numbers = Numbers::try_from((&values[..], 4))?;

    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(CompressorGraph::standard(StandardGraph::SelectNumeric))
        .build()?;
    let frame = compressor.compress_inputs(&[Input::from(numbers)])?;
    println!("{} -> {} bytes", values.len(), frame.len());

    let outputs = Decompressor::default().decompress_typed(&frame)?;
    assert_eq!(outputs[0].data_type(), DataType::Numeric);
    assert_eq!(outputs[0].width(), 4);
    assert_eq!(outputs[0].content(), values);
    Ok(())
}

The decoder needs no graph: the frame records the codecs that produced it. Structured data covers numbers, records and strings.

Pick a graph

Graphs has the full table, and codecs lists every node and the Cargo feature that enables it. You can also compose nodes yourself with GraphBuilder or register a custom codec.

Find your path

What you can rely on

• Byte-identical serial encoder. For the same input, graph and global parameters, frames are exactly those of OpenZL v0.3.0 at format versions 21 to 27. See what is claimed and how it is checked.
• Safe implementation, not a C wrapper. Production code forbids unsafe. Zstandard comes from the safe-Rust mzstd crate; every other codec is ported here. The C library is only a test and benchmark oracle.
• No hidden I/O or threads. Sessions touch only the buffers you pass in, and parallel compression hands tasks to your scheduler. The library does not create a thread pool.
• Measured against C. The benchmarks show where mopenzl is faster than C and where it is still behind.

Framing requires the opt-in framed feature. The book tracks main, including unreleased changes; documented recipes are checked against the local checkout.