Codecs

A codec is the decoder half of a node: every standard OpenZL transform or coder has a node (the encoder side, a StandardNode) and a decoder in this crate, and both live behind one Cargo feature. This page lists them. Wire formats, invariants and the deviations on malformed data are in decoder codecs; how nodes are wired into graphs is in graphs.

Types are serial (bytes), struct (fixed-width records), numeric (little-endian integers) and string. “Since” is the first format version that has the node. The crate writes versions 21 to 27, so a node with a “since” of 21 or less exists at every version you can select; nodes introduced at 23, 24, 26 or 27 (lz4, lz, the bit-split family, partition, sentinel, sparse_num, pivco_huffman) are not available when you pick an older format version.

Features

all-codecs (in default) enables every feature below. Without a codec’s feature the encoder never writes it (a chunk that would use it is stored; a graph that runs its node fails, falling back only in permissive mode) and the decoder reports frames that use it as DecodeError::Unsupported. The decoder parses the chunk layout in full either way, so the error names the missing codec before any stream is decoded.

Feature names with a hyphen are Cargo features (field-lz); the library selects them independently, so features = ["std", "decompression", "zstd", "delta"] decodes only those codecs. Several standard graphs need more than one feature (the entropy graph uses most coders); a missing one makes the graph fail at run time. The scripts/check_codec_features.py script builds every combination.

Nodes

StandardNode::X below is the variant name for the registry ID; run a node from a function graph with GraphContext::run_node, or in a chain with GraphBuilder::pipeline. “Key parameters” are local parameters (see parameterized nodes) whose IDs are the reference’s; nodes not listed take none that matter.

Numeric transforms

Dictionary-style transforms

Splitting, merging and routing

Type conversions

The conversion feature covers IDs 10-26 and 57. They change the type of a stream, not its bytes, and appear in frames whenever typed data meets a serial-only codec.

Entropy coders

These come from OpenZL’s own FSE and Huffman fork, not from mzstd.

LZ and general-purpose coders

use mopenzl::graph::params::{divide_by, zstd};
use mopenzl::graph::{GraphBuilder, LocalParams, StandardGraph, StandardNode};
use mopenzl::input::{Input, Numbers};
use mopenzl::{Compressor, Decompressor, EncoderConfig};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut builder = GraphBuilder::new();
    // divide_by with an explicit divisor of 1000, then zstd at level 19.
    let divide = builder.parameterize_node(
        StandardNode::DivideBy,
        LocalParams::default().with_bytes(divide_by::DIVIDE_BY, 1000u64.to_le_bytes().to_vec()),
    )?;
    let zstd19 = builder.parameterize_node(
        StandardNode::Zstd,
        LocalParams::default().with_int(zstd::COMPRESSION_LEVEL, 19),
    )?;
    let to_bytes = builder.static_graph(zstd19, &[StandardGraph::Store.into()])?;
    let start = builder.pipeline(
        &[divide, StandardNode::ConvertNumToSerialLe.into()],
        to_bytes,
    )?;
    let graph = builder.build(start)?;

    let values: Vec<u8> = (0..4096u32)
        .flat_map(|i| (i * 1000).to_le_bytes())
        .collect();
    let numbers = Numbers::try_from((&values[..], 4))?;
    let mut compressor = Compressor::builder(EncoderConfig::default())
        .with_graph(graph)
        .build()?;
    let frame = compressor.compress_inputs(&[Input::from(numbers)])?;
    let outputs = Decompressor::default().decompress_typed(&frame)?;
    assert_eq!(outputs[0].content(), values);
    Ok(())
}

Custom codecs

Application-defined codecs are supported for compression and decompression at format versions 21 to 27. The custom-codec feature is included in all-codecs and enabled by default; it also works with no_std and alloc.

Implement custom_codec::CustomCodec:

• signature() declares the encoder’s input types, mandatory outputs and variable ports. Streams can be serial bytes, fixed-width records, numbers or strings. Numeric content is little-endian.
• encode() reads borrowed CodecInput views, commits outputs through EncoderContext::emit or emit_strings, and can send a private header with send_header. Every mandatory output must be committed exactly once.
• decode() receives the encoder’s output streams and private header, then restores the original inputs through DecoderContext::emit or emit_strings. The context validates widths, string lengths and the per-stream output budget.

Register the encoder with GraphBuilder::custom_codec(id, codec) and use the returned NodeId in a static graph, pipeline or function graph. Register the matching decoder with Decompressor::builder(config).with_custom_codec(id, codec)?. Both sides must agree on the wire ID, signature and encoded representation. Custom u32 IDs occupy a separate namespace from standard codec IDs; duplicate custom decoder IDs are rejected.

Custom nodes support parameterization and the regular one-shot and streaming APIs. Minimum-stream-size and store-on-expansion rules still apply: a small or expanding chunk may be stored instead of using the custom codec. Callbacks and signatures remain registered across operations and buffer trimming.

A frame carries the custom ID and its data, so the receiving application must register the decoder implementation. An unregistered ID, or a build without custom-codec, reports DecodeError::Unsupported { feature: Unsupported::CustomCodec { id } }. Custom dictionary materializers are not implemented.

See the API example for a complete implementation and registration example, and the custom codec specification for ownership, wire format, validation and error propagation.

Legacy codecs

Legacy codecs whose newest format version is below 21 (rolz, fastlz, zstd_fixed and the deprecated FSE and Huffman nodes) are not registered and decode as Unsupported::Codec.