Encoder codecs

Every standard node of OpenZL v0.3.0 has an encoder in compressor::core::codecs, behind the Cargo feature of its codec. This page is the encoder-side counterpart of the decoder catalogue: it maps registry node IDs to wire codec IDs, lists their ports, and records the mechanics and deviations of each family. The engine that runs them is described in compression graphs; the graphs that wire them together in standard graphs.

Module map

graph TD
    REG["registry::nodes: STANDARD_NODES, 85 rows"] -->|"encoder: &dyn Encode"| ENC
    subgraph ENC ["compressor::core::codecs"]
        ZN["zstd_node + core::zstd"]
        CV[conversion]
        NUM["numeric/: delta, zigzag, divide_by, range_pack, bitsplit, float_deconstruct, sparse_num, merge_sorted, sentinel, partition"]
        STR["structure/: transpose, split, split_by_struct, dispatch, concat, dedup, interleave, prefix, tokenize, parse_int"]
        PK["bitpack, bitunpack, flatpack, constant, quantize, mux_lengths"]
        EN["entropy, pivco_huffman"]
        LZ["lz4/, lz/, field_lz/, lz_common/"]
        DIS["DISABLED / LEGACY"]
    end
    ZN --> MZ[mzstd]
    EN --> SE["shared::entropy + shared::stats"]
    PK --> SB[shared::bitpack_kernels]
    NUM --> SN[shared::numeric]
    LZ --> SS[shared::stats::count]

Each encoder is a FnEncoder static wrapping fn(&mut NodeCx, &[Stream]) -> Result<(), NodeError>. A registry row selects if cfg!(feature = "x") { &codecs::x::ENCODER } else { &codecs::DISABLED }; families with several features (numeric, structure) alias the single nodes to DISABLED inside their module. DISABLED fails the node with NodeError::Disabled, LEGACY (deprecated nodes 76 to 85, never reachable at v21 and newer) with NodeError::Legacy. There is no “not ported” encoder any more: every standard node runs.

The key invariants every encoder keeps:

• The output bytes equal the reference’s for the same input, parameters and format version (tests/encode_*.rs compare node by node).
• A node fails exactly where the C encoder fails, because permissive mode, trial compression and the store-on-expansion fallback depend on it.
• At most one codec header per run (NodeCx::send_header); outputs are created with emit, emit_with_metadata, emit_strings, reference or reference_strings, and integer output metadata with set_output_metadata.
• Local parameters follow ZL_Encoder_getLocalParam: NodeCx::int, ints, copy_param, ref_param (reference parameters carry Arc-shared Rust values instead of C pointers). Parser-function and callback parameters are not supported and fail the node.

Node table

Generated from registry/nodes.rs (node IDs are the registry’s, not the wire codec IDs). “Versions” is the format-version range the node exists in.

Node 56 does not exist in the reference table. Nodes 72 and 73 (tokenize_*_sorted) share the tokenize encoders with a default local parameter that sorts the alphabet. bit_split (75) and bitsplit_top8/fp/bf16 (45 to 47) share one encoder with different splits. Numeric, struct and string conversion nodes 10 to 26 are the implicit conversions of the engine (conversion feature).

Mechanics by family

Zstandard (node 62, zstd)

• Output stream is varint(element width) followed by the magicless Zstandard frame: mzstd writes FrameFormat::Magicless natively, so no staging or magic overwrite exists. The default-graph fast path (ZstdNode::encode) pushes [0x01] then compress_into.
• Parameters are applied in the reference’s order: the global level, then literalCompressionMode = Disabled when the decompression level is 1, then the node’s local integer parameters (raw ZSTD_c_* IDs) through CParameter::try_from, check_memory_cap(3) and with_parameter. format (10) and contentSizeFlag (200) are refused; window, hash, chain and long-distance hash logs stop three below their maximum.
• Block splitting (encode_elements: 2 to 8 elements of at least 1024 bytes, one flushed block per element with a pledged size) is ported and tested against the enc_zstd C oracle, but is unreachable from node 62 at v21 and newer (the registered node takes serial input only; the struct variant is v3 to 10).
• A node with a dictionary compresses through refCDict equivalent (compress_with_dictionary_into); the block-splitting route ignores it, as C does. See dictionaries.

Conversions (conversion)

Nodes 10 to 26 and set_string_lens (57): reinterpret between serial, struct, numeric (little and big endian) and string without copying (reference), separate_string_components splits a string into serial content and numeric lengths. These are also the engine’s implicit conversions.

Numeric (numeric/)

Shared helpers live in shared::numeric (feature-gated by the codecs that call them; unused NUMOP_* functions are #[cfg(test)]).

Structure (structure/)

Public helpers GraphContext::run_split_node, run_dispatch_node and run_dispatch_string_node wrap the reference-parameter forms for user graphs.

Packing

• bitpack_serial / bitpack_int (63, 64): nbBits = 1 + highbit(max); header byte log2(width) << 6 | (nbBits - 1) plus a padding byte; values LSB first. nbBits comes from bitpack_kernels::nb_bits, an OR reduction over 64-byte fearless_simd lanes (the highest set bit of the OR is that of the maximum), dispatched once on cx.simd and folded onto the element’s byte positions. Fields of up to 16 bits are packed eight at a time: eight fields are exactly nbBits bytes, composed in a u128 and stored whole into a pre-sized buffer (encode_groups); wider fields use the one-field encode_scalar.
• bitunpack (27): width from nbBits; header [nbBits] plus trailing bits from v7.
• flatpack (65): sorted distinct alphabet and the indices plus a terminator; eight indices (exactly bits bytes) are composed in a u64 per store, the < 8 tail and the terminator bit are written separately.
• constant (70, 71): one element and a varint count.
• mux_lengths (53): multiplexes literal and match lengths; header split | bias << 4, the bias taken from the LZ metadata.
• Kernels live in shared::bitpack_kernels (encode, decode, encode_bound, and writer::BitWriter, the port of ZS_BitCStreamFF).

Entropy and LZ

See entropy (fse_v2, fse_ncount, huffman_v2, huffman_struct_v2, pivco_huffman) and LZ (lz4, lz, field_lz, quantize, mux_lengths).

Deviations from the reference

• divide_by and partition use the portable kernels; merge_sorted scans k-way; SIMD variants of the reference are not ported (outputs are identical).
• Encoder-side transpose 2 and transpose_split2/4/8 are not ported: the encoder’s maximum useful version for them is 10, below the supported range.
• Custom tokenizers and parser-function parameters are unsupported.
• tokenize (struct and numeric) finds first occurrences with an open-addressing TokenMap (linear probing, load factor 1/2) and records each element’s id while scanning, so sorting only remaps ids to ranks; strings still use a BTreeMap.
• transpose_split runs shared::byte_transpose::split: for widths 2, 4, 8 the elements are deinterleaved sixteen at a time with log2(W) rounds of unzip_low/high on u8x16, dispatched on cx.simd; the tail and other widths are scalar.

lz4 (HC chain walk)

The graph level 6 maps to lz4hc hash chains (32 attempts, <PATTERN = false, SWAP = false>). The hot path is Hc::insert_and_get_wider_match, which is #[inline(always)] so each call site (find_best_match, the two insert_and_get_wider_match calls of the lazy parser, the optimal parser) gets its own copy: the lookback is a constant at the first-search site, and the call needs no frame or Match return through memory (the C compiler inlines it the same way). Reads in the chain loop are single-comparison checked: peek16 tests at <= len - 2 once (a wrapped, negative position fails it and reads as -2), count bounds both 8-byte reads with limit <= len, p_match < p_in and p_in + 8 <= limit, and count_back slices [ip - 8, ip) and [matched - 8, matched) after matched <= ip <= len. Anything outside those cases falls to the unchanged count_long / count_back_long loops.

flowchart LR
    A[find_best_match / lazy parser] --> B[insert_and_get_wider_match inlined]
    B --> C[LZ4HC_Insert: hash and chain every position before ip]
    C --> D{chain walk, up to 32 attempts}
    D -->|peek16 and read32 match| E[extend: count_back + count]
    D -->|no| F[next = index - chain delta]
    E --> F
    F --> D
    D -->|done| G[Match len, off, back]

Known gaps

• No SIMD in any encoder kernel except the XXH3 checksum; the hotpath anchors cover only the chunk writer, the zstd node and XXH3, so the new codecs have not been profiled.
• LZ4 has only the fresh-stream, not-limited entry points (no streaming or limitedOutput modes).
• Some shared helpers (shared::bitpack_kernels::writer, shared::stats::count and data_stats, parts of shared::estimate) carry expect(dead_code) gates for feature sets that do not use them.
• The C prefix encoder crashes on an all-empty string input, so the oracle tests cannot cover that shape.