Architecture

Specifications describe how the implementation works today: ownership, APIs, data flow, state transitions, SIMD dispatch, errors and known gaps. To use the crate, read the guides instead; for checks and repository layout, see development.

Where to start

Specifications share an outline: module and ownership boundaries, public surface, control and data flow with Mermaid diagrams, and error propagation. What is missing or slower than the reference is listed under known gaps.

Module map

graph TD
    Root[mopenzl public facade] --> Encode[compressor: compression feature]
    Root --> Decode[decompressor: decompression feature]
    Root --> Framed["framing: opt-in framed resource containers"]
    Framed --> FCore["private framing::core: resource assembly, units, directory"]
    FCore --> Encode
    FCore --> Decode
    Root --> GraphApi["graph, input: graph builder, typed inputs (compression)"]
    Root --> Custom["custom_codec: application callbacks, signatures and contexts"]
    Custom --> ECore
    Custom --> DCore
    Root --> DictApi["dict: Dict, DictBundle, BundleStore"]
    Root --> Common[Backend, RetentionPolicy, FormatVersion, DataType, io]
    Encode --> Parallel["parallel: caller-scheduled tasks, batches, streams (std)"]
    GraphApi --> Parallel
    Parallel --> PCore[private parallel::core: workers, staging, ordered assembly]
    PCore --> ECore
    Encode --> Dest[shared::destination: initialized or MaybeUninit output]
    Decode --> Dest
    GraphApi --> Reg
    Encode --> ECore[private compressor::core]
    Decode --> DCore[private decompressor::core]
    ECore --> EStream[core::stream: session state machine]
    ECore --> EChunk[core::chunk + core::frame: specialized default-graph writer]
    ECore --> EZstd[core::zstd: Zstandard node]
    ECore --> EGraph[core::exec + layout + rtgraph: graph engine]
    EGraph --> Reg["core::registry: 85 nodes, 90 graphs, dict state"]
    Reg --> ECodecs["core::codecs: every node encoder, one module per feature"]
    Reg --> EGraphs["core::graphs: standard graphs, selectors, segmenters"]
    EGraphs --> Special["transformer, ml_selector, clustering, sddl, sddl2, entropy, lz"]
    DCore --> DStream[core::stream: unit assembly]
    DCore --> DChunk[core::chunk: chunk graph layout]
    DCore --> DZstd[core::zstd: magicless Zstandard decoder]
    DCore --> DCodecs[core::codecs: every standard codec, one module per feature]
    DCore --> DData[core::data: typed streams]
    EChunk --> Shared[private shared]
    ECodecs --> Shared
    EGraphs --> Shared
    DChunk --> Shared
    DCodecs --> Shared
    Shared --> Prims["wire, varint, bitpack, xxh3, sha256, dictwire"]
    Shared --> Algo["entropy, stats, estimate, numeric, bitpack_kernels, byte_transpose, cbor"]
    Prims --> SIMD[fearless_simd: selected backend]
    ECodecs --> SIMD
    DCodecs --> SIMD
    EZstd --> Mzstd[mzstd: safe-Rust zstd 1.5.7]
    DZstd --> Mzstd
    C[openzl-ffi: OpenZL v0.3.0 C] -. tests and benchmarks .-> Root

Public modules own ergonomic configuration and high-level errors. Private core modules own algorithms and state machines; implementation errors, SIMD types and mzstd types do not appear in public signatures. Each codec compiles independently.

SIMD dispatch boundary: the fearless_simd::Level is chosen once per Compressor / Decompressor (its Backend) and reaches every node encoder as NodeCx::simd and every codec decoder as State::simd. Each codec dispatches on it once per stream (PivCo-Huffman once per tree node) and runs monomorphized kernels inside the dispatched function; every kernel keeps a scalar baseline that tests compare at each level the host offers.

Common boundaries

The wire format, feature gates and primitives both directions share.

Compression

From the public Compressor to the graph engine and node encoders.

Codec families

Selectors, interpreters and coders with enough machinery for a specification of their own.

Decompression

The universal decoder and its codec catalogue.

Containers

The opt-in framed container around ordinary frames.

Development tools

How the repository measures, tests and publishes itself.

Known gaps (whole crate)

• Scope: every non-legacy OpenZL v0.3.0 node, graph, selector and segmenter is ported, on both sides. Deprecated nodes 76 to 85 and format versions below 21 are not supported; unregistered custom codecs decode as Unsupported.
• Dictionaries: only Zstandard dictionaries materialize; no training (dictionaries).
• Performance: fearless_simd kernels cover XXH3, the LZ row-table tag match, PivCo-Huffman (128-bit, plus 512-bit expand/compress on AVX-512), bitpack bit counting, byte transposes and flatpack. FSE, Huffman, LZ4 and the LZ decoders are tuned scalar loops compiled inside the dispatched level. Huffman decoding and the numeric and field_lz graphs remain 0.7 to 0.9× of C on Silesia; LZ4, FSE and Zstandard are within about 10% (comparison). The partition+bitpack decoder fusion is not ported.
• no_std builds use the libm crate for floating point; the Transformer and estimator decisions are byte-identical with the reference only with std (transformer).
• No C ABI crate. Parallel compression supports serial inputs; streaming holds compressed chunks until the total input size is known.
• Verification gaps: no AFL targets for the graph engine, SDDL or CBOR configurations; new SIMD kernels and uninit paths require explicit Miri and ASan selections (fuzzing, CI).