Architecture

These specifications describe mzstd as it is implemented today: module and ownership boundaries, public APIs, data flow, state machines, errors and known gaps. zstd-ffi/vendor/zstd (zstd v1.5.7) is the reference implementation, and doc/zstd_compression_format.md in that tree is the format specification. To use the crate, read the guides instead; for checks and repository layout, see development.

Where to start

Specifications share an outline: module and ownership boundaries, public surface, control and data flow with Mermaid diagrams, and error propagation. What is missing or slower than the reference is listed under known gaps.

Module map

graph TD
    Root[mzstd public facade] --> Encode[compressor: compression feature]
    Root --> Decode[decompressor: decompression feature]
    Root --> Ret[RetentionPolicy]
    Root --> Back[Backend: SIMD level]
    Root --> IO["io: reader and writer adapters (std)"]
    Root --> Dict["dictionary: DictionaryContent, DictionaryId, DictionaryBuilder, PreparedDictionary, DecodeDictionary, DictionaryRef"]
    Decode --> Dict
    Encode --> Dict
    Dict --> DCore
    Dict --> ECore
    Encode --> Cfg[config + internal lowering]
    Encode --> ECore["compressor::core (private): engine, match finders, entropy, streaming"]
    Encode --> Par["parallel (std): batches, random-access sources, sessions, io adapters + parallel::core (private): segments, inputs, stream"]
    Par --> ECore
    Decode --> DCore["decompressor::core (private): frames, blocks, entropy, history"]
    Root --> Framing["framing (framed feature): neutral types, wire constants"]
    Framing --> FEnc["compressor::framing: FramedCompressor, sessions, std adapters + core: container, resource, driver"]
    Framing --> FDec["decompressor::framing: FramedDecompressor, sessions, FramedReader, FramedSeekReader + core: engine, wire, seek"]
    FEnc --> Encode
    FDec --> DCore
    ECore --> Shared["shared (private): format constants, XXH64"]
    DCore --> Shared
    IO --> Encode
    IO --> Decode
    Ref["zstd-ffi: zstd 1.5.7 + oracle + corpus"] -. tests, benches, fuzzing .-> Root
    CABI["mzstd-ffi: C ABI crate, one-shot and sessions"] --> Root
    Fuzz["fuzz/afl (excluded package)"] -.-> Root
    Fuzz -.-> Ref
    LibFuzz["fuzz/libfuzzer (excluded package): ClusterFuzzLite adapters"] -.-> Fuzz
    Cmp["benchmarks/comparison (excluded package): cold reports and warm profiling against C"] -.-> Root
    Cmp -.-> Ref
    Lean["verification/lean (excluded): Lean 4 proofs against RFC 8878"] -. Charon/Aeneas translation .-> ECore
    Lean -.-> DCore
    Lean -.-> Shared

Public modules own configuration, ownership, sessions and high-level errors. Private core modules own algorithms and state machines. Their errors (CoreError, the decoder’s internal errors) never appear in public signatures. Each codec compiles on its own, with or without std.

Common boundaries

Compression

From the public Compressor down to the engine, match finders and entropy coding.

Decompression

The independent decoder, its fast paths and its output contracts.

Containers and integrations

The framed container format, the OpenZL-facing surface and the C ABI.

Development tools

How the repository measures, tests, proves and publishes itself.

Known gaps

• Scope. Both codecs take raw and Zstandard-format dictionaries (dictionaries, decompressor); the parallel compressor and the C ABI do not. Magicless frames are written by Compressor and read by the decoder (not by the parallel or framed compressors, nor the C ABI; OpenZL integration). There are no external sequence producers, and no legacy (v0.x) frames.
• Speed. A Backend (fearless_simd level) is chosen once per codec and dispatched once per block. In the cold comparison against zstd 1.5.7 (benchmarks, Ryzen 9 7950X under KVM, AVX-512, 2026-11-09, host loaded by other tenants) compression runs at a median 1.248x the reference’s speed, faster in 174 of 192 dataset/level pairs, and decompression at 1.517x, faster in 151. These medians include empty and tiny inputs. The encoder still trails on Alice at ten levels (0.84-0.998x in that noisy run; 0.94-0.99x on 2026-10-30), and on the whole silesia.tar in lzbench at most levels; fast executes about 127 instructions per match against the reference’s 91 (encoder core). The decoder trails on compressible text (Alice 0.948x and cyclic Alice 0.937x in the median) and in lzbench at every level (0.92-0.96x of zstd): its four-stream Huffman loop executes about 242 instructions per round against the reference’s 221 (decoder fast paths).
• Parallel frames. They decode anywhere, but they are not identical to the serial encoder’s frames or to ZSTDMT’s: segments are primed with their overlap the way ZSTDMT primes a job, but the overlap is capped at a quarter of a segment and LDM runs per segment.
• Decoder memory. Streaming history may reach twice the window plus a block. Allocation follows actual output.