Decompressor
Ownership
classDiagram
class Decompressor {
DecoderConfig config
RetentionPolicy retention
Backend backend
Engine engine
Stream stream
}
class Engine {
Layout layout
State state
Vec~Stream~ buffers
Vec~Stream~ running
Vec hashed
Level simd
Option~Arc~BundleStore~~ store
Option~Arc~StoredBundle~~ bundle
}
class State {
ZstdDecoder zstd
Level simd
}
class Layout {
Vec~Node~ nodes
Vec~Source~ streams
lanes...
usize len
}
class Stream {
Phase phase
FrameInfo frame, pending_frame
Vec unit, peek, output
}
Decompressor *-- Engine
Decompressor *-- Stream
Engine *-- Layout
Engine *-- State
DecoderSession --> Decompressor : borrows
DecoderSessionOwned --> Decompressor : boxes
DecompressorBuilder::with_dictionaries(store) gives the engine the
BundleStore frames may name. DecoderConfig selects FrameMode (Single, the default, rejects trailing
bytes like ZL_decompress; Concatenated), DecodeLimits (input and output
caps) and both checksum checks (on by default, as the reference).
Parsing
core::reader::Reader returns Parse::Truncated when bytes run out and
Parse::Failed for invalid or unsupported data, so the same parsers serve
one-shot calls (truncation is UnexpectedEndOfInput) and sessions (truncation
means “wait”). frame::parse_with validates the magic, flags, output count
and types (DFH_decoderOutputTypes), known sizes, string counts, comment and
check byte, reporting each output’s OutputSpec; frame::parse wraps it for
the byte-oriented paths and rejects frames that are not one serial output
(Unsupported::MultipleOutputs / TypedOutput). Layout::parse reads a
chunk header’s lanes, maps each standard ID or registered custom ID to its codec descriptor
(codecs::standard; a disabled feature or unknown ID is Unsupported, a
codec newer than the frame’s version is a header error), then wires streams
as fillStoredStreams and DCTX_validateNodeStatic do and computes the
chunk’s length. Each node owns inputs (a stream range) and regens (a
range of Layout::regenerated, one stream per regenerated input); the last
outputs streams of the chunk are its outputs and may not feed a codec:
flowchart TD
A[chunk header lanes] --> B{every codec known?}
B -- no --> U[Unsupported Codec / CustomCodec]
B -- yes --> C[per codec: inputs after previous inputs]
C --> D{inputs past the outputs or over 100 000, regen out of range<br/>or taken, variable inputs or regen count off the signature?}
D -- yes --> G[InvalidData Graph]
D -- no --> E{codec input produced by a later codec?}
E -- yes --> G
E -- no --> F[stored streams fill the rest in order]
F --> H{stored count matches?}
H -- no --> G
H -- yes --> L[len = header + private headers + stored + checksums]
Chunk execution
Codecs are described by codecs::Desc: the types of the streams they
regenerate (regens, the last repeating when variable_regens), of their
fixed inputs and the type mask of variable inputs, the first format version
that has them, and a decode function. Each codec family lives in
codecs/<feature>.rs (or a directory) behind its feature; the
codec catalogue lists them all.
sequenceDiagram
participant E as Engine
participant L as Layout
participant C as codec decode fn
participant O as Out / TypedOutput
E->>L: parse(chunk)
E->>E: verify compressed checksum (before any codec)
E->>E: budget = 16 x remaining output + 1 MiB per stream
loop nodes in order
E->>E: input i = stream inputs.end - 1 - i (stored: serial width 1)
E->>E: check each input type against the signature
alt zstd regenerating the only output (byte paths)
E->>O: decode straight into the destination
else
E->>E: take regenerated streams out of buffers, prepare(type, budget)
E->>C: decode(ctx {header, state}, inputs, regens)
C->>C: Stream::fixed / strings size each output within the budget
E->>E: validate every regen (created, width, string sum, budget)
E->>E: put regens back into buffers
end
end
E->>O: hand over or copy the outputs (byte paths: one serial stream)
E->>E: verify content checksum over the outputs (+ string lengths)
How the last stream reaches the destination:
flowchart TD
S{source of the output stream} -- stored in the chunk --> P[Out::put copies it]
S -- regenerated by a codec --> E{destination}
E -- "empty Vec (decompress, empty decompress_into)" --> W["Out::put_vec swaps buffers: the stream becomes the output, the stream keeps the old empty allocation"]
E -- "non-empty Vec, slice, uninit" --> P
E -- "TypedOutput without content" --> A["TypedOutput::adopt takes the stream's buffers (Stream::take_content)"]
E -- "TypedOutput with content (later chunks)" --> C[TypedOutput::append copies]
data::Stream mirrors ZL_Data for regenerated streams: a type, an element
width (STREAM_validateTypeWidth: serial and string 1, numeric 1/2/4/8,
struct any non-zero), the content, string lengths for strings, and the
budget. Numeric content is little-endian on every host. Codecs read
data::Input views and may only create outputs through Stream::fixed and
Stream::strings, which reserve within the budget, so no frame can make the
decoder allocate beyond 16 times its declared output plus slack per stream.
Stream::fixed, Stream::strings, Stream::copy_of and Input::len are
compiled only with the codec features that call them (a #[cfg(any(..))]
list each, plus test), not kept under expect(dead_code): Rust 1.89 treats
an expected-dead function as used and its callees as live. Codecs report malformed data as InvalidData { CodecStream } and never
panic; differential tests check that they accept what the reference accepts
on damaged frames, apart from the documented cases where the reference reads
uninitialized memory.
Decompressor::decompress_typed returns one TypedOutput per frame output
(ZL_DCtx_decompressMultiTBuffer): Engine::decode_typed decodes a single
frame, appends each chunk’s output streams (output n is stream
total - 1 - n) after checking their type, width and declared size and
string count, and checks the content checksum over every output’s content
followed by its string lengths (STREAM_hashLastCommit_xxh3low32). A
regenerated output stream is moved into an output that has no content yet
(TypedOutput::adopt, with append’s width rules) rather than copied; the
engine’s buffer for that stream is reallocated by the next chunk. With a single
output and no string lengths the checksum is computed over the output in place;
otherwise the hashed bytes are staged in Engine::hashed.
On the byte paths the last regenerated stream is not read again after the
chunk’s codecs run, so Out::put_vec lets a Vec destination that is still
empty take the buffer by swapping it with its own (empty) allocation instead of
copying. Destinations that already hold bytes, slices and MaybeUninit
buffers copy through Out::put.
The zstd fast path decodes the magicless frame where it lies: mzstd reads
FrameFormat::Magicless natively, the content size is read first with
FrameHeaderInfo::parse (its absence is invalid, as in DI_zstd), and the
decoder writes straight into the sized destination; no staging buffer holds a
magic + frame copy any more. A node carrying a dictionary index decodes with
decompress_with_dictionary (slice, uninit and Vec destinations all take the
dictionary).
Dictionaries
sequenceDiagram
participant S as Session or one-shot
participant E as Engine
participant L as Layout
participant C as codec decode fn
S->>E: open_frame(frame info)
E->>E: frame.bundle? resolve in BundleStore, else DecodeError::MissingBundle
S->>E: decode chunk
E->>L: parse: dictionary lane gives Node.dict (index in the bundle)
E->>E: index past the bundle: InvalidData DictionaryIndex
E->>C: Ctx.with_dictionary(materialized dictionary)
Only Zstandard dictionaries are materialized. The bundle ID is read from the
frame header (FrameInfo::bundle, length byte plus ID with trailing zeros
trimmed); a frame without one skips the lane.
Frame metadata
The opt-in framing::FramedDecompressor validates outer resource records and
their seek directory, then calls frame_info and decompress_typed on each
independent frame. It preserves configured dictionaries, custom decoders,
backend, retention and inner limits; see framed containers.
Decompressor::frame_info and TryFrom<&[u8]> for FrameInfo
(decompressor::info) read a frame header without decoding any chunk, as
the reference’s ZL_FrameInfo_* getters do: the format version, every
output’s OutputInfo (type, content size, string count for strings and
None for struct and numeric outputs), the checksum flags, the header
comment and the dictionary bundle ID. The public type wraps the parser’s
core::frame::FrameInfo, which keeps the comment’s bytes for it.
tests/frame_info.rs compares every accessor with the C getters through
openzl-ffi’s frame_info shim.
flowchart LR
B[frame bytes] --> P["core::frame::parse_with"]
P --> I["FrameInfo: version, outputs, comment, bundle, checksums"]
I --> O["OutputInfo per output"]
P -. no chunk is read .-> X[chunks]
Sessions
stateDiagram-v2
[*] --> Frame
Frame --> Chunk: frame header complete (output limit checked)
Chunk --> Chunk: chunk complete -> decoded into pending output
Chunk --> Frame: end marker, Concatenated
Chunk --> Done: end marker, Single
Frame --> Done: Finish between frames (Concatenated, >= 1 frame)
Done --> [*]
A unit (frame header, chunk or end marker) is decoded once all its bytes are
present: in place when it lies whole in the caller’s input, otherwise from the
unit buffer. Its length comes from its header alone. A header split across
calls is measured over the buffered prefix plus a window of new input that
doubles from 64 bytes, so the session never accepts bytes past the unit it
builds (trailing data after a Single frame stays with the caller) and
measuring stays linear in the header’s length. The frame header found while
measuring is kept, not parsed again.
DecoderSession/DecoderSessionOwned add a sticky failure flag
(InvalidState afterwards), counters (total_in, total_out,
frames_decoded) and retention on completion. io::DecoderReader and
io::DecoderWriter wrap sessions with an 8 KiB buffer each.
Errors
DecodeError: InvalidData { kind } (Magic, FrameHeader,
HeaderChecksum, ChunkHeader, Graph, CodecStream, DictionaryIndex), Unsupported { feature } (FormatVersion, Codec, CustomCodec, UnknownSize,
MultipleOutputs, TypedOutput: the last two only from the byte-oriented
APIs), MissingBundle { bundle } (the frame names a dictionary bundle the
store lacks), UnexpectedEndOfInput,
TrailingData, OutputTooSmall, both checksum mismatches,
ContentSizeMismatch, both limits and InvalidState. It converts to
std::io::Error with matching kinds.
Known gaps
- Only Zstandard dictionaries are materialized; custom dictionary materializers are not implemented. See custom codecs for callback registration.
- Sessions and the byte APIs read frames with one serial output only;
typed and multi-output frames go through one-shot
decompress_typed. - The partition+bitpack decoder fusion is not ported; see each codec’s specification for which kernels are vectorized.
- Regenerated streams are copied where the reference references their inputs (conversions, dedup, concat).
- Session output is decoded into the pending buffer and copied out; one-shot calls decode in place (see uninitialized output).
- Slice and uninit destinations decode Zstandard through
mzstd::decompress_to_slice, which trails the C library by about 15% on records-like data, whileVecdestinations (mzstd::decompress_into) are ahead of it (see the comparison).
Registered application codecs use the same chunk layout, regenerated-stream
budgets and checksum validation. DataType also serves as the private stream
type, avoiding conversion at the callback boundary; its wire helpers remain
private. See custom codecs.