Decoder fast paths

How decompressor::core regenerates blocks at speed: SIMD dispatch, the bit readers, the Huffman and sequence loops, the copies, and the output paths. Every fast path decodes exactly the bits the reference decoder (zstd 1.5.7) reads, in the same order, so it accepts and rejects the same streams as the straightforward port it replaced; only refill timing and copy granularity differ. The crate stays forbid(unsafe_code): bounds are checked, and the code is shaped so the compiler can drop most of the checks (see index masks).

Dispatch

sequenceDiagram
    participant D as Decompressor (Backend)
    participant S as Stream
    participant F as FrameDecoder
    participant B as BlockDecoder::decode
    participant H as HuffmanTable::decode_streams
    participant Q as SequenceTables::execute
    D->>S: Stream::new(config, backend)
    S->>F: decode_block(backend, ...)
    F->>B: decode(backend, ...)
    alt raw / RLE block
        B->>B: copy_from_slice / fill
    else compressed block
        B->>B: MatchCopy from backend.has_fast_shuffle()
        B->>B: fearless_simd::dispatch!(backend.block_decode_level(), simd => ...)
        B->>H: #[simd] kernel: literals
        B->>Q: #[simd] kernel: sequences
    end

Backend (public, src/backend.rs) wraps a fearless_simd::Level detected once when the decoder is built (DecompressorBuilder::with_backend overrides it). BlockDecoder::decode dispatches once per compressed block. The two hot loops are separate #[fearless_simd_macros::simd] functions: each is compiled per level with that level’s target features (on AVX2 that includes BMI2, so bit reads use shrx/bzhi), and keeping them apart keeps each loop’s register allocation independent. MatchCopy::Shuffle is chosen on levels with a native byte shuffle (SSSE3 and up, NEON, WASM SIMD); sse2 and fallback use MatchCopy::Scalar.

Backend::block_decode_level runs the block loops of an AVX-512 backend with AVX2’s features. The loops are scalar apart from 16- and 32-byte copies, which AVX2 already has; with AVX-512’s cheap gathers the compiler vectorized the four Huffman streams’ independent lookups across lanes (vpgatherdd, vpsllvq, lane extracts), lengthening every stream’s chain: a pair loop written as the reference’s took 101.8 µs on Alice level 1 compiled for AVX-512 against 87.2 µs for AVX2 (Ryzen 9 7950X).

That choice cannot hold where AVX-512 is in the compile-time baseline (-C target-cpu=native on Zen 4 or Ice Lake and later, as lzbench builds): fearless_simd::dispatch! promotes every level to Avx512 there (Level::__dispatch_target), and target features can only be added, so the block loops always see AVX-512. HuffmanTable::decode_streams and decode_streams_shared therefore also ask their own feature context (serial, folded per backend) and, given AVX-512, decode the fixed-buffer pair loop one stream at a time (pair_rounds_serial, see Huffman literals), which the compiler leaves scalar.

Bit readers

classDiagram
    class BitContainer {
        container: u64
        avail: u32
        pos: usize
        +new(src) Option
        +look(n) / look_nonzero(n) / read(n) / skip(n)
        +reload_fast(src)
        +reload(src) bool
        +overflowed() bool
        +is_exhausted() bool
    }
    class Sentinel {
        bits: u64
        pos: usize
        +from_container(BitContainer, src, start)
        +into_container(src, start) BitContainer
        +reload_within(src, limit)
        +peek(shift) / skip(n)
    }
    BitContainer <--> Sentinel : Huffman fast loops

• BitContainer keeps the eight bytes at pos and the count avail of unread low bits. look subtracts the requested width, shifts and masks; a zero-width read returns zero. A full refill while pos >= 8 moves back by (64 - avail) / 8 bytes. Near the start, the cautious refill is cold and takes the container by value so the hot loop can keep it in registers. At pos == 0 it returns directly without that cold call. Valid exhaustion is pos == 0 && avail == 0; overreading wraps avail above 64 and refills preserve that overflow until validation rejects it. The Huffman weight decoder checks the same overflow state.
• Sentinel is the reference’s fast-loop stream (HUF_DecompressFastArgs): unread bits at the top, a 1 below them shifted up with every read, so the consumed count is the sentinel’s position (trailing_zeros) and a lookup is one shift. Positions index the buffer holding all four streams. reload_within(src, limit) refills at the position clamped to limit, the last in-bounds one: the fast loops’ round counts keep every position below it, so the clamp never changes a value, but it replaces a bounds-checked load and its fallback branch with a compare and a cmov. Each stream keeps its own position and output count, so its tail takes over where its fast rounds ended. Pair rounds may refill across a later stream’s start within the common input allocation. into_container moves back to that stream’s first byte and subtracts prefix bits from avail, preserving both exact exhaustion and overread errors before the stream-local tail runs.

Huffman literals

flowchart TD
    Read["HuffmanTable::read: weights 4 per refill, rows filled by weight in runs"] --> Kind{four streams?}
    Kind -->|no| Tail1[decode_tail: HUF_decodeStreamX1]
    Kind -->|yes| Open[open four BitContainers]
    Open --> Fast{every pos >= 8?}
    Fast -->|no| Tails
    Fast -->|yes| Sel{prefers_pairs: HUF_selectDecoder}
    Sel -->|X1| Single[single_rounds: 5 symbols per stream per round, 4 if 12-bit tree]
    Sel -->|X2, tree up to 11 bits| Fixed{buffer of FIXED_OUT bytes?}
    Fixed -->|yes| Avx{AVX-512 features?}
    Avx -->|no| PairsFixed[build_pairs if stale; pair_rounds_fixed: one masked position per stream, streams 1-3 interleaved, stream 4 with the refills]
    Avx -->|yes| PairsSerial[build_pairs if stale; pair_rounds_serial: streams 1-3 interleaved, stream 2's refill behind a branch, then stream 4 in its own block]
    Fixed -->|no| Pairs[build_pairs if stale; pair_rounds: per-round windows, streams 1-3 interleaved, then stream 4]
    Single -. one WriteAhead step per round .-> Ahead[prefetchw: next block's output]
    PairsFixed -. one WriteAhead step per round .-> Ahead
    PairsSerial -. one WriteAhead step per round .-> Ahead
    PairsFixed --> Back
    PairsSerial --> Back
    Single --> Back[Sentinels back to containers]
    Pairs --> Back
    Back --> Tails[decode_tail per stream: exact end check]
    Tail1 --> Verdict{exact?}
    Tails --> Verdict
    Verdict -->|no| Reference["decode_as_reference (cold, out of line): huffman::reference"]

• Single rows are built for 12-bit lookups whatever the tree depth (each code’s rows repeated 1 << (12 - log) times), so lookups need no mask. A row packs the code length in its low byte and the symbol in its high byte (HUF_DEltX1), so a Sentinel shift by the whole row (shlx masks to six bits) consumes the code. HuffmanTable::fill counting-sorts the symbols by weight and writes each weight’s runs, all of one length and aligned to it, with fixed-size stores (fill_runs::<1|2|4|8>, 16-row chunks beyond), as HUF_readDTableX1_wksp does; weight streams decode four symbols per refill while a whole word can be refilled, then follow the reference’s per-symbol overflow tail. Reading a tree costs about 35% fewer instructions than the per-symbol fill, with far fewer mispredictions.
• single_rounds splits the four output segments once and walks them as fixed-size chunks (windows, single_round), storing each stream’s five (or four) symbols straight into its chunk, with no per-stream window check and no array gathered on the stack: 176 instructions per 20 symbols against the reference’s 163.
• The fast loops count the rounds every stream can afford up front (each refill moves back at most seven bytes; outputs need room for the round’s stores), run them without per-stream tests, then recount. Single rounds remain within each input stream. Pair rounds use the common input bounds; conversion back to stream-local containers restores strict end checks.
• Pair rows (HUF_DEltX2) use 11-bit lookups (HUF_DECODER_FAST_TABLELOG): 2048 rows, half the size and build cost of 12-bit ones, a row pairing the code at the top of its index with the next only when both fit in 11 bits. A PairRow packs the bit count (byte 0, so a whole-row shift consumes them), the symbols (bits 8-23) and the symbol count (the top two bits, read with one shift and no mask). pair_rounds does five lookups per stream per refill (55 bits), as the reference’s fast loop does; each stores two bytes into a 14-byte window of the stream’s segment, and the count’s range (at most 3) lets the compiler prove the five stores stay inside it without a mask or per-store checks. The output is split into four non-overlapping segments once; every round takes a 14-byte window inside each segment. The first three streams interleave their five lookups, exposing three independent lookup chains to the processor. The fourth stream then decodes its five lookups separately, limiting the live state during the interleaved work. Refill positions and each segment’s window offsets are clamped to their last in-bounds values, which the round counts never pass. Rounds are limited to (end - op - 4) / 10 per stream. Trees of 12 bits, which the reference accepts but never writes, build no pair rows and decode with the single loop. Pair rows are rebuilt only after a new tree, and a test checks every row of trees of every depth against the definition.
• pair_rounds_fixed is the reference’s register layout: each stream keeps only its container and one output position live, where a window costs a base and an offset. Large four-stream sections (at least FIXED_MIN, 4 KiB) decode into a literal buffer of at least FIXED_OUT (BLOCK_SIZE_MAX + PAIR_WINDOW) bytes, which decode_streams receives with the section size; smaller sections keep a buffer of their own size, so small frames zero nothing more. Each round masks the four positions to BLOCK_SIZE_MAX - 1 (no section position exceeds it) and advances them by at most three per lookup by the count’s type, so every two-byte store provably lands inside the array with no check; the round counts still keep each stream inside its own segment. As in the reference, the fourth stream’s lookups interleave with the four refills and its position ends the inner loop (op4 < olimit, with olimit five symbols per affordable round), so no round counter is live. On Alice level 1 the round executes 242 instructions instead of 295 (the reference: 221), and decoding takes 2.6% less time.
• pair_rounds_serial replaces pair_rounds_fixed in AVX-512 feature contexts. Given AVX-512, LLVM’s SLP vectorizer packs the lockstep loop’s four branch-free containers into one ymm register: one vpsrlq makes four lookup indices, vpgatherqd loads four rows, vpsllvq consumes them, every row is extracted again for its scalar store, and the four refills become a vpgatherqq. That runs the four streams as one dependency chain through the gathers. The AVX-512 loop runs the same rounds with the same bounds over the same masked positions, but no basic block holds all four streams: streams 1-3 interleave their five lookups and refill, stream 2 with Sentinel::reload, whose bounds test is a branch the compiler cannot remove, streams 1 and 3 clamped with reload_within; stream 4 then decodes its round in its own block, which the rotated loop ends with its exit test. SLP finds no four isomorphic chains in one block, and with the branch no three refills either; the processor overlaps three lookup chains at a time. Earlier the loop ran one stream at a time, each refill behind a branch: that left the scheduler full of one stream’s waiting lookups, and on x-ray (1 MiB, level 1, almost all four-stream literals) a decode took 3.66M cycles against zstd 1.5.7’s asm loop at 2.55M; the current order takes 3.25M (2026-11-08, Ryzen 9 7950X VDS, native build). Measured and rejected there: four streams interleaved with every refill behind a branch, or with branch-free refills in a rotated loop (both vectorized again, 1.6-2x slower), two pairs of streams (3.55M), and a pair row whose symbols sit in the low 16 bits with the bit count rotated into place (rorx; LLVM emits mov and shr for it, no fewer instructions).
• A code of b bits fills a run of 1 << (11 - b) pair rows whose jth row continues with the code at row j << b, the same series for every code of that length (the reference’s HUF_fillDTableX2Level2 per rank). The first run of each length is built from the single rows; later runs of that length copy it with one add per row that swaps in their own first symbol.
• Selected pair streams continue with four pair lookups per refill while the remaining output has an 11-byte window. Every stream then finishes with the single-symbol table in the cautious loop and must be consumed exactly; pair decoding never decides how a stream ends. A section that fails that check is decoded again by the scalar port of the reference (huffman::reference, see Huffman stream ends), whose verdict stands. It is #[cold] and #[inline(never)], and only error paths reach it, so the loops above and the sections they accept are unchanged; the hot path’s choice of pair loops (prefers_pairs over the streams’ size) is independent of the reference’s decoder type, which only the port uses.
• The loops reach a table through TableAccess: the decoder’s own table (&mut HuffmanTable, decode_streams) builds its pair rows on first use, a dictionary’s shared table (&HuffmanTable, decode_streams_shared) only uses rows built when the dictionary was parsed. Both are monomorphized, so the own-table path compiles as before.

Sequences

flowchart TD
    Header[read_header_on: count and per-symbol modes] --> Mode{table mode}
    Mode -->|Basic| Static[borrow static predefined rows]
    Mode -->|RLE or Compressed| Custom[allocate or rebuild private rows]
    Mode -->|Repeat| Reuse[reuse static, private or dictionary rows]
    Static --> Init[initial states, reload]
    Custom --> Init
    Reuse --> Init
    Init --> Next{sequences left?}
    Next -->|yes| Dec[SequenceDecoder::decode: ZSTD_decodeSequence]
    Dec -->|zero repeat offset| EOff[Offset]
    Dec --> Exe[Executor::execute: ZSTD_execSequence]
    Exe -->|literals overrun| ELit[LiteralsOverrun]
    Exe -->|match_end + 32 <= out.len| Wild[offset check; wildcopy literals; wildcopy match]
    Exe -->|"else, GROW = false"| Exact[execute_exactly, cold: BlockSize / Offset / OutputFull; exact copies]
    Exe -->|"else, GROW = true"| Cold[leave the loop with the sequence: append, grow or execute_exactly, then resume]
    Wild --> Next
    Exact --> Next
    Cold --> Next
    Next -->|no| End{bitstream exhausted?}
    End -->|no| ESeq[SequenceStream]
    End -->|yes| Last[copy remaining literals; save repeat offsets]

• SeqTable rows are ZSTD_seqSymbol: baseline, next-state base, extra-bit and state-bit counts. Both counts are a BitCount, a 32-variant #[repr(u8)] enum: its range reaches the compiler with every load, so reads by these counts need no & 63 before the bzhi or shift. Constructing a count uses an exhaustive match, which compiles to range masking without the memory lookup that indexing an array of enum variants required. The sequence loop still holds references to rows and reads fields as needed.
• Basic sequence modes borrow immutable LL, OF and ML decoding rows built at compile time from the predefined distributions. A TableView carries the rows and initial-state width, whether the rows are static, owned by the decoder or borrowed from a dictionary. Selecting a basic mode needs no allocation or table construction, including the first block of a cold decoder. RLE and described modes allocate private storage on first use and reuse it thereafter. A frame start forgets which table repeat mode may use; selecting a basic mode again simply restores its static view. Private allocations remain reusable when a basic mode replaces them.
• SeqTable::fill_rows consumes static per-alphabet symbol rows containing the baselines and extra-bit counts. It no longer zeroes and rebuilds a 64-row symbol workspace for every described table. spread still ports ZSTD_buildFSETable_body: without “less than one” symbols it lays the symbols down eight at a time and scatters them with a fixed step, unrolled by two with no per-cell tests; otherwise it keeps the skipping loop. Tests compare every predefined row with the runtime builder and compare custom tables across the alphabets’ symbols and accuracy logs. Header parsing passes the block’s existing SIMD token through read_header_on to fill_rows_on; table construction therefore uses the selected target features without detecting the CPU per table or row. Dictionary parsing outside a block selects its backend once per table. read_header_on is #[inline(never)]. Being generic over the token, it was otherwise inlined into the block decoder that also holds the sequence loop, and with AVX-512 in the baseline the loop’s code then took 9% more instructions per decode (1645.0M against 1498.0M on silesia.tar at --fast=1) and 8-12% more time at every level. Portable builds were unaffected.
• Executor<const GROW: bool>: fixed outputs (decompress_to_slice, streaming History) compile to the loop above with execute_exactly inline and cold, and so does UninitOutput, which zeroes a block’s room before the block. Growing outputs (VecOutput) leave the loop with any sequence the wildcopies cannot take (no room for the overlength, a match before the output or past the window); the cold part of SequenceTables::execute appends it when it is at least APPEND_MIN (1024) bytes and valid, else grows the zeroed room and retries it, else runs execute_exactly (dictionary, end of output, errors in the old order: block size, offset, output full). Then the loop resumes.
• An offset with more than one extra bit is read with BitContainer::read (sub, shrx, bzhi), not look_nonzero, whose two shifts each need a negated count: four instructions fewer per such sequence.
• Decoding refills once per sequence and once more when the sequence’s extra bits exceed 64 - 7 - (9 + 9 + 8), as the reference does. Literal and match lengths with zero extra bits use their baselines directly; they do not perform a zero-width shift and mask.
• Errors keep the old order: literal overrun, block size, offset (and output room for caller-sized buffers).

Copies

copy.rs holds the wildcopies (ZSTD_wildcopy, ZSTD_overlapCopy8):

Fast copies write up to WILDCOPY_OVERLENGTH (32) bytes past a sequence and read as far past its source; the executor uses them only when the output has that room, and literals always carry 32 readable bytes after their end (Literals): raw literals are borrowed in place when the block holds 32 more bytes after them, and copied otherwise.

Index masks

Release-mode usize additions may wrap, so out[i..i + 16] normally costs an overflow test and a bounds test. The sequence executor cuts the output to start at max(history_start, op - window) — no valid match reaches further back — so every index is below 2^31 + 2^19 (compile-time asserted against DecodeLimits::MAX_WINDOW_LOG), and copy::idx masks operands to 32 bits. The masked sums provably do not wrap, which lets the compiler relate the executor’s room tests to the copies’ bounds checks and drop many of them. The masks never change a valid index.

Dictionaries

A dictionary adds nothing to the per-sequence fast path. Its content is not copied into the output: Executor carries it as a slice that only the cold execute_exactly reads. The fast path’s test is the one it always had, offset > lit_end || offset > window; a sequence failing it, which without a dictionary is always an error, now calls execute_exactly, which reports the error or, with a dictionary, copies the match’s head from the dictionary’s tail and the rest from the output. The dictionary’s tables are picked once per block (TableSlot::get), and its repeat offsets once per frame. The index bounds above are unchanged: the cut output still starts at the history’s first byte or a window before the block.

Measured on a loaded i7-13700KF (AVX2, pinned), cargo bench --bench decompress -- 'decompress-(dictionary|small)': a thousand JSON-like records of about 120 bytes, each its own level-3 frame with a 16 KiB dictionary trained on similar records, decode in 198 µs against zstd 1.5.7’s 135 µs (1.47x), and the same records compressed without a dictionary in 1.60 ms against 1.11 ms (1.44x). The dictionary adds no gap of its own beyond the decoder’s per-frame overhead on tiny frames. Frames without a dictionary decode as before: on a quiet host, text level 1 2.12 → 2.06 ms, text level 3 1.85 → 1.80 ms, binary level 3 3.49 → 3.43 ms, random 19.7 → 19.1 µs, streaming mixed 2.46 → 2.44 ms (Criterion medians, before and after the change).

Output paths

flowchart LR
    ToSlice[decompress_to_slice] --> Direct{decode_to_slice: every frame declares a size that fits?}
    Direct -->|yes| Slice[SliceOutput: frame cut at its declared end]
    Direct -->|no, or any error| Session[incremental session over History]
    ToUninit[decompress_to_uninit] --> DirectU{decode_to_uninit: same test}
    DirectU -->|yes| Uninit[UninitOutput: BorrowedBuf zeroed a block's room before each compressed block, cut at the declared end]
    DirectU -->|no, or any error| SessionU[process_uninit over History]
    Into[decompress / decompress_into] --> Cut[decode_all: VecOutput, declared frames cut at their end]
    Cut -->|any error| Uncut[reset, rerun with VecOutput uncut: defines the error]
    Process[DecoderSession::process] --> History[History: prepare / commit]

decompress and decompress_into write through VecOutput, which owns three decisions per frame and block:

flowchart TD
    Begin[begin_frame: content size, unread input] --> Declared{declared size?}
    Declared -->|yes| Reserve["reserve min(declared end, filled + 32 x unread input)"]
    Declared -->|no| Grow
    Reserve --> Block
    Grow[no reservation] --> Block
    Block[next block] --> Kind{raw or RLE?}
    Kind -->|yes| Place["place: extend_from_slice / resize with the byte, one pass"]
    Kind -->|compressed| Slack["reserve capacity only; BlockOutput: grow 4 KiB of zeros on demand, append long sequences and trailing literals"]
    Place --> Account[FrameDecoder::account_block: checksum, size checks]
    Slack --> Decode[FrameDecoder::decode_block] --> Account

• Stream::walk is the one-shot frame grammar over an Output. SliceOutput cuts the caller’s slice at each frame’s declared end, so chunked copies never touch bytes past the output and the rest of dst stays unchanged.
• VecOutput cuts a declared frame’s blocks at its end, like SliceOutput, so no slack past the output is zeroed or kept. A frame that overruns its declared size then fails with OutputTooSmall instead of the reference’s ContentSizeMismatch, so decode_all resets the stream and reruns the input uncut on any error; the uncut pass defines every error and leaves the vector as it was.
• Capacity grows by doubling (grow_output/reserve_output), never past twice the requested length. When the frame’s last request is within that factor, capacity jumps straight to it, so a declared frame ends with no reallocation and no spare capacity. At a frame start, a declared size is reserved only up to 32 output bytes per unread input byte (PLAUSIBLE_RATIO): an allocation always needs input in proportion, and a frame that claims more grows with its actual output.
• Raw and RLE blocks are appended in one pass (Output::place): extend_from_slice or resize with the run byte write straight into spare capacity. Compressed blocks write through BlockOutput (block.rs): the buffer’s initialized prefix, which VecOutput extends on demand (grow, 4 KiB of zeros at a time, up to the block’s room cut at the frame’s end) and by append (cut at the output’s end, then extend_from_slice for the literals and doubling extend_from_within for the match). Sequences of 1 KiB or more, a block’s trailing literals and literal-only blocks are therefore written once; shorter sequences still need zeroed room for the wildcopies’ 32 bytes of overlength, since safe Rust cannot hand the decoder uninitialized memory. Caller slices, UninitOutput and History are fixed (grows() == false).

flowchart TD
    Loop["Executor::<true>::run: decode + wildcopy"] -->|all done| Rest[room.put remaining literals]
    Loop -->|sequence does not fit, or leaves the window or output| Cold{valid, inside window and block?}
    Cold -->|"yes, >= APPEND_MIN"| Append["room.append: extend_from_slice + extend_from_within"]
    Cold -->|yes, short| Grow["room.grow: zero 4 KiB more"] -->|grew| Retry[retry the sequence] --> Loop
    Cold -->|no, or cannot grow| Exact["grow to the sequence end; execute_exactly"]
    Append --> Loop
    Exact --> Loop

• UninitOutput is SliceOutput over a BorrowedBuf: reserve zeroes a compressed block’s whole room (cut at the frame’s end) before the block, which then runs the fixed sequence loop, and place writes raw and RLE blocks once, over the zeroed bytes and then appended. See uninitialized output.
• decompress_to_slice tries the direct path and, when a frame has no content size, does not fit, or anything fails, reruns the whole input through the incremental session, which defines the method’s results and errors.
• History keeps an initialized buffer with a fill mark; prepare compacts as before and ensures room for a block plus the overlength, commit records the block’s end.
• The literal buffer only grows, so blocks of varying sizes zero nothing but its new tail. A four-stream section of at least 4 KiB grows it to FIXED_OUT bytes once, for the fixed-size pair loop.

Write prefetch of the next block

sequenceDiagram
    participant S as Stream::walk
    participant O as Output (Uninit / Slice / Vec)
    participant B as BlockDecoder::decode_compressed
    participant H as Huffman fast loop
    S->>O: reserve(block_max)
    O->>O: zero this block's room (UninitOutput only)
    O->>O: ahead = next_room(room end, block_max, limit)
    S->>B: decode(out)
    B->>O: out.ahead()
    B->>H: decode_streams(..., ahead)
    loop every round
        H->>H: lookups and refills
        H->>H: ahead.step(): prefetchw two lines, advance 128 bytes
    end
    B->>B: sequences (unchanged)
    S->>O: commit; next reserve zeroes lines already cached

A caller’s buffer is usually not cached when the decoder reaches it. The reference decodes a large block’s literals into the end of the block’s own output (ZSTD_in_dst), so its literal loop takes those misses while it computes. Here literals go to the decoder’s own buffer, and an uninitialized output is zeroed a block’s room at a time before the block (UninitOutput::reserve), a memset whose misses nothing overlaps.

Output::reserve therefore also records the block_max bytes after the current block’s room (next_room, cut at the frame’s end for caller slices, at the capacity for a vector) as a WriteAhead (decompressor::core::ahead), which BlockOutput::ahead hands to the block’s literal decoding. single_rounds, pair_rounds_fixed and pair_rounds_serial call WriteAhead::step once per round: two prefetchw hints (branches::prefetch_write_data, safe inline assembly; nothing on targets without one, and nothing under Miri) and an advance of 128 bytes, until the range is covered. The loops are bound by their lookup chains and leave memory bandwidth idle, so the misses overlap decoding; the next block’s zeroing (or its first writes into a caller’s slice or a vector’s spare capacity) then finds the lines cached. A prefetch never faults and changes no result, so the range needs no bounds proof; history outputs, dictionaries and the cold paths pass WriteAhead::NONE.

Measured on silesia.tar through mzstd_decompress in lzbench master a6fa8aa (one pinned vCPU of a Ryzen 9 7950X VDS, native build, three interleaved rounds, ratio to zstd 1.5.7 in the same run; the host’s noise is about ±3%): without prefetch 0.876-0.900 at level 1 and 0.887-0.926 at level 8; one line per round 0.913-0.938 and 0.923-0.927; two lines 0.903-0.941, 0.945-0.975 and 0.913-1.016 at level 19. Four lines measured no better than two. The hint costs the literal loops about 1% on x-ray, whose literals leave little else to overlap.

Measurements

cargo bench --bench decompress -- --warm-up-time 1 --measurement-time 3, pinned to one core of an i7-13700KF (AVX2 backend). Frames are written by zstd 1.5.7 from zstd_ffi::corpus inputs and decoded with reused contexts, decompress_to_slice against ZSTD_decompressDCtx. Times are Criterion medians; “before” is the scalar port this replaced.

Both decoders run near five instructions per cycle on these inputs, so instruction counts (callgrind) track time closely. When these were measured, the remaining gap was in the sequence loop (about 140 instructions per sequence against the reference’s 125, mostly bounds checks and bit-count masks) and the pair Huffman loop; the loop has since reached the reference’s count (see the known gaps).

mzstd_decompress over silesia.tar (211,957,760 bytes, streams from the zstd 1.5.7 CLI with -T1), best of six one-shot calls into a reused destination, one pinned core of a Ryzen 9 7950X, cdylibs with fat LTO and one codegen unit, MB/s averaged over two interleaved rounds (2026-11-03). “native” is -C target-cpu=native, as lzbench builds; “portable” is the default target. “before” is v0.2.9; “after” adds pair_rounds_serial and zeroing each block’s room at once (uninitialized output):

Keeping read_header_on out of line (the “fix” columns; same host, three interleaved rounds, best of 18 decodes per value) then made the native build faster than the portable one at every level:

Portable builds never run the serial loop (their block loops use AVX2), so their gain is the zeroing alone. With the zeroing held fixed, the serial loop takes level -1 from 1372 to 1622 MB/s in the native build and leaves the other levels within 1.5%.

XXH64 checksums are not vectorized: each of its four lanes is a serial multiply-rotate-multiply chain, and AVX2 has no 64-bit vector multiply. A u64x4 kernel measured 11.8 GB/s against 20.9 GB/s for the scalar code, which already matches the reference’s 21 GB/s.

Known gaps

• The block decoder inlines the whole sequence loop, and its code moves with whatever else is inlined into it: v0.2.9’s generic read_header_on cost 8-12% in AVX-512 builds alone. Check changes to block, literals and sequences with -C target-cpu=native as well as a portable build, by instruction counts per decode as well as time.

• On a Zen 4 EPYC 9554, lzbench measured v0.2.10 8% slower than 0.2.5 at the --fast levels and within 3% at levels 1-19 (inikep/lzbench#363); the same EPYC already measured 0.2.6’s per-block zeroing 1-3% slower than 0.2.5’s whole-destination zeroing, the reverse of the Ryzen 9 7950X. The out-of-line read_header_on is not yet measured there.

• Literal-heavy inputs decode slowest against the reference: Silesia x-ray at level 1 0.85x (Silesia run of 2026-10-30, Ryzen 9 7950X, reused contexts; since replaced by lzbench), where both decoders spend 77-79% of their time in the four-stream pair loop. pair_rounds_fixed takes 72.6 cycles per round against the reference’s 61.5 (242 instructions against 221): the compiler keeps two streams’ containers on the stack between rounds, on their dependency chains (22.6M store-to-load interlocks against 6.9M for 30 decodes). A branch-checked refill and other stream orders measured slower. In AVX-512 builds the three-plus-one pair_rounds_serial still executes about 13 instructions per pair lookup against the asm loop’s 9.75 that lzbench’s zstd links (huf_decompress_amd64.S): 12.4M instructions and 3.25M cycles per decode of 1 MiB of x-ray against 9.2M and 2.55M (2026-11-08). The remaining instructions are the stack traffic of twenty live values in fifteen registers and a mov per lookup that the asm avoids with three narrow loads from each row.

• Measured and rejected (2026-11-08, same input): the four streams’ containers in one fearless_simd u64x4, with the lookup index, the consumed bits (shlv_u64x4) and the rounds in vector form, and the row loads, stores and refills per lane. It executed the fewest instructions (8.0M against 12.6M) but took 7.36M cycles against 3.32M at 1.1 instructions per cycle: LLVM turned the per-lane row loads into vpgatherqd, whose latency sits on every stream’s lookup chain. A gather cannot be written explicitly either: _mm256_i64gather_epi32 takes a raw pointer and stays unsafe inside fearless_simd::kernel!, whose other intrinsics (_mm256_sllv_epi64 and the like) are the operations the portable types already give.

• Measured and rejected (2026-11-09, C-written frames at level 1): the four-stream sections of four consecutive blocks decoded together, every lookup step taking one pair from each of the sixteen streams, which a decoder could do without changing the format. Even in an SSE4.2 build, where nothing can become a gather, sixteen chains decoded at 0.69x the speed of one section at a time on x-ray and 0.73-0.76x on mozilla and dickens (0.60-0.82x with AVX2 or AVX-512 features, whose builds also vectorize the step into gathers). Sixteen containers, positions and output positions do not fit in fifteen registers: every chain step then loads and stores its state, and the store-to-load latency joins the chain. More chains do not help a loop that already overlaps three or four.

• Measured and rejected (2026-11-08, same input): the AVX-512 pair loop over one shared buffer, the output at its start and the streams copied after it (FIXED_OUT + BLOCK_SIZE_MAX + 8 bytes), with refills masked into the copy instead of clamped to a limit. One base and no limit freed registers and cut the loop to 11.85M instructions from 12.56M, but not its cycles (3.23M against 3.23M, eight rounds): the three-plus-one order is bound by the fourth stream’s lookup chain, which runs alone. All four streams interleaved, as the reference’s asm does, vectorized again into gathers (6.4M cycles) even with one or two streams indexing by rotate_left(11) & 0x7ff, which LLVM folds back into the same shift.

• Large outputs: with the write prefetch, silesia.tar decodes at 0.90-0.97 of zstd 1.5.7’s speed at levels 1 and 8 in lzbench, about even at level 19. The rest of the gap is the uninitialized output’s zeroing, which safe code cannot skip, and the part of each block’s room the literal loops do not reach in time: blocks with few or raw literals prefetch little.

• Measured and rejected (2026-11-08, lzbench on silesia.tar, ratio to zstd 1.5.7 in the same run, medians of three): writing the uninitialized output in place behind a checked initialized front, as a safe WriteFront type would (each chunk write tests that it starts at or before the front and moves the front; reads test that they end below it), instead of zeroing each block’s room. Simulated in the copy kernels, the checks added 22-27% instructions per decode (dickens 2 MiB level 1: 25.1M to 30.7M) and decoded at 0.866-0.876 of zstd’s speed at levels 1, 8 and 15, against 0.932-0.949 with zeroing and the write prefetch. Even with no zeroing and no checks at all (an unsafe cast, for the measurement only) the decoder reached 0.960-0.984: the zeroing left costs about 3%.

• Measured and rejected (2026-10-29, Ryzen 9 7950X, Alice level 1): the repeat offsets as scalars, a packed u64 pair, permuted slots, a ring of four slots, the states kept local to the loop, a clamped refill position, and the window test left out where the window covers the output. Each changed the sequence loop’s register allocation for the worse: 2-12 more instructions per sequence and 2-14% slower, although the ring cut store-to-load interlocks from 196M to 109M (the reference: 136M) for 3000 decodes. Shifting [u32; 3] in memory reloads two offsets across two stores of different widths, about two interlocks per sequence, but at about five instructions per cycle the loop is bound by its instruction count, which the interlocks barely change.

• Measured and rejected for decompress (2026-10-14): appending every sequence to the vector with extend_from_slice/extend_from_within is slower (Alice level 1: 110 µs against 96 µs) because every chunk reloads and stores the vector’s length through memory, which serializes the copies; appending only sequences of 1 KiB or more is what VecOutput does now. Decoding a batch of sequences before executing it, in two out-of-line loops, executes about 15% more instructions; one allocation for the three sequence tables changes nothing.

• Measured and rejected (2026-10-15, silesia.tar levels 1, 3 and 19): copying matches with offsets of 32 or more in 16-byte chunks, as ZSTD_wildcopy does, instead of 32-byte ones (no faster; the hot store after the match load is a cache miss on the match source, which the reference pays too).

• With the default 16 codegen units, a size change in huffman.rs or fse.rs moves the unchanged SequenceTables::execute by about ±1.6% of its instructions (26.45M against 26.87M on Alice level 3, identical source), and its time by about ±2%. The table builds rejected on 2026-10-15 for a 2.5-4% slowdown were measured through that noise; with execute controlled for they are faster and are now in place. Check loop changes by execute’s instructions per sequence or a codegen-units = 1 build, not by time alone.

• Measured and rejected (2026-10-18, i7-13700KF): per-stream staging buffers in the pair loop (fewer instructions per round, more mispredictions and more Huffman instructions overall); a run-based pair table build (no fewer instructions, more mispredictions); zeroing growing output in 2-16 KiB chunks without appends (±0.5%), or with prefetchw 16 KiB ahead of the zeroed front (Alice 2% slower); 64 KiB zeroing chunks (14-23% slower on the 64 KiB binary input, which they zero whole); the cold path for growing outputs as a separate #[cold] function (1% fewer instructions, 6-8% slower on Alice from layout). Sequence-loop rewrites each added instructions per sequence on Alice level 1: one bounds test per refill instead of two (+1.0), literals as an advancing slice (+5.0, +7.0 with a merged offset/window test), current rows held by value (+24.9) or by reference (+11.0), the decoder copied into run (+12.1), the fixed-output loop out of the function (+3.0); repeat offsets as [u64; 3] (+2.9% in total, no faster) or packed into a u64 and a u32 (+12.4%, slower). Almost any change to this function moves its register allocation by 1 to 25 instructions per sequence. Executing matches that start in a dictionary inline (the extDict branch of ZSTD_execSequence) made dictionary decoding 9% faster but large decodes without a dictionary 1.5-3% slower through the same code-generation shift; it is not kept.

• The one-shot sequence loop executes 126.7 instructions per sequence on Alice level 1 (AVX2 code, Ryzen 9 7950X, 2026-10-29) against the reference’s 118.8; both spill the decoder state to the stack on every sequence. It is the largest remaining gap: 74-89% of decoding time at every level, against the reference’s 77-90%. Literal-heavy text (Alice) still decodes a few percent slower than the reference through decompress: the Huffman pair loop originally executed about 13.2 instructions per lookup against the reference’s 11.25 (spills: four streams’ bits, positions and outputs plus their bases do not fit in 15 registers), before the interleaved three-stream schedule above and the fixed-size loop, which keeps two of its four streams’ containers in registers across a round and stores the other two between rounds. The single-symbol loop used 8.8 per symbol against 8.2, and about 2% of the time zeroes room for short sequences (VecOutput::grow). Both loops are limited by each stream’s chain of dependent lookups, as the reference’s are: removing 9-12% of their instructions made no measurable difference in a codegen-units = 1 build. Raw-literal frames (Alice level -5) decode at parity.

• SeqTable::fill_rows still runs outside the SIMD dispatch, without BMI or LZCNT in a portable build. Its alphabet rows are now static and constructing a BitCount no longer needs a table lookup. Building the pair table costs about 6K instructions per tree beyond its 8 KiB zeroed allocation.

• decompress_to_slice on 4 MiB of text is about 11% slower than ZSTD_decompressDCtx (level 1: 2.18 ms against 1.85 ms; level 3: 1.89 ms against 1.65 ms) while executing only about 4% more instructions (111.9M against 107.3M), so it retires fewer instructions per cycle; finding the stall needs a cycle-level profiler, which this WSL2 host lacks. One suspect is the repeat-offset shift, which compiles to an 8-byte reload spanning two earlier stores that cannot be forwarded; u64 repeat offsets removed that pattern without measurable gain.

• decompress_to_slice takes the incremental path, with its extra copies, for frames without a content size.

• No prefetching variant for long offsets (ZSTD_decompressSequencesLong) and no split literal buffer.