Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Shared Brotli (RFC 9841)

What this crate implements of RFC 9841 today, and where the boundaries of that implementation are. RFC 9841 updates RFC 7932 with three separable features:

FeatureState
Large Window Brotliimplemented for qualities 3 to 11
LZ77 prefix dictionaries: preparation, indexes, addressing, searchimplemented
LZ77 prefix dictionaries: use by an encoderimplemented for qualities 5 to 11 — refused below, not ignored
Serialized shared dictionaries and custom static encodingimplemented behind experimental; see rfc9841-encoding.md
Framing container formatexperimental writer and decoder

This file covers Large Window and prefix mechanics. The linked specifications cover the experimental extensions; remaining limitations are recorded below.

1. Module boundaries

mbrotli::dictionary is the public home of RFC 9841’s prefix dictionaries: the immutable PreparedDictionary, the builder that indexes one, the limits it is built under, and the error preparation reports. Large Window Brotli is not there, because it is a configuration value rather than a dictionary — it is one of the two headers a Window can carry — so it lives in EncoderConfig alongside the quality and the mode.

mbrotli::compressor::shared is what is left: a private module holding the low-level error the encoders raise, which the public DictionaryError, ConfigError and EncodeError are built from.

Below the API, compressor::core::rfc9841 owns the wire primitives. It is distinct from shared, which predates it and means “code more than one quality needs”.

graph TD
    subgraph public["Public API"]
        comp["mbrotli::EncoderConfig<br/>Window, WindowEncoding, ConfigError"]
        entry["Compressor<br/>compress_with_dictionary,<br/>compress_with_dictionary_into,<br/>compress_with_dictionary_to_slice,<br/>start_with_dictionary,<br/>writer_with_dictionary, reader_with_dictionary"]
        sharedmod["mbrotli::dictionary"]
        ctx["PreparedDictionary, DictionaryBuilder,<br/>DictionaryLimits"]
        err["DictionaryError, EncodeError<br/>::DictionaryUnsupportedForQuality"]
    end

    subgraph private["Private implementation"]
        sharederr["compressor::shared<br/>SharedBrotliError"]
        rfc["core::rfc9841"]
        window["core::rfc9841::window<br/>ResolvedWindow"]
        inner["core::rfc9841::context<br/>SharedContextInner, Budget,<br/>SharedDictionaryData,<br/>PreparedDictionaryIndexes"]
        pfx["core::rfc9841::prefix<br/>PrefixSources<br/>(addressing, match scan)"]
        prep["core::rfc9841::prepared<br/>PreparedPrefix, Candidates"]
        search["core::rfc9841::search<br/>find_match, find_all_matches<br/>(the match finders' view)"]
    driver["core::driver<br/>check_large_window, check_shared,<br/>quality routing"]
    gsearch["core::greedy::backward_references<br/>(q5 to q9)"]
    hsearch["core::hq::zopfli<br/>(q10, q11)"]
    cmd["shared::command<br/>extend_last_command"]
        dist["shared::distance<br/>DistanceParams, distance_code_limit"]
        mlen["prefix::common_prefix_len<br/>(scalar word scan)"]
        gparams["core::greedy::params::GreedyParams"]
        hparams["core::hq::params::HqParams"]
        fast["core::fast::FastEncoder"]
        gmeta["core::hq::metablock<br/>choose_distance_params"]
        bitstream["shared::bitstream<br/>meta-block writer"]
        ring["shared::ringbuffer::RingBuffer"]
    end

    comp --> entry
    entry --> sharedmod
    sharedmod --> ctx
    sharedmod --> err
    ctx --> inner
    ctx --> sharederr
    entry --> driver
    driver --> window
    driver --> fast
    driver --> inner
    rfc --> window
    rfc --> inner
    inner --> pfx
    inner --> prep
    inner --> search
    search --> pfx
    search --> prep
    search --> gsearch
    search --> hsearch
    search --> cmd
    pfx --> mlen
    window --> gparams
    window --> hparams
    window --> fast
    gparams --> dist
    hparams --> dist
    hparams --> gmeta
    gmeta --> dist
    dist --> bitstream
    gparams --> ring
    hparams --> ring
    fast -->|refuses| err
    driver -->|refuses a non-empty context| err

    classDef privateNode fill:#f6e8c3,stroke:#8a6d3b;
    class sharederr,rfc,window,inner,pfx,prep,search,driver,dist,mlen,gparams,hparams,fast,gmeta,bitstream,ring,gsearch,hsearch,cmd privateNode;

2. Selecting a large window

Large Window mode is reached one way only: the constructor a caller names. It is never inferred from the size, the input, the quality, the target, or anything else.

let config = EncoderConfig::default()
    .with_quality(Quality::Q5)
    .with_window(Window::large(30)?);

Window carries both the size and the header encoding. Its constructors validate their ranges: standard accepts 10..=24, and large accepts 10..=62. Window::large(22) and Window::standard(22) select different headers and distance alphabets at the same size.

There is no separate large_window flag that could disagree with the size, and nothing downstream re-checks a range, because no Window can exist that no header can express.

classDiagram
    class EncoderConfig {
        -Quality quality
        -Window window
        +new() EncoderConfig
        +with_window(Window) EncoderConfig
        +window() Window
    }
    class Window {
        -u8 bits
        -WindowEncoding encoding
        +MIN_BITS = 10
        +MAX_STANDARD_BITS = 24
        +MAX_LARGE_BITS = 62
        +DEFAULT = standard(22)
        +standard(u8) Result
        +large(u8) Result
        +bits() u8
        +encoding() WindowEncoding
    }
    class WindowKind {
        <<private enum, internal::WindowBits>>
        Standard(u8)
        Large(u8)
    }
    class ResolvedWindow {
        -usize declared_bits
        -bool large
        +new(&CompressParams) ResolvedWindow
        +at_least(usize) ResolvedWindow
        +is_large() bool
        +encoder_bits() usize
        +header() (u16, u32)
    }
    EncoderConfig *-- Window
    Window ..> WindowKind : lowers into
    ResolvedWindow ..> WindowKind : resolved from

3. Declared window versus retained history

ResolvedWindow separates two numbers that an ordinary stream never has to distinguish:

  • declared bits — what the header says, 10..=62;
  • encoder bits — what the encoder actually keeps history for, capped at MAX_ENCODER_WINDOW_BITS (30).

Everything that costs memory or bounds a distance is derived from the encoder bits: the ring buffer size, the block size, the match finders’ reach, and the largest backward distance. Only header() reads the declared bits.

flowchart TD
    p["EncoderConfig::window()"] --> q{"encoding == Large?"}
    q -->|no| std["ResolvedWindow<br/>declared = bits (10..=24)<br/>large = false"]
    q -->|yes| lw["ResolvedWindow<br/>declared = bits (10..=62)<br/>large = true"]
    std --> hdr{"header()"}
    lw --> hdr
    hdr -->|"large"| h14["14 bits:<br/>0b00010001 then 6 window bits"]
    hdr -->|"lgwin == 16"| h1["1 bit: 0"]
    hdr -->|"lgwin == 17"| h7a["7 bits"]
    hdr -->|"lgwin > 17"| h4["4 bits"]
    hdr -->|"lgwin < 16"| h7b["7 bits"]
    std --> enc["encoder_bits() = declared"]
    lw --> encl["encoder_bits() = min(declared, 30)"]
    enc --> use["ring buffer, block size,<br/>match reach, max backward distance"]
    encl --> use

This is what makes a 62-bit declaration free: it allocates nothing, and the emitted payload is byte-identical to the same stream declared at 30 bits.

4. The distance alphabet

Large Window is not only a header. The distance alphabet written to the stream is sized for 62 distance bits instead of 24, while the symbols that may actually occur stop at MAX_ALLOWED_DISTANCE ((1 << 31) - 4). alphabet_size_max and alphabet_size_limit, which coincide for every RFC 7932 stream, genuinely differ here — and the meta-block writer already kept them apart, which is why that layer needed no change.

distance_code_limit ports BrotliCalculateDistanceCodeLimit: it finds the largest alphabet that cannot express a distance past the limit, cut on a complete interleaved group so neither side ever sees a half-represented group. Its widest result over every legal (NPOSTFIX, NDIRECT) pair is exactly 544, the size of a distance histogram — which is a unit test, not a comment.

sequenceDiagram
    participant P as GreedyParams / HqParams
    participant W as ResolvedWindow
    participant D as DistanceParams
    participant M as hq::metablock
    participant B as bitstream

    P->>W: new(&params)
    W-->>P: { declared, large }
    P->>D: for_window(large, npostfix, ndirect)
    alt large
        D->>D: alphabet_size_max = 16 + ndirect + (62 << (npostfix+1))
        D->>D: distance_code_limit(0x7FFFFFFC, npostfix, ndirect)
        D-->>P: limit < max
    else ordinary
        D->>D: alphabet_size_max = 16 + ndirect + (24 << (npostfix+1))
        D-->>P: limit == max
    end
    Note over M: quality 10 and 11 only
    M->>D: for_window(large, candidate npostfix, ndirect)
    M-->>M: keeps the cheapest candidate
    P->>B: dist
    B->>B: num_distance_symbols = alphabet_size_max
    B->>B: num_effective_distance_symbols = alphabet_size_limit

The per-meta-block retune at qualities 10 and 11 is the subtle part: it walks candidate (NPOSTFIX, NDIRECT) pairs and replaces the block’s alphabet with the cheapest. Building a candidate with the RFC 7932 constructor there would emit a 64-symbol alphabet under a header that promised 140, and every stream past the first few bytes would fail to decode. for_window carries the flag so it cannot.

5. The prepared dictionary

A PreparedDictionary is the caller’s. It owns the dictionary bytes, it is built by a builder that consumes them, and it is handed to a compression call by shared borrow. There is no Arc, no Rc, no Mutex, no RwLock, no atomic, no global registry and no interior mutability anywhere in it or below it — the type is Send and Sync because its fields are, and because nothing in it is mutable, any number of compressors may borrow one at the same time, on any number of threads, with no synchronisation of this crate’s making. A caller who wants shared ownership wraps it in an Arc, and that is their policy.

Building one takes no quality. The indexes a dictionary carries are the same whichever quality later reads them, so one prepared dictionary serves every quality that can consult one.

classDiagram
    class DictionaryBuilder {
        <<public, consuming>>
        -DictionaryLimits limits
        -Vec~Box~u8~~ attachments
        +new() Self
        +add_prefix(B: Into~Box~u8~~) Self
        +with_limits(DictionaryLimits) Self
        +build() Result~PreparedDictionary, DictionaryError~
    }
    class PreparedDictionary {
        <<public, immutable, Send + Sync>>
        +attachment_count() usize
        +source_bytes() usize
        +retained_bytes() usize
        +backward_distance(u64, u64) Option~u64~
        +prefix_offset(u64, u64) Option~u64~
        +longest_match(&u8) Option~PrefixMatch~
    }
    class SharedContextInner {
        <<private>>
        -SharedDictionaryData dictionaries
        -PreparedDictionaryIndexes prepared
        +longest_prefix_match(S, &u8) Option~PrefixMatch~
    }
    class SharedDictionaryData {
        <<private, immutable>>
        -PrefixSources prefix
    }
    class PrefixSources {
        <<private, immutable>>
        -Box~Box~u8~~ segments
        -Box~u64~ starts
        +locate(u64) Option
        +address_of(u64, u64) Option~u64~
        +distance_of(u64, u64) Option~u64~
        +match_length(S, u64, history, target, limit) usize
    }
    class PreparedDictionaryIndexes {
        <<private, immutable>>
        -Box~PreparedPrefix~ prefixes
    }
    class PreparedPrefix {
        <<private, immutable>>
        -u32 bucket_bits
        -u32 slot_bits
        -Box~u32~ slot_offsets
        -Box~u16~ heads
        -Box~u32~ items
        +candidates(u64) Candidates
    }
    DictionaryBuilder ..> PreparedDictionary : build()
    PreparedDictionary *-- SharedContextInner
    SharedContextInner *-- SharedDictionaryData
    SharedContextInner *-- PreparedDictionaryIndexes
    SharedDictionaryData *-- PrefixSources
    PreparedDictionaryIndexes *-- PreparedPrefix

Every collection that has stopped growing is a Box<[_]>, not a Vec<_>: the capacity word is dead weight after preparation, and dropping it removes push and truncate from the type, so “immutable after preparation” is a property of the type rather than a comment. The builder’s attachment list is the one collection that grows, and it is a Vec.

5.1. Preparation is a transaction

build is all-or-nothing, and every check runs before the first table is allocated — including the allocation check, which compares a computed upper bound rather than the finished size, so a dictionary that would not fit its limit is never built and thrown away. A dictionary with no bytes in it is refused outright rather than behaving like no dictionary at all: the two would be indistinguishable, which is exactly the confusion this crate avoids elsewhere by refusing a dictionary a quality cannot read.

flowchart TD
    b["DictionaryBuilder::build"] --> z{"any bytes at all?"}
    z -->|no| e0["Err(Empty)"]
    z -->|yes| c{"attachments &le; 15?"}
    c -->|no| e1["Err(TooManyAttachments)"]
    c -->|yes| d{"each segment &le; 2^31 - 1?"}
    d -->|no| e2["Err(TooLarge)"]
    d -->|yes| f{"total &le; max_prefix_bytes<br/>and max_source_bytes?"}
    f -->|no| e2
    f -->|yes| g{"peak estimate &le;<br/>max_retained_bytes?"}
    g -->|no| e3["Err(PreparationTooLarge)"]
    g -->|yes| h["build one PreparedPrefix per attachment"]
    h --> i["PrefixSources::new: cumulative offsets"]
    i --> j["PreparedDictionary"]

The estimate bounds the build’s peak, not the finished size: step 3 fills the slim tables while step 1’s and step 2’s scratch is still alive, so preparation costs roughly eight bytes per source byte at its high-water mark, and the peak is what a limit should refuse. It over-counts on purpose — the item table is bounded at one entry per hashable position, the most step 1 can chain, and every other table is counted exactly — so it also bounds the finished context. context::the_estimate_is_never_smaller_than_the_context_it_predicts is what keeps the two in step.

5.2. Virtual concatenation, and the distances that address it

The attachments are never copied into one buffer. PrefixSources gives them one logical address space with a cumulative offset table, and everything else works in that space. Attachment order is prefix order: the first attachment holds the oldest bytes, the last one the bytes immediately before the stream’s own output.

flowchart LR
    subgraph logical["Logical prefix: one address space"]
        direction LR
        a0["attachment 0<br/>oldest"] --> a1["attachment 1"] --> a2["attachment 2<br/>newest"]
    end
    logical --> stream["stream output<br/>(ordinary sliding window)"]
    d1["distance 1..=M"] -.->|ordinary history| stream
    d2["distance M+1"] -.->|last prefix byte| a2
    d3["distance M+D"] -.->|first prefix byte| a0

For a total prefix length D and an ordinary maximum backward distance M:

address_of(distance, M)  = D - (distance - M)   for M < distance <= M + D
distance_of(address, M)  = M + (D - address)    for 0 <= address < D

Both are checked u64 arithmetic and return None rather than wrapping, outside the prefix range in either direction. prefix::addressing_round_trips_through_the_distance walks every address of a three-attachment prefix through both directions, and dictionary::a_dictionary_reports_its_own_shape does the same through the public PreparedDictionary::backward_distance and PreparedDictionary::prefix_offset.

5.3. The prepared index

One index per attachment, a port of CreatePreparedDictionary. The shape is the reference’s because the shape is observable: the bucket count, the hash width and the per-bucket cap decide which candidates a search sees and in what order, and those candidates become the commands an encoder emits.

flowchart TD
    s["source bytes"] --> t{"len &lt; 8?"}
    t -->|yes| empty["empty index<br/>(no hashable position)"]
    t -->|no| shape["shape_for(len):<br/>bucket_bits 17..=22, slot_bits = bucket_bits - 10"]
    shape --> s1["step 1 — bloated hasher<br/>chain every position into its bucket,<br/>newest first, count capped at 32"]
    s1 --> s2["step 2 — slot limits<br/>shorten chains until every head offset<br/>in a slot fits sixteen bits"]
    s2 --> s3["step 3 — slim index<br/>slot_offsets, heads, items;<br/>top bit of the last item ends the chain"]
    s3 --> out["PreparedPrefix"]

prepared::the_index_is_identical_to_the_c_reference builds both and compares bucket_bits, slot_bits, slot_offsets, heads and items entry for entry over six corpora, including one large enough to trigger the shape scaling. The C side is reached through brotli-ffi/shim/static_dict_probe.c, which copies the reference’s three tables out of the single flat allocation it carves them from.

sequenceDiagram
    participant C as PreparedDictionary::longest_match
    participant I as SharedContextInner
    participant P as PreparedPrefix
    participant S as PrefixSources
    participant M as prefix::common_prefix_len

    C->>I: longest_prefix_match(input)
    I->>I: head = first 8 bytes, little-endian
    loop attachment 0..n, oldest first
        I->>P: candidates(head)
        P-->>I: source offsets, newest first, chain-capped
        loop each candidate
            I->>S: match_length(base + offset, history, input, limit)
            loop each segment run from that address
                S->>M: compare run against the unmatched tail
                M-->>S: bytes agreed
            end
            S-->>I: total, across segment seams and into history
            I->>I: keep only a strictly longer match
        end
    end
    I-->>C: Option<PrefixMatch { offset, length }>

This is the standalone diagnostic PreparedDictionary::longest_match exposes behind the diagnostics feature, not the search a match finder runs; that one is §5.5. It is feature-gated because the candidate order it breaks ties by is an implementation detail no application should depend on.

The scan uses scalar whole-word comparisons with a byte tail. It is separate from the encoders’ vector kernel in shared::match_len.

Being scalar, the answer cannot depend on the backend, so there is no identity test to run for it. The tie rule is the reference’s: strictly-longer wins, so of two equally long matches the one in the older attachment, and within an attachment the one at the newer position, is kept.

A match may begin in one attachment and run into the next, and on into the stream’s own history — the virtual concatenation RFC 9841 allows. The candidate that starts a match must still be indexed, so its own eight hashed bytes have to lie inside one attachment.

5.5. How a match finder consults a prefix

core::rfc9841::search is the attached context in the form a match finder needs it. It ports FindCompoundDictionaryMatch, LookupCompoundDictionaryMatch, FindAllCompoundDictionaryMatches and LookupAllCompoundDictionaryMatches.

The whole integration rests on one number, the reference’s gap: the total attached bytes. The concatenated prefix ends exactly where the stream begins, so a backward distance addresses the window while it is at most max_ring_buffer_distance, and the prefix beyond that. Written the reference’s way, logical address zero of attachment d sits at distance max_ring_buffer_distance + gap - chunk_start(d).

graph LR
    subgraph addr["one distance axis"]
        direction RL
        a0["attachment 0"] --- a1["attachment 1"] --- a2["attachment 2"] --- win["the stream's own window"] --- cur["current position"]
    end
    note["distance grows leftwards:<br/>0 at the position,<br/>max_ring_buffer_distance at the window edge,<br/>+ gap at the far end of attachment 0"]

Three things shift together wherever a prefix is attached, and all three collapse to their ordinary form when gap is zero:

SiteWithout a prefixWith one
Static-dictionary boundary handed to the matcherdictionary_startdictionary_start + gap
Prefix search boundary—dictionary_start
compute_distance_code, distance-cache guarddictionary_startdictionary_start + gap

find_match improves a SearchResult in place, attachment by attachment in attachment order, replacing the incumbent only on a strictly higher score — so a tie leaves the earlier finder’s match. It probes the four cached distances first, exactly as the reference does, then walks the bucket chain behind a four-byte pre-filter at best_len. find_all_matches is the high-quality sibling: no cached distances, every length improvement reported, at most sixty-four per position, with min_length carried forward between attachments.

A candidate is measured inside the attachment it was found in and stops at its end, even though the addressing spans every attachment. That is the reference’s behaviour, not an omission. The one place a copy does run across a seam is extend_last_command, which continues an already emitted command into the concatenation using PrefixSources::run_from.

Which qualities reach this at all is fixed by the reference: it compiles its compound-dictionary search only for H5, H6, H40, H41, H42, H55, H65 and the binary tree, so qualities five and above consult a prefix and qualities zero to four have nowhere to put a match. Where the reference then silently ignores the dictionary, this crate refuses.

5.6. What a dictionary compression call does

stateDiagram-v2
    [*] --> Build: Compressor&#58;&#58;new(config)
    Build --> LW: large window at quality 0, 1 or 2
    LW --> [*]: Err(ConfigError&#58;&#58;LargeWindowUnsupportedForQuality)
    Build --> Q: a compressor
    Q --> Low: quality below 5
    Low --> [*]: Err(EncodeError&#58;&#58;DictionaryUnsupportedForQuality)
    Q --> Attached: quality 5 to 11
    Attached --> Consulted: nonempty input consults the prefix
    Consulted --> [*]: a stream whose distances may address the prefix
    Attached --> EmptyInput: input empty
    EmptyInput --> [*]: shared finish preserves configured header

The order is fixed and every check runs before any input is consumed: the window against the quality when the compressor is built, then the quality against the dictionary when the operation starts.

Below quality five, dictionary operations return EncodeError::DictionaryUnsupportedForQuality before consuming input. The compressor remains usable for an ordinary call. DictionaryBuilder::build rejects an empty dictionary; an operation without a dictionary uses the ordinary encoding path.

An empty input follows the same encoder finish path in every API shape, preserving the configured stream header. Dictionary validation still applies, although an empty stream cannot reference dictionary bytes. Native C one-shot’s empty shortcut is deliberately omitted; see universal-encoding.md.

Every entry point that takes a dictionary — the three one-shot forms, the session, the writer and the reader — reaches the same bytes for the same declared size, which dictionary::every_dictionary_entry_point_reaches_the_same_bytes checks. A flush carries the dictionary too, so the bytes after one are still compressed against it.

5.7. Reuse determinism

Nothing a dictionary owns is stream state. There is no LZ77 history in it, no distance cache, no pending command, no meta-block state and no input position — only the caller’s bytes and indexes derived from them by a pure function. That is what makes a shared borrow sound, and it makes the reuse contract hold by construction rather than by a reset.

The compressor is the mutable half, and it does reset: dictionary::reusing_one_compressor_with_a_dictionary_is_deterministic runs a deliberate failure, an ordinary call and an abandoned session between two dictionary calls and requires the same bytes from both.

6. Where a large window is refused

stateDiagram-v2
    [*] --> Check: Compressor&#58;&#58;new / reconfigure
    Check --> Ordinary: the window is not large
    Check --> Q012: quality 0, 1 or 2
    Check --> Large: quality 3..=11
    Q012 --> [*]: Err(ConfigError&#58;&#58;LargeWindowUnsupportedForQuality)
    Ordinary --> Encode: any operation
    Large --> Encode: any operation
    Encode --> Empty: input empty
    Empty --> [*]: shared finish preserves configured header
    Encode --> [*]: stream

The check runs when the compressor is built, so a refused request never reaches an operation at all and cannot be silently dropped, even for empty input. The encoders keep their own copies of the refusal in FastEncoder::new and GreedyParams::new, which is unreachable from a validated configuration and is reported as EncodeError::InternalInvariant if it ever fires.

Qualities 0, 1 and 2 are refused rather than downgraded because all three may write distances through a code built for the 64-symbol RFC 7932 alphabet: the fast qualities always do, and quality 2 does whenever a meta-block carries at most a hundred and twenty-eight commands.

7. Error propagation

graph LR
    lw["large window at q0, q1, q2"] --> cfg["ConfigError::LargeWindowUnsupportedForQuality<br/>(Compressor::new / reconfigure)"]
    tm["TooManyPrefixDictionaries"] --> shared
    dl["DictionaryTooLarge"] --> shared
    ct["SharedContextTooLarge"] --> shared
    shared["SharedBrotliError<br/>(private)"] --> de["DictionaryError<br/>(DictionaryBuilder::build)"]
    q5["dictionary below quality 5"] --> ee["EncodeError::DictionaryUnsupportedForQuality<br/>(the operation)"]
    de --> caller["the caller"]
    cfg --> caller
    ee --> caller
    ee -->|"From&lt;EncodeError&gt;"| io["std::io::Error<br/>(the io adapters)"]

The split is by domain rather than by layer: a window the quality cannot carry is a configuration mistake and is reported when the compressor is built; a dictionary that cannot be prepared is a dictionary mistake and is reported when it is built; a dictionary the quality cannot read needs an operation to happen at all. SharedBrotliError is private and exists only to carry the low-level refusals up to whichever public error owns them.

8. Determinism and SIMD

The window touches no SIMD dispatch at all. ResolvedWindow is resolved once per session from CompressParams alone; no window decision depends on the instruction set, the machine, or the input. every_backend_produces_the_same_large_window_stream runs every backend the host supports at four declared windows across all nine supporting qualities and requires byte identity.

The shared context adds no dispatch point at all, and touches none of the existing ones. Preparation and the prefix search are both scalar, both pure functions of their inputs. Section 42.3 of the implementation specification lists the prefix hash loop and the prefix match length as kernels to profile before vectorising, and this repository’s rules require measurement before optimisation; Section 45 puts scalar parity first in any case.

The consequence is worth stating plainly: prepared indexes, candidate order and search results are identical on every machine, so a context prepared once may be used by a compressor that later resolves a different backend, and no test has to prove it.

9. Verification topology

LayerWhat it checks
core::rfc9841::window unit testsheader bits for all 53 large windows and every ordinary window; retained history never exceeds the declaration
shared::distance unit teststhe large alphabet against every legal (NPOSTFIX, NDIRECT) pair; the 544-symbol histogram ceiling; the degenerate branches of distance_code_limit
tests/large_window.rsheader golden bits; round trips through the pinned C decoder with BROTLI_DECODER_PARAM_LARGE_WINDOW for 10..=30; header-only equivalence for 31..=62; refusal at qualities 0, 1 and 2; empty and tiny inputs; the bound; streaming and one-shot agreement over sixteen chunk sizes; backend identity
core::rfc9841::prefix unit testsattachment ordering; addressing over empty attachments; the distance round trip; saturating arithmetic at u64::MAX; the match scan against a materialised oracle over every start, seam and limit; the word scan against a byte-by-byte comparison at every shared length and limit
core::rfc9841::prepared unit teststhe shape ladder; every hashable position indexed once; newest-first, capped bucket chains; entry-for-entry equality with CreatePreparedDictionary through the workspace shim, over six corpora including one that triggers shape scaling
core::rfc9841::context unit testsattachment order and per-attachment indexes; every construction limit; the allocation estimate bounding the real size; the search’s longest-match, seam-crossing and longest-over-nearest behaviour
tests/dictionary.rsbyte identity with the C encoder over six dictionary-and-payload shapes at qualities 5 to 11, with the same bytes prepared by BrotliEncoderPrepareDictionary and attached by BrotliEncoderAttachPreparedDictionary; a round trip through the C decoder with the same dictionaries attached; a ratio floor that fails if the dictionary were ignored; agreement between all six dictionary entry points; the refusal below quality 5 on every one of them; attachment order changing the stream and both orders matching the reference; one dictionary shared by four threads; flush with a dictionary attached; every preparation limit; reuse determinism across a failure and an abandoned session; and that a dictionary call never changes the next ordinary one
fuzz/afl/src/bin/dictionary.rsthe same oracles driven from fuzz input: attachment counts past the format limit, impossible budgets, the addressing round trip, and the refusal below quality five
tests/differential_c.rs, tests/roundtrip.rs, and the restbyte identity with equivalent C streaming FINISH settings and independent C decoding; native C one-shot rewrites are intentionally excluded

Known gaps

  • An attached prefix reaches qualities 5 to 11 only. The reference compiles its compound-dictionary search for H5, H6, H40, H41, H42, H55, H65 and the binary tree, so qualities 0 to 4 have no match finder that could carry a prefix match. Where the reference then ignores the dictionary, every dictionary entry point refuses with EncodeError::DictionaryUnsupportedForQuality.
  • Custom dictionaries and continuations are experimental. Their current search, context selection, transformed-index limits and checked stream offsets are specified in rfc9841-encoding.md. Parser mechanics remain in serialized-dictionary.md.
  • Framing has no pinned C container oracle. Independent wire fixtures and writer/decoder round trips validate containers; C checks compressed payload streams. See the writer and decoder for supported forms, verification and limits.
  • Large window is refused at qualities 0, 1 and 2. See §6.
  • Declared windows above 30 bits lack independent end-to-end decoding. The pinned C decoder rejects their original headers. Tests check those headers separately and compare the payload with a 30-bit declaration.
  • Retained history stops at 30 bits. Matchers and distance arithmetic use this physical ceiling regardless of the declared window.