C ABI (mbrotli-ffi)

mbrotli-ffi is a separate workspace crate that exports mbrotli’s one-shot codecs and owned streaming sessions to C. It builds libmbrotli_ffi as a cdylib and a staticlib (plus an rlib for its Rust tests) and ships the declarations in mbrotli-ffi/include/mbrotli.h. It depends on mbrotli with std, compression and decompression, and adds no codec logic of its own beyond reproducing two special cases of Google’s one-shot encoder.

Module and ownership boundaries

graph TD
    Header[include/mbrotli.h] -. declares .-> Root
    C[C or C++ caller] --> Root[mbrotli_ffi crate root: exported symbols and MbrotliResult]
    Root --> Core[private mbrotli_ffi::core]
    Core --> Validate[LengthSlot / Region / Arguments: pointer validation]
    Core --> Guard[guarded: catch_unwind]
    Core --> Sessions[private core::sessions: owning handles and poison guard]
    Sessions --> Owned[EncoderSessionOwned / DecoderSessionOwned]
    Core --> Codec[compress / decompress / compress_bound]
    Codec --> Compressor[mbrotli::Compressor::compress_to_uninit]
    Codec --> Decompressor[mbrotli::Decompressor::decompress_to_uninit]
    Codec --> Stored[stored_stream: uncompressed fallback]
    Tests[tests/c_abi.rs, benches/one_shot.rs, fuzz c_abi] --> Root
    Tests --> Google[google-brotli-ffi oracle]

The crate root owns only the ABI surface: the #[repr(C)] MbrotliResult enum, opaque session types, operation constants and #[unsafe(no_mangle)] extern "C" functions with their contract documentation. Everything else is in the private core module, which owns the one unsafe pointer-to-slice conversion (the output becomes &mut [MaybeUninit<u8>], so a C buffer need not be initialized), the panic guard, parameter conversion, the private thiserror error FfiError, and its reduction to a status code. mbrotli’s src/ stays free of unsafe; every unsafe block is in this crate, each with a // SAFETY: comment.

Public API surface

*output_len is capacity on entry and bytes written on return: the count on MBROTLI_OK, and 0 on every other status except a null or misaligned output_len, which is never written.

mbrotli_compress is byte-identical to BrotliEncoderCompress(quality, lgwin, BROTLI_MODE_GENERIC, ...). That holds because compress_to_uninit declares the input length as the size hint, as the C one-shot API does, and because core::compress reproduces the two places where the C one-shot API does not simply return its streaming encoder’s bytes:

• an empty input is the single byte 0x06, whatever the window;
• a stream longer than the bound, or one that does not fit in an output of at least the bound, is replaced by MakeUncompressedStream’s stored stream.

Control and data flow

sequenceDiagram
    participant C as C caller
    participant Root as extern "C" fn
    participant Core as core::call
    participant Codec as core::compress / decompress
    participant M as mbrotli
    C->>Root: pointers, lengths, parameters
    Root->>Core: forward (unsafe contract)
    Core->>Core: LengthSlot::new(output_len)
    alt null or misaligned
        Core-->>C: INVALID_PARAMETER (length untouched)
    end
    Core->>Core: read capacity, Arguments::new (null, size, overlap)
    Core->>Core: slices() — the only from_raw_parts, output as MaybeUninit
    Core->>Codec: guarded(|| codec(src, dst))
    Codec->>M: Compressor::new + compress_to_slice or compress_to_uninit / Decompressor::new + decompress_to_slice or decompress_to_uninit
    M-->>Codec: Result<usize, EncodeError / DecodeError>
    Codec-->>Core: Result<usize, FfiError>
    Core->>Core: write written or 0 to output_len
    Core-->>C: MbrotliResult

Each one-shot call builds its codec, runs it and drops it, so no state is shared between one-shot calls or threads. SIMD dispatch is mbrotli’s: the backend is detected once per codec construction, which happens once per call here, never per block.

Compression decision logic

flowchart TD
    Start[mbrotli_compress] --> P{quality 0..=11 and lgwin 10..=24?}
    P -->|no| Invalid[INVALID_PARAMETER]
    P -->|yes| Empty{input empty?}
    Empty -->|yes| One{capacity >= 1?}
    One -->|yes| Byte[write 0x06, OK]
    One -->|no| Small[OUTPUT_TOO_SMALL]
    Empty -->|no| Encode[Compressor::compress_to_uninit]
    Encode --> Fits{Ok and written <= bound?}
    Fits -->|yes| Ok[OK]
    Fits -->|no| Cap{capacity >= bound?}
    Cap -->|yes| Store[stored_stream, OK]
    Cap -->|no| Fail[OUTPUT_TOO_SMALL or ERROR]

The bound is input_len + 2 + 4 * (input_len >> 14) + 4 (2 for empty input). The stored stream is a 10-bit window header and empty metadata block (0x21 0x03), one uncompressed meta-block per 2^24 bytes with a 3- or 4-byte header using the fewest length nibbles, and an empty last meta-block (0x03). It never exceeds the bound, so an output of mbrotli_compress_bound bytes always succeeds.

Destination paths

flowchart TD
    C[mbrotli_compress] --> CU[compress_to_uninit]
    D[mbrotli_decompress] --> L{"capacity <= LINEAR_SMALL (1 KiB)?"}
    L -->|yes| ZD["BorrowedBuf::ensure_init zeroes it"] --> Linear["decompress_to_slice: output is the first member's history"]
    L -->|no| DU[decompress_to_uninit]
    DU --> Stored{one complete stored member that fits?}
    Stored -->|yes| Copy[copy payload from input]
    Stored -->|no| Ahead["zero one meta-block (at most 32 KiB) ahead, decode with output as history"]

Each pair of paths returns the same bytes, counts and errors. compress_to_uninit copies each block out of the encoder’s scratch, which on the harness corpora (q0 and q1, a fresh encoder per call as here) took as long as writing fragments in place into an initialized slice; zeroing the destination first, as earlier versions did at q0/q1, made the same calls up to 21% slower on repetitive data. decompress_to_uninit decodes into the destination as history and zeroes it only one meta-block ahead of the decoder (see uninitialized output). Earlier versions zeroed the whole destination for decompress_to_slice when its capacity exceeded the input but stayed within max(64 KiB, 4 x input_len), and decoded through the history ring otherwise. On exact-sized destinations (harness corpora: text, binary, repetitive and stored data, q1/q5/q9/q11, i7-13700KF, one core) the current paths take 9% less time in geometric mean: 15-20% on 1 MiB of text, which compresses past the ratio limit and took the ring, 28-41% on data compressed more than 100:1, where decoding outruns a separate up-front zeroing pass, and up to 12% on an 8 MiB mixed corpus; no case got slower by more than noise. Only up to 1 KiB does zeroing everything win: one decoding call instead of one per meta-block, about 7% on 1 KiB of text.

Error propagation

classDiagram
    class FfiError {
        <<private, thiserror>>
        NullLength
        MisalignedLength
        NullBuffer
        OversizedBuffer
        Overlap
        Quality
        Window
        Config(ConfigError)
        Encode(EncodeError)
        DecodeConfig(DecodeConfigError)
        Decode(DecodeError)
        Panic
    }
    class MbrotliResult {
        <<public, repr C>>
        Ok
        InvalidParameter
        OutputTooSmall
        Error
    }
    FfiError ..> MbrotliResult : conversion from FfiError reference

Argument and parameter failures map to InvalidParameter; EncodeError or DecodeError::OutputTooSmall to OutputTooSmall; everything else, including corrupt, truncated or trailing compressed input and a caught panic, to Error. Codec errors are kept as #[from] sources inside FfiError, but only the status code crosses the ABI.

Invariants

• Pointer validation before any dereference. output_len must be non-null and aligned before it is read or written. Buffers may be null only with a zero length; lengths above isize::MAX or ending past the address space are refused; input, output and the output_len slot must be pairwise disjoint, since &[u8] and &mut [MaybeUninit<u8>] may not alias. Empty buffers become empty slices regardless of their pointer.
• Output is never read before it is written. The output is viewed as &mut [MaybeUninit<u8>], so an uninitialized C buffer is sound. Compression always calls compress_to_uninit. Decompression zeroes a destination of at most LINEAR_SMALL, 1 KiB, through borrowed_buf::BorrowedBuf::ensure_init, which hands back the zeroed bytes as &mut [u8] without unsafe in this crate, and decodes with decompress_to_slice; larger destinations go to decompress_to_uninit. See uninitialized output.
• Undetectable misuse remains undefined. Dangling or too-short buffers, and concurrent access by another thread, are the caller’s contract.
• No unwinding into C. catch_unwind wraps each codec call; the closure is AssertUnwindSafe because the only state it can leave half-updated is the output buffer, whose contents are unspecified after any failure.
• Output contents after failure. Unspecified, except that MBROTLI_OUTPUT_TOO_SMALL from mbrotli_decompress leaves the decoded prefix in the full buffer.
• Decoding stops at the first failure it meets. A stream that is corrupt or truncated only after the point where the output fills up returns MBROTLI_OUTPUT_TOO_SMALL; telling it apart from a valid stream would mean decoding past the buffer into nowhere. A larger buffer then yields MBROTLI_ERROR. Google’s one-shot decoder reports both cases as its single error result.

Verification

• mbrotli-ffi/src/core.rs unit tests: region validation and overlap, parameter ranges, panic guard, status mapping and error sources, stored headers at every nibble boundary, stored streams up to 2^24 + 1 bytes, the fallback and the bound.
• mbrotli-ffi/tests/c_abi.rs: byte identity with BrotliEncoderCompress for qualities 0–11 at windows 10/16/22/24 on text, binary, incompressible, one-byte and empty inputs; identical accept/reject at capacities between the stream and the bound; bound equality with BrotliEncoderMaxCompressedSize; round-trips through both decoders; every rejected argument; concurrent calls; and that the header’s enumerators match MbrotliResult.
• fuzz/afl target c_abi (see fuzzing).
• mbrotli-ffi/benches/one_shot.rs: both one-shot APIs on identical inputs, validated for identity and round-trip before timing.

Streaming sessions

MbrotliEncoderSession / MbrotliDecoderSession are opaque C handles. Their private allocation is core::sessions::Handle<T>: an owned Rust session and a poison flag. Handle<T> deliberately has Rust layout and stores the session inline. Its layout is not the C ABI: the header declares incomplete struct types, every exported signature passes pointers, and only this library allocates, accesses and destroys the allocation. Callers never need its size or alignment. Changing the Rust session, field order or handle allocation size therefore does not change the C ABI. A second Box<T> or repr(C) on the private storage would add no compatibility guarantee. The public Rust opaque marker structs also do not describe the private allocation and must never be instantiated or passed by value. Handles belong to the library instance that created them; this contract does not support moving live allocations between separately loaded library versions.

There are no borrowed codecs, self-references or caller buffers stored in the wrapper. create returns null on invalid parameters or failure; destroy drops the allocation exactly once, and accepts null. Independent handles may run concurrently; each individual handle requires exclusive access.

MBROTLI_PROCESS=0, MBROTLI_FLUSH=1, MBROTLI_FINISH=2 are passed as int so unknown discriminants can be rejected safely. Decoder process rejects Flush; its separate flush function uses Process. Both process functions return MBROTLI_NEEDS_INPUT=4, MBROTLI_NEEDS_OUTPUT=5, or MBROTLI_FINISHED=6 on success. Existing one-shot status discriminants 0–3 remain unchanged.

sequenceDiagram
    participant C as C caller
    participant S as core::sessions
    participant R as Owned Rust session
    C->>S: create(configuration)
    S->>R: new + into_session(default stream)
    S-->>C: opaque owning pointer
    C->>S: process(handle, buffers, counts, operation)
    S->>S: validate handle, slots and all pairwise regions
    S->>S: mark poisoned, enter catch_unwind
    S->>R: process_uninit(input, output, operation)
    R-->>S: progress or failure
    S->>S: clear poison only on success, publish exact counts
    S-->>C: NeedsInput / NeedsOutput / Finished / Error
    C->>S: reinit or destroy
    S->>R: reinit(default stream) or drop

On entry counts describe available input and output capacity; on return they are consumed/produced counts for that call. Input is advanced by consumed; NeedsOutput requires another output buffer even when input is empty. The Finish operation must be retried with its exact unconsumed suffix. Encoder Flush ends with NeedsInput and permits more input afterward. The default single-member decoder leaves protocol suffix bytes unconsumed on Finished. Streaming encoding follows the Rust session API, without one-shot empty-input normalization or a stored-stream fallback to enforce the one-shot bound.

All three output-taking functions use process_uninit: a C output buffer need not be initialized. Success initializes exactly the reported prefix and never writes beyond it. Decoder codec errors also report exact progress and preserve that guarantee; encoder errors and panics return zero counts with unspecified output. Null/misaligned arguments, unknown operations, impossible lengths and region overlaps return InvalidParameter without changing counts or state. Dangling pointers, undersized allocations, wrong-kind handles and concurrent access remain caller violations. Validation precedes all slice/handle references.

stateDiagram-v2
    [*] --> Active: create
    Active --> Active: process / flush / NeedsOutput
    Active --> Finished: Finished
    Finished --> Finished: idempotent process/finish
    Active --> Poisoned: codec error or panic
    Active --> Active: invalid arguments (no change)
    Finished --> Active: reinit succeeds
    Poisoned --> Active: reinit succeeds
    Active --> Active: reinit succeeds
    Active --> Poisoned: reinit fails
    Poisoned --> Poisoned: process returns Error
    Active --> [*]: destroy
    Finished --> [*]: destroy
    Poisoned --> [*]: destroy

The guard sets poison before invoking codec code and retains it on failure or panic. Reinit is guarded too and clears poison only after success. Destruction also catches unwinding. SIMD detection happens at codec construction and is retained across calls and reinitialization; no dispatch is added to inner loops.

Verification: tests/sessions.rs exercises every exported session function with uninitialized output, tiny buffers, empty input, flush/finish, trailing input, invalid pointers/overlaps and reuse after failures. Private unit tests inject panics into construction, processing and reinitialization. These run under Miri and ASan via scripts/check_simd_uninit.py (shard 1, or --ffi-only). The existing AFL/libFuzzer c_abi body adds bounded session round trips and reuse. benches/sessions.rs compares q5/w22 streaming C calls with Google’s encoder and decoder, with creation/destruction included, 4 KiB output chunks and validation outside timing; it reports compressed sizes alongside throughput. examples/sessions.c compiles and runs as C or C++, linking only the public header and library. An attempted sizeof(mbrotli_encoder_session) or sizeof(mbrotli_decoder_session) is rejected by C as an incomplete type.

Focused verification commands (workspace root unless stated otherwise):

cargo test -p mbrotli-ffi --locked
cargo llvm-cov -p mbrotli-ffi --lib --test sessions --locked
python3 scripts/check_simd_uninit.py miri --ffi-only
RUSTFLAGS=-Ctarget-cpu=x86-64-v3 CARGO_TARGET_DIR=target/miri-v3 \
  python3 scripts/check_simd_uninit.py miri --ffi-only --target x86_64-apple-darwin
python3 scripts/check_simd_uninit.py asan --ffi-only
cargo bench -p mbrotli-ffi --bench sessions --locked
# From fuzz/afl, after cargo afl build --release --bin c_abi:
AFL_MAP_SIZE=1048576 AFL_SKIP_CPUFREQ=1 AFL_NO_AFFINITY=1 \
  AFL_I_DONT_CARE_ABOUT_MISSING_CRASHES=1 \
  cargo afl fuzz -i regressions/c_abi -o /tmp/mbrotli-c-sessions-afl \
  -S smoke -V 30 -c - -- target/release/c_abi

The 1 MiB AFL coverage map fits the local macOS shared-memory limit without system tuning; the fuzzer still negotiates the instrumented target’s map size. The initial ARM64 smoke run (AFL++ 4.40c, cargo-afl 0.18.2, Rust 1.99.1, non-experimental release target) executed 94,363 inputs with 99.93% stability, zero saved crashes and zero saved hangs. Generated reports and findings remain local artifacts.

Known gaps

• Large-window compression (lgwin 25–30, which BrotliEncoderCompress accepts) and compression modes other than generic are not exposed.
• Sessions expose default stream settings and reusable codec state, but no dictionaries, input-size hints, decoder budgets, member modes or configuration changes. Recreate a handle to change encoder quality or window.
• On aarch64, clang contracts the C oracle’s floating-point multiply-adds into FMA instructions, so Google’s qualities 10–11 can differ from mbrotli on rare inputs; built with -ffp-contract=off the C library agrees. The tests and the fuzz target skip that comparison there for the input that triggers it.
• There is no no_std build: the crate needs std for catch_unwind, the system allocator and runtime SIMD detection.
• Decoding into a C buffer still zeroes every output byte once, just ahead of the decoder. On exact-sized buffers (i7-13700KF, one core, harness corpora) mbrotli_decompress therefore takes 1.2% longer than decompress_to_slice into an already initialized buffer (geometric mean; previous versions 11.9%), most on data compressed more than 100:1 and on 1 MiB of repetitive text, where decoding runs as fast as zeroing; stored, ordinary text and binary data are at parity.