Same bytes, wrong order

Inco released Splash this week: an open-source inference engine for Apple silicon, built around one model. The speed claims are real enough to be interesting, and the code is substantial - about 101,000 lines of C++, Objective-C++ and Metal, Apache-2.0, no telemetry.

It also only runs the models Inco publish. Splash ships a package reader. There is no writer. If you want to serve a fine-tune, you cannot.

I wanted an abliterated Qwen3.8-27B on it. So I wrote the writer.

the format is not a quantizer

The first useful thing was in their own manifest:

"upstream": {
  "target": { "repo_id": "mlx-community/Qwen3.8-27B-4bit" }
}

They did not quantize from BF16. They repacked an MLX 4-bit model. And MLX’s affine 4-bit at group size 64 is numerically the same scheme Splash stores: 4-bit codes, bf16 scale, bf16 bias, w = q * scale + bias.

That changes the whole problem. Conversion is not requantization, it is a relayout. No weight gets re-encoded, so there is no quality question to argue about. It also means the source is a 15 GB MLX repo rather than a 55 GB BF16 one.

The layout itself came out of the Metal kernels. Weights are tile-major: 256 output rows per tile, then quant group, then the row inside the tile. Scales and biases follow the weight region as two planes. Fused projections are row concatenations padded to a multiple of 256, which for gated DeltaNet means [in_proj_qkv, in_proj_z, in_proj_b, in_proj_a, 160 zero rows].

MLX packs eight codes into a little-endian uint32, which means every byte already holds two consecutive codes with the earlier one in the low nibble. That is byte for byte what Splash’s uint4b_format tensor reads. So whole 32-byte quant groups move unchanged and the packer is mostly a permutation.

proving it instead of asserting it

Reverse-engineering a binary format gives you a pile of plausible guesses. The question is how you know.

The answer here was cheap: pack the model they already published, and diff against their file. Inco distribute Qwen3.8-27B-Splash. mlx-community distribute the 4-bit source they packed it from. Both public, both Apache-2.0. Run my packer on their input and the output should be their output.

layer-0.bin   (gated DeltaNet)  216,203,264 bytes   identical
layer-3.bin   (full attention)  209,469,440 bytes   identical
layer-40.bin  (gated DeltaNet)  216,203,264 bytes   identical

Byte-identical, across both layer types and all three source shards. That is a much stronger statement than “the numbers look about right”, and anyone can re-run it.

Downloading whole 5.3 GB shards to check one layer is wasteful, so I wrote a slicer that pulls individual tensors out of a remote safetensors file with HTTP range requests. Two layers cost 425 MB instead of 5.3 GB. Safetensors puts a JSON header with exact byte extents at the front of the file, which makes this easy and which more tooling should exploit.

Getting to byte-identical rather than nearly identical took one odd detail. The gated DeltaNet decay is stored as float32 -exp(A_log), and MLX’s float32 exp disagrees with numpy’s by one ULP on about a third of those values. Using mlx.core.exp closed the last 15 differing bytes in the file. Numerically it means nothing. As a signal that you have understood the producer exactly, it means a lot.

then it loaded, validated, and emitted noise

The package passed Splash’s own validator. Every artifact matched its SHA-256. The engine loaded it in 14 seconds and reported ready.

Every request came back with exactly one token.

Not a refusal. Not a template problem. Forcing sampling past the first token gave five tokens of ")\n, - the logits were garbage, and at temperature zero the argmax of garbage had landed on a stop token.

So began the part worth writing about, which is mostly me being wrong.

I blamed the compiler. We had built with Xcode 27’s Metal toolchain, newer than whatever Inco used, and q4_mpp_tiles.h carries explicitly version-dependent reasoning about cooperative-tensor fragment layouts. Plausible. Disproven by running their full Metal kernel suite against CPU reference oracles: everything passes, sub-ULP, with GPU validation on.

I blamed the draft. We pair their DFlash draft, trained on the official model, with abliterated weights. Also plausible. Disproven by one line in the scheduler: “A stop token or a one-token budget is selected by prefill itself.” The first token never touches the draft.

I blamed my own build of the engine. Disproven by downloading Inco’s complete 15 GB package and running it on the same binary, where it answered normally.

Each of those took real time, and each was killed by a measurement rather than an argument. That is the only part of the process I would defend.

the bug

embedding.bin is not the tile-major matmul layout the layers and head.bin use.

The token embedding is a gather, not a multiply. shared/embedding.metal indexes it directly:

weights[token * (Hidden / 2) + dim / 2]
scales [token * QuantGroups + dim / 64]

That is plain row-major, and the loader reads it with readQ4ProjectionComponents, three separate sections, where head.bin uses readQ4Projection, one tile-major section. Two different spellings of the same tensor.

Here is why it survived everything. Both layouts occupy exactly the same number of bytes, because each region is already a multiple of the 16 KB section alignment. So:

• the file size matched their published file, to the byte
• the manifest verified, because I had computed the hashes from my own output
• Splash’s validator accepted it
• my pack-then-unpack round-trip test passed

That last one is the trap. A round-trip test validates your assumption against itself. I packed with the wrong layout and unpacked with the same wrong layout, and got my input back, and called it verified.

What broke it open was decoding their file with my assumption instead of my own:

tile-major:  cosine 0.0001   std 0.01402 vs 0.01402
row-major:   cosine 0.9984   std 0.01402 vs 0.01402

Identical standard deviation, zero correlation. Same values, different order. That is a permutation, and it is unmistakable once you look at it from the outside.

If you take one thing from this: when you reverse-engineer a format, your oracle has to be a file you did not produce.

two things I was wrong about out loud

With it running, I told the user the speed gap to Inco’s published numbers was the mismatched draft, and offered to train a new one. Then I measured it.

The draft accepts at the same rate against the model it was trained on as against mine. It is not mismatched, a bespoke one would gain nothing, and the repack is faster than the official package on the same machine. I had been about to propose training a 2.3B-parameter draft on a laptop to solve a problem that did not exist.

The actual limiter is memory. The scheduler almost never batches, four concurrent requests measure worse than one, and the package only fits at all because the engine’s preflight demands host memory equal to the whole package size while the loader maps the weights, and counts reclaimable inactive pages as used.

what is public

The converter and tooling are in a fork: srijanshukla18/splash, branch uncensored-splash-packer. Four commits, kept separate so Inco can take any of them individually: a Makefile fix for the Metal toolchain move in Xcode 26+, a launcher bug where the chosen port is never passed to the server, the opt-in memory-gate change, and the packer itself with its tests.

Not the weights. With the converter public, anyone reproduces the exact package in about twenty minutes from public, ungated, Apache-2.0 inputs, and I would rather not host 16 GB or personally own redistribution of a model with no guardrails. The interesting artifact was never the weights.