loc bengaluru, ist | local --:-- srijanshukla18@gmail.com →
[post]/ai/same-bytes-wrong-order

writing a model packer for Inco Splash

/ 7 min read· ai

Inco's Splash engine ships a package reader and no writer, so it only serves the models they publish. I wrote the writer to run an uncensored Qwen3.8-27B, proved it byte-identical to theirs, and then spent hours on a layout bug that passed every check I had.

Inco released Splash this week. It’s an open-source inference engine for Apple silicon, built around one model. The speed claims looked real enough to try it out, and the codebase is huge - around 101,000 lines of C++, Objective-C++ and Metal, Apache-2.0, no telemetry.

Catch is, it only runs the models Inco publishes. Splash ships a package reader, not a writer. So if you wanna run a fine-tune of your own, you can’t.

I wanted to run an abliterated Qwen3.8-27B on it. So I wrote the writer.

The format is just a relayout, not a quantizer

First useful clue was right there in their manifest,

"upstream": {
  "target": { "repo_id": "mlx-community/Qwen3.8-27B-4bit" }
}

They didn’t quantize from BF16. They just repacked an existing MLX 4-bit model. And MLX’s affine 4-bit at group size 64 is numerically the exact same scheme Splash uses: 4-bit codes, bf16 scale, bf16 bias, w = q * scale + bias.

That makes life much easier. We don’t have to requantize anything or worry about losing quality, we just have to rearrange the bytes into the layout Splash expects. Plus we only have to download a 15 GB MLX repo instead of 55 GB of BF16 weights.

Reading through their Metal kernels, the layout works like this:

  • Weights are tile-major: 256 output rows per tile, then quant group, then the row inside the tile.
  • Scales and biases sit after the weight region as two planes.
  • Fused projections are row concatenations padded to a multiple of 256. For gated DeltaNet that’s [in_proj_qkv, in_proj_z, in_proj_b, in_proj_a, 160 zero rows].

MLX packs eight 4-bit codes into a little-endian uint32, which means each byte already has two consecutive codes with the earlier one in the low nibble. Splash’s uint4b_format tensor reads the exact same byte representation. So whole 32-byte quant groups can be copied over as-is, and the packer is basically just permuting where they sit in the file.

Checking if the packer is actually right

When you reverse-engineer a binary format, how do you know you didn’t mess up some offset?

Simple trick here: Inco already published Qwen3.8-27B-Splash, and mlx-community has the 4-bit source model they built it from. So if I run my packer on the mlx-community weights, my output files should match Inco’s published files byte-for-byte.

layer-0.bin   (gated DeltaNet)  216,203,264 bytes   identical
layer-3.bin   (full attention)  209,469,440 bytes   identical
layer-40.bin  (gated DeltaNet)  216,203,264 bytes   identical

Byte-identical across both layer types and all three source shards.

Btw, downloading full 5.3 GB safetensors shards just to test one layer was super annoying, so I wrote a small script using HTTP range requests to pull only the tensors I needed. Safetensors puts a JSON header at the very start of the file with exact byte offsets, so pulling two layers took 425 MB instead of 5.3 GB.

One weird thing I ran into while chasing 100% byte-identical output: the gated DeltaNet decay is stored as float32 -exp(A_log). Using numpy.exp gave me 15 bytes that differed by 1 ULP from Inco’s file. Switched it to mlx.core.exp and those 15 bytes matched too. Doesn’t matter for model quality obviously, but when you want diff to return zero, it matters.

It loaded, validated, and spat out garbage

So I packed the abliterated model. It passed Splash’s own validator. Every SHA-256 matched. Engine loaded it in 14 seconds and said ready.

And then every prompt returned a single token and stopped.

Not a refusal or chat template issue. When I forced it to keep sampling past the first token, it output ")\n, - complete garbage logits, and at temperature 0 the argmax of the garbage happened to be a stop token.

I wasted a bunch of time chasing the wrong theories here:

  1. Thought it was the Xcode 27 Metal compiler. q4_mpp_tiles.h had comments about compiler-version-specific fragment layouts. Ran Inco’s Metal kernel test suite against their CPU reference to check - everything passed fine. Not the compiler.
  2. Thought it was the speculative decoding draft model. We were pairing Inco’s official DFlash draft model with our abliterated target weights. Checked the scheduler code and saw: “A stop token or a one-token budget is selected by prefill itself.” First token doesn’t even touch the draft model. Not the draft.
  3. Thought my engine build was broken. Downloaded Inco’s full 15 GB official package and ran it on my binary. Worked fine.

The actual bug

Turns out I had tested layer-0.bin, layer-3.bin, layer-40.bin, and head.bin against Inco’s files, and assumed embedding.bin worked the same way. It doesn’t.

embedding.bin is not a matmul, it’s a lookup (gather). Look at shared/embedding.metal:

weights[token * (Hidden / 2) + dim / 2]
scales [token * QuantGroups + dim / 64]

That’s plain row-major! And the C++ loader reads it with readQ4ProjectionComponents (three separate sections), whereas head.bin uses readQ4Projection (one tile-major section).

Why didn’t any of my checks catch this?

  • Both layouts take up the exact same number of bytes because each region already aligns to 16 KB.
  • The manifest hashes matched because I generated the manifest from my own output.
  • Splash’s validator only checks sizes, alignments, and hashes.
  • My round-trip test passed because I wrote the unpacker using the same wrong assumption as the packer. Classic trap - testing your own bug against itself.

As soon as I ran my unpacker against Inco’s embedding.bin instead of mine, it was obvious:

tile-major:  cosine 0.0001   std 0.01402 vs 0.01402
row-major:   cosine 0.9984   std 0.01402 vs 0.01402

Same standard deviation, zero cosine similarity. Same bytes, wrong order. Fixed embedding.bin to write row-major and the model immediately started talking normally.

Draft model speed

I also assumed the official DFlash draft model would have terrible acceptance rates against the abliterated weights, and that I’d have to train or distill a new draft model. Checked the actual numbers before doing anything stupid:

acceptancetokens/stepdecode
Uncensored repack30.0%3.1128.6-32.1 tok/s
Official Qwen3.8, Inco’s own package29.8%3.0917.5-21.0 tok/s

30.0% vs 29.8%. Basically identical. Good thing I checked before trying to train a 2.3B draft model on a laptop.

The real bottleneck on my machine is just RAM. The scheduler barely batches because of memory pressure, and Splash’s preflight check wants free RAM equal to the entire 15 GB package size before mmap-ing the weights (counting macOS inactive/reclaimable pages as “used”).

Code

Pushed the packer and fixes to my fork: srijanshukla18/splash on the uncensored-splash-packer branch. Split into four clean commits:

  • Makefile fix for where Xcode 26+ moved the Metal toolchain
  • Launcher fix where the custom --port flag wasn’t actually passed to the server
  • Flag to bypass the overly strict macOS memory preflight check
  • The packer script + tests

Not hosting the 16 GB weights myself, you can generate the whole package locally in ~20 mins from the public mlx-community repo using the script.