On-chain LLM inference under instruction budgets, measured live on ICP mainnet

Developers Showcase
1

Hi everyone,

We have just published a research paper measuring what actually governs LLM inference inside ICP canisters, with the forward pass running fully in consensus, not through off-chain workers, oracles, or proof systems.

Paper: On-Chain LLM Inference Under Instruction Budgets: An Instruction-Budget Cost Model, Ternary Floor Evidence, and Session Costs Authors: Julien Aerni (Meotis Sàrl), Siméon Fluck (Kaizen Corp SA), Dustin Becker (ORIGYN Foundation) Paper + artifact (MIT code, CC BY 4.0 data): https://doi.org/10.5281/zenodo.20607598 Reproduction repo: https://github.com/Simlowker/instruction-bounded-inference-artifact

The core idea
On ICP, decode throughput is not bandwidth bound the way it is on GPUs. It is bounded by the deterministic 40B instruction budget per update call. That turns inference into a budgeted compute problem with a simple first order law:

tok/call ≈ B / (α_eff × 2P)

where α_eff is an effective cost coefficient that is a property of the software stack, not the hardware. Across 11 modern decoder models from 8 families, α_eff clusters at 1.53 (leave-one-architecture-out MAPE 7.7%).

Main results
1.⁠ ⁠A 2.9× gap from software alone. The same Qwen 2.5 0.5B Q8_0 GGUF, the same mainnet canister framework, the same 40B cap: 10 tok/call on the onicai baseline vs 29 tok/call on our fork. Identical weights, identical boundary; the difference is kernel path vectorization (SIMD dispatch of the quantized matmul inner loops). The result was independently re-validated from a clean rebuild of public sources (REPRODUCE.md, ~30 min CPU, no cycles required).

2.⁠ ⁠Matmul is 98.7% of decode cost. Everything else (normalization, attention plumbing) is ≤ 1.3–1.7%. We also prove a small theorem: per-element unstructured sparsity strictly increases cost on the measured integer paths. There is exactly one first order optimization lever, and it is the matmul kernel.

3.⁠ ⁠A ternary floor, live under consensus. We built a custom TQ2_0 (2-bit ternary) WASM SIMD kernel, absent from upstream llama.cpp, and ran purpose-trained TriLM models at the empirical floor of the decode path: α_eff ≈ 1.00 at both 560M and 3.9B. Outputs are byte-identical across a local replica, the 13-node Swiss Subnet, and a standard ICP mainnet subnet, across build generations and independent weight conversion pipelines.

4.⁠ ⁠Sessions can be budgeted analytically. Multi-call stateful sessions pay a predictable two-term IO tax (KV cache save/load), linear in cache size (R² 0.9998 on the load sweep). A 10-turn chat demo ran on the Swiss Subnet under 13-node consensus.

5.⁠ ⁠Retrieval fits the regime well. EmbeddingGemma-300M Q4_0 runs fully on-chain at ~126 input tok/call. Batched encode amortizes per-weight loads that single token decode cannot.

Why this matters for the IC
The practical recipe that falls out: choose the model for quality (today the binding constraint), choose parameter count and quantization from the instruction law, optimize the matmul kernel, and treat saved session state as part of the budget. Interestingly, the 40B budget is not even the operative deployment ceiling at current scales; wasm heap and page access limits are (the paper includes an IC0524/IC0502 characterization).

Nothing here requires trusting us. Every load-bearing number resolves to a named CSV row in the artifact, the fork gap rebuild is fully scripted.

Happy to answer questions on methodology, the Swiss Subnet runs, the ternary kernel, or anything else. Feedback and refutation attempts are very welcome; the claim-to-evidence matrix in the repo is designed exactly for that.

2

Hey, this is awesome work!

I am the creator & maintainer of llama_cpp_canister, the baseline in your comparison, so this of course piqued my interest greatly, since the instructions limit is indeed the Achilles heel of fully on-chain LLM inference.

First of all, thank you for sharing this, and for the care you put into making it checkable.

The published artifact, the claims-evidence matrix, the REPRODUCE.md walkthrough — I was able to clone the repo and trace your 2.9× result down to the specific line of code in our own build in under an hour. That is a rare thing to be able to say about a performance claim, and I appreciate it. I did not yet actually run it to verify the increased throughput, but will do so.

In some more detail, here’s what I found on our side in the llama_cpp_canister implementation:

  • Our icpp.toml does pass -msimd128 to the C sources, so this was never a missing-flag issue.
  • The real cause is the vintage of the llama.cpp we vendor. In our ggml-cpu-quants.c, only ggml_vec_dot_q5_0_q8_0 and ggml_vec_dot_q5_1_q8_1 have a wasm_simd128 branch. ggml_vec_dot_q8_0_q8_0 has NEON and AVX2 paths but no WASM path, so it falls through to the scalar #else. Given that matmul is ~98.7% of decode cost (your finding!), a Q8_0 model on our build runs its dominant kernel entirely scalar. That most likely fully accounts for the gap.
  • Upstream llama.cpp has since added ggml/src/ggml-cpu/arch/wasm/quants.c, with hand-written WASM SIMD kernels covering q4_0, q5_0, q5_1, q8_0, and q2_K through q6_K — which your fork compiles in.

Beyond the fork comparison, the tok/call ≈ B / (α_eff × 2P) result is the part I expect we’ll actually use day to day. Being able to predict tokens-per-call from parameter count before deploying a model is genuinely useful for anyone building under the instruction budget.

Great work. I’ll report back once I’ve had a chance to run it myself and dig into it a bit deeper.

3

Heyy! Thank you for your feeback, you nailed the diagnosis. For the 2.9x the credit goes where you found it: our fork vendors a recent upstream llama.cpp, which ships hand-written WASM SIMD kernels in ggml-cpu/arch/wasm/quants.c, including the q8_0 path that your build runs in scalar. So yes, vintage, not flags. Same hardware, software-determined cost, which is exactly the point of the paper. To be clear, our own kernel work (the custom ternary TQ2_0 path and the Flash Attention SIMD path) is separate from the fork-gap number.

One thing on the 98.7% matmul figure you quoted. In the paper’s CSV that number was backed by a single profiled config, Pythia-70M Q8_0. We had a second one (Qwen-0.5B, around 98.3%) in our notes but never published it. The artifact now has profiles for 5 more configs. Short version: matmul dominance holds for transformers, 95.6 to 98.5% across Q4_0, Q8_0 and ternary, but it breaks for pure SSM. Mamba-370M drops to 75%, with 20% going into SSM_SCAN. So treat it as transformer-scoped. Data is in artifact/data/profiling/per_operation_multiarch_2026-07-22.csv.

Looking forward to your run. If any number doesn’t match on your side, that’s exactly what we want to hear.

4

I did the upgrade to the latest upstream llama.cpp and can confirm your findings match. :tada:

The new llama_cpp_canister build gets 28 tokens per update call on the same Qwen2.5-0.5B-Instruct Q8_0 model — up from the ~10 with the previous build, so ~2.8×, basically your 2.9x.

It was measured the same way (binary search on max_tokens_update until it hits the 40B instruction limit), and verified live on main net.

btw — a couple things came out of this port that we’re upstreaming: we lifted the env/getenv handling down into icpp-pro itself (fixing it at the toolchain level so future ports don’t have to work around it), and we reused your shim approach for the WASI side — worked really well – will make sure to attribute your work.

QA for both icpp-pro and llama_cpp_canister is running now, releases expected today/tomorrow.

Great diagnosis, really appreciate the exchange. :folded_hands:. This is going to push on-chain LLM capabilities a lot further.

5

Quick toolchain update :hammer_and_wrench: — we just released icpp-pro 5.4.2, which will make future llama.cpp upgrades a lot easier.

It fixes the getenv trap at the toolchain level (__wasilibc_environ now initializes to an empty environment, so getenv() returns NULL instead of trapping).

Since your build already runs on icpp-pro, bumping to 5.4.2 lets you drop the per-site getenv guards entirely — thanks again, your shim setup pointed us at the clean fix.

Release notes: https://docs.icpp.world/release-notes.html (5.4.2)

6

This is very exciting to read, great work.

On chain embedding and inference could bring heap of very cool new competitive things.

7

Update: llama_cpp_canister v0.12.0 is out :rocket:

This is the llama.cpp b10076 upgrade we’ve been talking about — it recovers the throughput you diagnosed: ~28 tokens/call for Qwen2.5-0.5B Q8_0, up from ~10 (~2.8×), verified live on mainnet.

I added an Acknowledgments section in the README crediting the three of you — your diagnosis of the missing WASM SIMD q8_0 kernel, and the WASI shim approach we adapted from your work. :folded_hands: