Suggestion: make the maximum single memory operation an explicit limit

Developers
1

We run LLM inference on-chain (llama_cpp_canister), so our canisters allocate multi-GiB heaps. After the -deterministic-tracker GuestOS variant rolled out, loading our Qwen3-1.7B model with a 16K context stopped working. We load the model from stable memory (a virtual file in C++ canisters) into working memory:

IC0522: Canister attempted to perform a large memory operation that used
4_057_019_087 instructions and exceeded the slice limit 2_000_000_000.

An existing, deployed canister stopped working.

We found that using a smaller context still works fine, so we have a workaround, though not ideal.

After some digging, and with help from @Lorimer, we found the root cause: b7225383e unified page charging at 5 000 instructions/OS page (heap writes were 1 000, heap reads 0). A single bulk memory operation can’t be sliced, so its whole cost lands in one slice, and DTS rejects it once it exceeds slice_limit + max_slice_limit = 4 B instructions. Expressed as a size:

max single memory op = 2 × MAX_INSTRUCTIONS_PER_SLICE / dirty_page_overhead × 4 KiB

   before:  2 × 2B / 1000 × 4KiB = 15.26 GiB   (heap reads were free → effectively unbounded)
   after:   2 × 2B / 5000 × 4KiB =  3.05 GiB
   ours:    4_057_019_087 / 5000 × 4KiB = 3.10 GiB   → just over

We’re not arguing against the new cost model, but would like to point out that this particular size limit isn’t declared anywhere. It’s an emergent result of MAX_INSTRUCTIONS_PER_SLICE (a latency knob) and dirty_page_overhead (a cost knob), so it moves whenever either changes for unrelated reasons, and it isn’t obvious from either PR that a size limit is being affected at all.

Two small things that would help canisters like ours:

  1. An explicit, documented constant for it — ideally with a test pinning it — so that a change in either input surfaces the effect on maximum operation size at review time.
  2. An error that states the size: “attempted a 3.10 GiB memory operation, limit is 3.05 GiB”. The current message gives instruction counts, and the conversion to bytes isn’t obvious unless you already know the page cost.

And a related question, if the limit does become explicit: would it be possible to raise it a little? At 3.05 GiB it now sits just below the ~6 GiB heap the platform already supports, so a canister can’t initialise its whole heap in a single operation. We don’t have a strong view on the mechanism — we appreciate MAX_INSTRUCTIONS_PER_SLICE is deliberately tied to block time — but a limit at least as large as the maximum supported heap would feel natural, and would let us go back to the 16K-context configuration.

We’ll try to chunk our allocations on our side regardless although it is not going to be easy to implement.

2

Second interesting one, after the BTC cannister issue .
https://forum.dfinity.org/t/proposal-to-elect-new-release-rc-2026-07-17-04-19/74609/7?u=zackds

3

Hmm interesting. It’s unfortunate that the change had a negative impact on your workload.

Increasing the slice limit (how much of a multi-round message fits into a round) is not really an option, because that would directly affect the round duration.

In your case, would it be possible to just do multiple, smaller allocations? Because that would allow the protocol to split it up over several rounds. With the 6GB heap size limit soon getting lifted, it won’t be possible to allocate everything all at once either way.

4

Thanks for the explanation.

Not easily done. We’re vendoring regular llama.cpp code inside our llama_cpp_canister code, and this will require updating that vendored code.

Not impossible though and we will go that route.

5

Hi icpp,

This is unfortunate and I am sorry for your inconvenience. Prior to the change, we rolled out metrics on all subnets which estimated the new costs so that we could rule out cases such as yours. However, we focussed on message limits, not on slice limits. It is rather unusual (or so we thought) that a single instruction would go close to the slice limit. It is obviously a miss on our part and we should have announced this change ahead of time.

An explicit, documented constant for it

That does not make sense to me, as you have explained in the paragraph above: It’s the result of two existing constants.

we appreciate MAX_INSTRUCTIONS_PER_SLICE is deliberately tied to block time — but a limit at least as large as the maximum supported heap would feel natural,

This is again a contradiction. These read a bit like an LLM is talking :frowning:

But the error message improvement is noted.

would it be possible to raise it a little

We are still on the low end of a fair price for a page, so this is unlikely. The price may increase further in the future. But why do you think it’s difficult to fix it on your end? Why not just go from

bulk_instr(2N);

to

bulk_instr(N);
bulk_instr(N);

? That can be DTS’d between the two system API calls.

Edit: I have now seen your answer to this question.

Semi-related: This change unblocks a future change to make the heap limit much larger, which might be good news for your canister. But it also shows that the total heap size cannot ever be completely accessible in a slice or even message.

6

Sorry for that. It was indeed a lot of LLM speak there. Thanks for unwrapping that for me.

7

Actually, this is the solution. The update appears to be quite straight forward. I will report back if it indeed works out.

Thanks again for your detailed response.

8

Also kudos to @Zane for his review on the relevant commit. That made it much easier to track the issue down when I searched the forum.

9

Good news. It turns out llama.cpp can already split a large allocation into several smaller buffers, which are then zeroed one at a time instead of in a single operation — the CPU backend just doesn’t switch it on by default.

Enabling it in our wasm build was enough and now llama_cpp_canister can again load Qwen3-1.7B at the larger context window. Verified on mainnet and available in the next release (0.16.4).

Thanks again for the pointers.