Heap out of bounds , use motoko

canister id : 5su7s-vqaaa-aaaai-abgua-cai
When upgrade, get err:
The invocation to the wallet call forward method failed with the error: An error happened during the call: 5: Canister … trapped: heap out of bounds

method:

heap_size        = Prim.rts_heap_size();
        memory_size      = Prim.rts_memory_size();
        max_live_size    = Prim.rts_max_live_size();
        total_allocation = Prim.rts_total_allocation();
        reclaimed        = Prim.rts_reclaimed();
        rts_version      = Prim.rts_version();
        cycles           = Cycles.balance();

Why heap out of bounds ,How can I solve this problem??This problem is that I feel extremely painful and sad. I also want to know: does the canisters written by rust have this problem?

Is the canister’s memory management similar to linux’s virtual memory?
public func push(x : T, l : List) : List = ?(x, l);
Adding items to the linked list, will it be similar to the malloc implementation of c language, allocating memory from virtual space?

Where can I find more information on Prim?

I’m getting this same error when I try to upgrade my canister locally.

Installing canisters...
Upgrading code for canister service, with canister_id rrkah-fqaaa-aaaaa-aaaaq-cai
The invocation to the wallet call forward method failed with the error: An error happened during the call: 5: Canister rrkah-fqaaa-aaaaa-aaaaq-cai trapped: heap out of bounds

Here is the canister info from the replica UI:

Scheduler state
last_full_execution_round	2220754
compute_allocation	0%
freeze_threshold (seconds)	2592000
memory_usage	768365333
accumulated_priority	-1600
Cycles balance	4000000000000

I’m surprised I’m running out of heap space, when I’m only using 768 MB of memory pre-upgrade. I knew Motoko stable variable serialization took up space, but I didn’t think it would take up 3+ GB.

@claudio @rossberg Do you know what may be going wrong here? To confirm, I didn’t modify any stable variable types (or make any changes) between the last canister upgrade and the current one.

EDIT: When I reduce the memory_usage of my canister (by reinstalling and populating it with fresh data) to 753 MB and run dfx deploy, it works… Is the pre-upgrade memory_usage cutoff really somewhere between 753 MB and 768 MB above which canister upgrades don’t work??

@ggreif or @ulan might have an idea what’s going on and what the message means.

When preupgrade is running in a fragmented heap, there may not be enough capacity to push all the information into stable variables and externalize them into a monolithic buffer (which then will be copied to the stable memory space). We are aware of these two problems:

  • need to run an explicit GC before preupgrade (and possibly after)
  • need to use a streaming technique to write stable variables to stable memory.

I am working on the former and @claudio is about to tackle the latter.

1 Like

Thanks for the response.

I knew there was an intermediate buffer, but I didn’t know it would take that much space… From <800 MB to “heap out of bounds” (i.e. 3.2 GB used) seems quite extreme.

But yeah, those two workarounds sound good. Do you happen to have an ETA on either of those solutions? (No rush but would be helpful to plan.) Also, would you recommend users migrate to using the ExperimentalStableMemory Motoko library, which I assume avoids these problems?

That doesn’t look like Motoko allocation failure to me, which produces a different error message (IIRC) and more like trying to address wasm memory that hasn’t been pre-allocated by a memory grow.

Is there any chance you could share the offending code or a smaller repro?

If you have a repro, is there any chance you could share the code? I’d like to check we don’t have a bug.

More likely than not, you have changed your data structure used in stable variables which violated the upgrade rules. Please see a previous discussion Dfx deploy --network=ic heap out of bounds - #11 by PaulLiu

1 Like