MemoryId(1): grow failed

Developers Rust
6

Hi @gravity_vi,

Not sure what’s the best way to check memory allocation, but here is one way (if it’s a Dapp canister directly controlled by SNS Root, you can hit the canister_status endpoint on SNS Root, but this one doesn’t seem to be):

  1. From the canister link you posted above, navigate to its subnet: Subnet: o3ow2-2ipam-6fcjo-3j5vt-fzbge-2g7my-5fz2m-p4o2t-dwlc4-gt2q7-5ae - ICP Dashboard
  2. Pick an active node machine (e.g. this one: Node Machine: 23jbm-6z6mi-ki2ut-fkz5x-yy4uy-6llls-yyodh-xv537-2qonk-2iod3-jae - ICP Dashboard), and get the IP address of the node machine
  3. Hit the replica dashboard using the IP address: http://[2403:f000:300:6:6801:e2ff:fe86:7bd0]:8080/_/dashboard
  4. Find the canister by searching its ID on the page

For this canister, I’m seeing 1024MiB memory allocation and 28495226 memory usage (28MiB/1024MiB).

I noticed that this canister used to be controlled by SNS Root (based on this proposal), and note that we recently found and fix an issue with SNS Root setting Dapp canister memory_allocation to 1GiB (see here). We also added a ManageDappCanisterSettings SNS proposal type for SNSes to change their memory_allocation so that those canisters with 1GiB memory allocation can now be changed.

For this specific issue, the memory usage seems to be nowhere close to the memory allocation. However, the memory grow failure depends on both the remaining capacity (>900MiB) and the amount the canister tries to grow to. Therefore I tried to take a look at the Storable implementation of the Key/Value types in the StableBTreeMap you provided and found this: hot-or-not-backend-canister/src/lib/shared_utils/src/common/types/wasm.rs at 23d51473b4661d9b6fd27fcdd458a61123d8507c · go-bazzinga/hot-or-not-backend-canister · GitHub. If I understand correctly, the max_size is actually set to 200MB (is it intentional, given the comment saying it’s 2MB?). I’m not an expert in stable structures, but in my understanding it can also try to allocate not just the node being inserted into but its children (with degree 6-11), which can easily hit the 1GiB limit (@ielashi could you comment on that?)