We have an issue with a canister containing NFT’s it is eating through cycles like no tomorrow. 2T an hour. We notice that even though we loaded around 1.2 GB of assets the canister size is hitting 4 BG.
Canister is t2mog-myaaa-aaaal-aas7q-cai. What are we doing wrong?
Hi @domwoe I maintain the canister. The code is not public but I will make it public when I get back to my laptop.
We are using the same code as @bob11 but are seeing a much higher burn rate.
.5T cycles/day vs. 1T cycles/hour [correction: 2T per hour]
I recently upgraded the canister to include some log/monitor methods and that’s when we noticed it was reporting a memory size of 4GB. It’s my understanding that canisters have a strict 4GB memory limit so I was thinking that might be contributing to the increased burn rate somehow.
As @jonit mentioned, we uploaded our assets (jpegs) directly to the canister. We aren’t using an asset canister at the moment. This collection of images only totaled 1.3GB in size but the canister is about 3x that size.
One last piece of context; after a deployment our canister will eventually enter a state in which it can’t be upgraded. This usually happens right around the time heap memory hits 2GB. Regular method calls still work (transfer, list, settle, etc.) but no upgrade and no ability to stop the canister (it hangs in a “stopping” state). We can only resolve this by raising the freezing threshold to the point the canister stops; then we lower it again and deploy/start the canister. This brings it back to a state where the heap is at 1GB and upgrades are successful
There is no non-linear increase in (storage) costs when you get close to the 4 GB limit.
I don’t know if there are Garbage Collector related costs that might be relevant here.
One thing I noticed:
You might have multiple xnet inter-canister calls (ledger, cap) in your heartbeat which could open a lot of call contexts, in particular if the callee needs time to respond because it is busy. I think (and I might be wrong) that these call contexts are also stored on your available heap.
Do you hit the cycle limit in the preupgrade hook or do you get another error?
I must have misread this the first time, i thought you were saying there is a non-linear storage cost. That is unfortunate, i was hoping it would be something obvious like that
Do you have an idea for why the canister is so large when the assets on disk are a 1/3 of the size?
Thank you for pointing out the xnet inter-canister calls. I will keep this in mind as I continue troubleshooting the issue. One of the things I want to test out is reducing the frequency of these calls.
I don’t have the exact error on-hand, but I believe it was something along the lines of “the canister_pre_upgrade attempted with outstanding message callbacks (please stop the canister and attempt the upgrade again)”
The “memory” as reported by the CanisterGeek interface is constant at 4GB. The “heap memory” starts out at 1GB after an upgrade, fluctuates between 1GB & 2GB for a bit, and then it holds steady at 2GB.
(quotes are intended to convey that I’m not familiar enough with the terms to understand the difference in this context)
Also, which version of dfx or motoko are you using. We have optimized the memory requirements of upgrades. These used to take up to 2x space during upgrade (to serialize stable variables into ordinary memory before copying them stable memory). The new releases serialise directly to stable memory in a streaming fashion.
I am using the ic-py python agent to call the ‘addAsset’ method. The ‘asset’ type defined in the canister expects a ‘blob’ entry but the ic-py agent doesn’t provide a ‘blob’ type so i had to structure it as ‘vec vec nat8’.