Two questions about canister storage

Hi guys, can you answer these questions.thanks.

  1. What is the current maximum storage of canister? 4GB or 8GB?

  2. How can I query the storage that the canister has now used?

1 Like

Hi Rickey.
With almost every release, the size increases in DFX 0.8.1 4 GB in DFX 0.7.2, it is better to set it to 2 GB. The canister, like the actor, is the storage, you need an ID. Next, knowing the ID canister = actor(text_canister_id). That is, you have to implement the storage of identifiers somewhere

1 Like

The stable memory of a canister is 8 GB (used to be 4 GB). The Wasm heap is still limited to 4 GB.

I will post an example to denote why I emphasized stable memory:

  • Storing a 6 GB video - very possible
  • Declaring a 6 GB Motoko Array that is thrown away after execution - not possible

Think about the Wasm heap as the “RAM memory” accessible to a normal program and the stable memory as the “disk” and this is close enough. You can have 6GiB on the “disk”, but not all of it in “RAM” at the same time.

5 Likes

Second question:
You can use the functions of Prim.mo to get the memory has used:

Prim.rts_memory_size() -> Nat : the maximum memory has been used
Prim.rts_heap_size() -> Nat : wasm(canister) heap size at present
1 Like

Hi @diegop thank you for your detailed answer.
For example, I have a userMap now. When I upgrade canister, it will run preupgrade and postupgrade. If the data stored in my userMap is larger than 4GB, my upgrade will fail. Is that right?

var userMap = HashMap.HashMap<Text, User>(1, isEq, Text.hash); 

stable var userEntries : [(Text, User)] = [];

system func preupgrade() {
  userEntries := Iter.toArray(userMap.entries());
};

system func postupgrade() {
  userMap := HashMap.fromIter<Text, User>(userEntries.vals(), 1, isEq, Text.hash);
  userEntries := [];
};

Correct. userMap is not in the stable memory, so it can never be larger than 4GB. You can use TrieMap and define it as stable. This way you store more than 4GB and you don’t need the pre/post upgrade hooks, because the map is already in the stable memory, and it will persist across upgrades.

1 Like

@chenyan, no, actually, stable variables live in regular memory, too, so can’t take up more than 4 GiB either. They just happen to be saved to stable memory during an upgrade.

We are working on providing a Motoko API for direct access to stable memory. Until that is available, Motoko cannot make use of the larger stable memory.

2 Likes

Also note that Motoko internally uses the pre and post upgrade hooks to do this copying, and you will very likely hit the cycle limit of that long before you hit the storage limit. I’m actually surprised this hasn’t bitten anyone more seriously so far…

@rossberg it sounds like my original answer is accurate but potentially misleading because Motoko cannot yet take advantage of the stable memory. I have seen demos of access stable memory, so I assume those are all in Rust? Is that a more helpful description I can give for the future?

1 Like

OMG, if I have used userMap to store a lot of user data, and this data is still increasing, how can I upgrade?

Hey rossberg, if I use TrieMap to replace HashMap, my userMap will have 8GB of stable storage, so I don’t need to use preupgrade and postupgrade to upgrade, so it won’t exceed the 4GB limit of Wasm heap, right?

@diegop, yes, your answer is correct. Plus what @nomeata said.

As for an outlook, I would say: We are working on adopting the IC’s new 64 bit stable memory and providing a Motoko API (almost done :wink: ) for direct, unlimited access to stable memory. With that, it will be possible to design and implement stable data structures that avoid copying for upgrades. But until that is released, Motoko cannot make use of large memory.

@HelloRickey, unfortunately no. As I said, stable vars still live in regular Wasm memory. The only advantage you get is that you will not need to write pre/postupgrade hooks, because the Motoko compiler does that for you. To actually use unlimited stable memory, you’ll need the upcoming API that I just mentioned – or a new library implementing a map on top of that.

1 Like

@rossberg, what are the chances to get stable-memory-backed values that are not in the main memory in between messages? I.e. that if one defines stable var myState = … that the data therein is backed by stable memory, and read on demand?

I know it’s not trivial (some form of fat pointers; generated code might have to dynamically handle objects in either place etc; a garbage collector for the main memory), but probably doable, and ultimately I think maybe the only way forward? (The current scheme doesn’t scale; raw memory access is too low level for the Motoko target audience)

2 Likes

@nomeata, the idea is that the raw memory API enables us (or inspired devs) to implement things like that as a library abstraction, in Motoko itself. I agree that most users wouldn’t want to go there themselves.

If it works as a library abstraction, sure. But really transparent support, where any (shared) data can live here or there, and GC takes care of it, probably needs to be built in.

Or put differently: can we achieve the current semantics of the stable annotation with an implementation that scales (in particular no use of the dangerous pre_upgrade hook, and in general per-message cycle consumption that depends on the work done, independent of the total size of data)?