Two questions about canister storage

@diegop, yes, your answer is correct. Plus what @nomeata said.

As for an outlook, I would say: We are working on adopting the IC’s new 64 bit stable memory and providing a Motoko API (almost done :wink: ) for direct, unlimited access to stable memory. With that, it will be possible to design and implement stable data structures that avoid copying for upgrades. But until that is released, Motoko cannot make use of large memory.

@HelloRickey, unfortunately no. As I said, stable vars still live in regular Wasm memory. The only advantage you get is that you will not need to write pre/postupgrade hooks, because the Motoko compiler does that for you. To actually use unlimited stable memory, you’ll need the upcoming API that I just mentioned – or a new library implementing a map on top of that.

2 Likes

@rossberg, what are the chances to get stable-memory-backed values that are not in the main memory in between messages? I.e. that if one defines stable var myState = … that the data therein is backed by stable memory, and read on demand?

I know it’s not trivial (some form of fat pointers; generated code might have to dynamically handle objects in either place etc; a garbage collector for the main memory), but probably doable, and ultimately I think maybe the only way forward? (The current scheme doesn’t scale; raw memory access is too low level for the Motoko target audience)

3 Likes

@nomeata, the idea is that the raw memory API enables us (or inspired devs) to implement things like that as a library abstraction, in Motoko itself. I agree that most users wouldn’t want to go there themselves.

If it works as a library abstraction, sure. But really transparent support, where any (shared) data can live here or there, and GC takes care of it, probably needs to be built in.

Or put differently: can we achieve the current semantics of the stable annotation with an implementation that scales (in particular no use of the dangerous pre_upgrade hook, and in general per-message cycle consumption that depends on the work done, independent of the total size of data)?

1 Like

When will 8 GB become 300 GB?

My understanding is that the proposal that passed was to increase stable memory to 300 GB, and that 8 GB was just a temporary stopgap.

2 Likes

It definitely is intended to keep growing as issues and bottlenecks are identified and removed at each stage. I will let @akhilesh.singhania lend some more clarity

1 Like

Thanks for the ping @diegop .

@jzxchiang : after increasing to 8GiB, as expected, we uncovered some performance bottlenecks that the team is currently working on addressing. One of the key bottlenecks that we discovered is discussed in more detail in Quick Fix for Smart Contract Memory Bottleneck | E007 - YouTube. The work here is not completely done yet.

In general, the problem is that when canisters produce too large a heap delta, they put a lot of load on the node’s memory system. Previously this was slowing down the entire subnet. @abk then worked on an optimisation where we track the heap delta per canister. This allowed us to mitigate the problem so that we only slow down the offending canister. Next @ulan and Adam are working on a file backed PageMap allocator so that we can further reduce the load on the memory system.

Note that in tandem they are also working on the canister sandboxing feature so progress here is a bit slow. Once, their PageMap allocator work is done, barring any other critical performance bottlenecks are discovered, we should be in a good place to further increase the canister capacity.

8 Likes

Interesting, thanks for the response.

I’m curious how wasmtime handles heap deltas for linear 4 GB memory? Is a large heap delta not an issue there because wasmtime is smart about memory allocation or because 4 GB is not big enough where it’s a problem?

@jzxchiang, maybe I misunderstand your question, but wasmtime is a plain Wasm engine and as such neither implements persistence nor consensus, so “heap deltas” are simply not a notion it has to bother computing.

1 Like

Hm yeah, that makes sense. Heap delta is just the act of persisting memory pages from the replica process heap (and not the wasmtime or canister process heap) to a checkpoint file on disk, right?

Precisely. You only have to worry about the deltas if you are planning on implementing something like orthogonal persistence or query executions where you need to be able to roll back the changes or execute messages against a different version of the state.

1 Like

Just for clarification…

To store a bunch of pictures, let’s say 1 MB per picture with the following methodology:
private stable var stable_pictures : [(Text, [Nat8])] = [ ];

and linking it with this in the pre/post loading methods
private var pictures = HashMap.HashMap<Text, [Nat8]>(0, Text.equal, Text.hash);

How many pictures would a canister get me around?
And would I run into any upgrading issues?

I am also wondering about real world example (whether theoretic or otherwise) of the current canister storage limitations to get a better idea about how much data I could use in the canisters in total for everything?

Also, would love to see example of storing actual image into the canister? I can see you would be using an array of Text for key and Nat8 array as value? How would you store actual image into [Nat8] ?

Hey there…

Don’t use Nat8 Array use a Blob to store a pic. And I also used ExperimentalStableMemory to store them. Currently I think its possible to store about 8GB. Works really smooth.

1 Like

Can you explain to me how do you send the image as blob to the canister to store it there? I would like to see an example of this?

2 Likes

thats a typescript example

public async sendPic(pic: Uint8Array) {
    const picArr: Array<number> = [...pic];

    let icActor = await this.getActor();

    return icActor.sendPic(picArr)
  }

and expose the sendPic method on motoko side with a blob as input parameter.

1 Like

Thanks for the code snippet :slight_smile: Maybe I wasn’t clear enough in my question above, but what is confusing to me is how to present image as Uint8Array type ?

What value of Uint8Array type would a google logo have for example (if that makes more sense as a question) ?

Would this be the way to do it in Javascript (convert image to Uint8Array) ? arrays - Image to Uint8Array in Javascript - Stack Overflow

1 Like

Well the Blob or Uint8Array are just binary representations of an image. There are several ways in JS on how to convert it. But your link perfectly shows how to convert a JPG to an img.

1 Like

The reason you want Blob over [Nat8] is described here. Basically, Blob takes up less space when stored in a canister’s memory. (Both are serialized to the same format over the wire though… not sure if that includes stable serialization though.)

1 Like

FWIW, this is the code I use to upload a blob (here, it’s an image):

async function uploadAsset(
  assetId: string,
  path: string,
  width: number,
  height: number,
) {
  const buffer = await Fs.readFile(path);
  const assetBytes = Array.from(Uint8Array.from(buffer));

  const numChunks = Math.ceil(assetBytes.length / Globals.MAX_ASSET_CHUNK_SIZE);

  const putChunkPromises: Promise<null | undefined>[] = [];

  for (let chunkIndex = 0; chunkIndex < numChunks; chunkIndex++) {
    const byteStart = chunkIndex * Globals.MAX_ASSET_CHUNK_SIZE;
    const chunkData = assetBytes.slice(
      byteStart,
      byteStart + Globals.MAX_ASSET_CHUNK_SIZE,
    );

    if (chunkIndex === 0) {
      await Actor.createAssetWithFirstChunk({
        assetId,
        numChunks: BigInt(numChunks),
        firstChunk: chunkData,
        width: Math.round(width),
        height: Math.round(height),
      });
    } else {
      putChunkPromises.push(Actor.addChunk(assetId, chunkIndex, chunkData));
    }
  }

  return Promise.all(putChunkPromises);
}

My Motoko backend exposes two functions, one for creating the asset/image with the initial blob chunk, another for adding additional blob chunks to an existing asset/image.

6 Likes