Two questions about canister storage

Developers
21

Hm yeah, that makes sense. Heap delta is just the act of persisting memory pages from the replica process heap (and not the wasmtime or canister process heap) to a checkpoint file on disk, right?

22

Precisely. You only have to worry about the deltas if you are planning on implementing something like orthogonal persistence or query executions where you need to be able to roll back the changes or execute messages against a different version of the state.

1 Like
23

Just for clarificationā€¦

To store a bunch of pictures, letā€™s say 1 MB per picture with the following methodology:
private stable var stable_pictures : [(Text, [Nat8])] = [ ];

and linking it with this in the pre/post loading methods
private var pictures = HashMap.HashMap<Text, [Nat8]>(0, Text.equal, Text.hash);

How many pictures would a canister get me around?
And would I run into any upgrading issues?

25

I am also wondering about real world example (whether theoretic or otherwise) of the current canister storage limitations to get a better idea about how much data I could use in the canisters in total for everything?

Also, would love to see example of storing actual image into the canister? I can see you would be using an array of Text for key and Nat8 array as value? How would you store actual image into [Nat8] ?

26

Hey thereā€¦

Donā€™t use Nat8 Array use a Blob to store a pic. And I also used ExperimentalStableMemory to store them. Currently I think its possible to store about 8GB. Works really smooth.

1 Like
27

Can you explain to me how do you send the image as blob to the canister to store it there? I would like to see an example of this?

2 Likes
28

thats a typescript example

public async sendPic(pic: Uint8Array) {
    const picArr: Array<number> = [...pic];

    let icActor = await this.getActor();

    return icActor.sendPic(picArr)
  }

and expose the sendPic method on motoko side with a blob as input parameter.

1 Like
29

Thanks for the code snippet :slight_smile: Maybe I wasnā€™t clear enough in my question above, but what is confusing to me is how to present image as Uint8Array type ?

What value of Uint8Array type would a google logo have for example (if that makes more sense as a question) ?

Would this be the way to do it in Javascript (convert image to Uint8Array) ? arrays - Image to Uint8Array in Javascript - Stack Overflow

1 Like
30

Well the Blob or Uint8Array are just binary representations of an image. There are several ways in JS on how to convert it. But your link perfectly shows how to convert a JPG to an img.

1 Like
31

The reason you want Blob over [Nat8] is described here. Basically, Blob takes up less space when stored in a canisterā€™s memory. (Both are serialized to the same format over the wire thoughā€¦ not sure if that includes stable serialization though.)

1 Like
32

FWIW, this is the code I use to upload a blob (here, itā€™s an image):

async function uploadAsset(
  assetId: string,
  path: string,
  width: number,
  height: number,
) {
  const buffer = await Fs.readFile(path);
  const assetBytes = Array.from(Uint8Array.from(buffer));

  const numChunks = Math.ceil(assetBytes.length / Globals.MAX_ASSET_CHUNK_SIZE);

  const putChunkPromises: Promise<null | undefined>[] = [];

  for (let chunkIndex = 0; chunkIndex < numChunks; chunkIndex++) {
    const byteStart = chunkIndex * Globals.MAX_ASSET_CHUNK_SIZE;
    const chunkData = assetBytes.slice(
      byteStart,
      byteStart + Globals.MAX_ASSET_CHUNK_SIZE,
    );

    if (chunkIndex === 0) {
      await Actor.createAssetWithFirstChunk({
        assetId,
        numChunks: BigInt(numChunks),
        firstChunk: chunkData,
        width: Math.round(width),
        height: Math.round(height),
      });
    } else {
      putChunkPromises.push(Actor.addChunk(assetId, chunkIndex, chunkData));
    }
  }

  return Promise.all(putChunkPromises);
}

My Motoko backend exposes two functions, one for creating the asset/image with the initial blob chunk, another for adding additional blob chunks to an existing asset/image.

6 Likes