How to store right amount of data in a CanDB partition canister?

Developers Motoko
1

I am thinking about implementing HTTPS secure streaming from a CanDB partition canister. That is a user obtains an access token and then can stream.

My question is how to manage memory of such a canister.

Suppose we don’t want to allocate more than 500MiB of heap memory. If I limit CanDB to 500MiB per canister, I don’t take into account the memory used for storing access tokens.

So, suppose, I allow 400MiB to be used by CanDB. But then if I fill 50MiB by access tokens, it will use only 350MiB what is wrong, because I want it to use 400MiB.

So, how to solve it? Or should I not worry, just limit 1MiB to be used for storing current access tokens?

2

Maybe, it’s a good idea to store access tokens in a separate canister?

Will in this case checking access tokens much slow down streaming? (I know that IC is highly optimized, but I am talking about streaming, a kind of activity that is particularly time-sensitive.)

So, will checking access tokens by a separate canister be fast enough?

3

I did an experiment. greet and greet2 in the main canister work at about the same speed. So, nothing prevents me to allocate a special canister for checking access tokens.

import dep "canister:dep";

`main`:
actor {
  public query func greet(name : Text) : async Text {
    return "Hello, " # name # "!";
  };
  public composite query func greet2(name : Text) : async Text {
    await dep.greet(name);
  };
};

dep:

actor {
  public query func greet(name : Text) : async Text {
    return "Hello, " # name # "!";
  };
};
4

For most ICP applications, several MB should more than sufficient for current access tokens for the time being. I wouldn’t over-complicate and extend your application unless there’s a strong reason to do so. It’s easier to repartition after the fact than to recombine everything into a single canister.

On another note, given the upcoming Motoko Orthogonal Persistence feature, being able to eventually use up to the full canister size with solely heap memory (current canister size limit as of this post is 400GB) will lend itself towards monolithic canister architectures.

Needless to say, unless you have a serious need to start with an auto scaling solution I’d recommend keeping things simple and in a single, or few canister design.