Diode Message Storage coming to Difinity

Developers Getting Started
1

Hey there,

this is Dominic from diode.io team. We’re starting a three month sprint here to use the internet computer for end to end encrypted chat message relaying using canister logic. The Diode App (Download Hub | Diode™) already features e2e messaging but so far has been relying on peers to be online for the messages to be delivered. Instead of adding centralized servers caching messages we have been looking at various alternatives. Finally we the ability to store large amounts of data (32gb) and bundle them with logic decentralized on the ICP convinced us that this might be the right tool for the job. We’re going to use the internet computers canisters and large available storage to hold the encrypted messages and make them thus always available even if no peer is online.

As we’re freshly getting into the ICP there are a couple of beginner questions for us to solve coming from Solidity. Any pointers would be helpful as we’re starting to develop the canister code:

  1. Is the motoko playground here the standard equivalent of remix? Is there any newer or more idiomatic version we should be using instead? Is it a good starting point at all?
  2. To ensure the canister is only used by and pays for “in-group” interactions we would like to enable a permission checking scheme on the canisters methods using secp256k1 signatures. What are the go-to packages for doing this, any libraries or even group access frameworks we should be looking at?
  3. Last but not least as we want to use the canisters for message storage and I’ve read about Motoko limitations to access stable memory here: 32 GB Canister Storage - #5 by tomijaga — Should we use Motoko at all or start immediately with rust? Is it possible to mix? Again any pointers would be helpful

We will work our way through these and more questions here this month and publish our progress as well as findings / decisions on these questions for the rest of the community.

Happy to be here
Cheers!

5 Likes
2

  1. The playground is just an online IDE for quick examples and demos on Motoko, it’s not a framework like Remix. Right now there are 3 main language options with an SDK: Motoko, Rust and Typescript (Azle).

  2. There’s no need to check any signatures to differentiate callers, instead check the principal of the caller with e.g. the caller() method. This principal is basically the id (hash) of the public key that signed the incoming message. When you look for “securing methods” in either the Rust, Mokoto or Azle examples you’ll find how this is done, it’s quite straightforward.

  3. No idea if Motoko is still more limited, all I know is that the stable storage limit nowadays is more than 32gb, north of 200gb, basically a lot.

2 Likes
3

Have you considered using TypeScript/JavaScript? My company is developing the Azle CDK: The Azle Book (Beta) - The Azle Book

It’s in beta, has great access to Stable Structures, can even do Express/Nest to act as an http server, has access to various (not all) npm packages.

3 Likes
4

Thanks for all the pointers!

  1. We started out with the simple canister log example and then switched from the included Makefile to using mops for testing and quick turn-around during development. Had to learn about mops pocket-ic integration in order to make the test cycles acceptably fast. (One call to mops test with an actor test take >20 seconds with the default dfx replica, but <5s with pocket-ic)

  2. For the secp256k1 validation, we found that this is implicitly supported and exposed in Motoko as the calling principal. We had figure out the calling and signature conventions to figure out how to sign the request messages here: The Internet Computer Interface Specification | Internet Computer – most surprising was that the sender_pubkey field reference was expecting the DER encoded version of the key.

  3. Last but not least we found that the Motoko Region API is doing what we need: Region | Internet Computer - although it comes with a surprising 8mb pre-allocation for every new Region. Which caused us to change our architecture mid-development. Also for the future there is the Enhanced orthogonal persistence | Internet Computer coming which would be exciting to use as it might make the design even easier.

Our code so far is here:

It think it could have saved us a significant amount of time if the example that we started out with canister log example would have had mops integration and pocket-ic in the mops.toml already.

We will publish more as we come along including the elixir client for the icp that we’ve been using.

Cheers!

3 Likes
5

Thanks for the feedback! We’re considering ways to update the example projects, and it’s helpful to know you’d have preferred having mops and pocket-ic integrated

3 Likes
6

Hey @dominicletz, have you also considered integrating IC WebSockets into your new solution?

Not sure if you strictly need it, as it looks like you’re using the IC just as a “storage” for the messages for the users to download them when needed, rather than a real-time message relay. In case you need it though, I’d be interested in knowing if you have considered using IC WebSockets and, eventually, what were the blockers that you faced, if any.

We also have a pretty straightforward tutorial with the link to the example source code, if you’re interested.

7

Hey great question @ilbert and I agree the IC not having a native websocket interface for this too bad. That said in our case we already had a native peer-to-peer push implemented before adding the canisters. So if two peers are online at the same time the push notification about a new message is actually traveling in real-time peer-to-peer. At least for us it’s enough when the canister is available when a notification comes in or when the user just starts the app initially and the first message check is done.

2 Likes
8

We just announced going live into production with the new zone canisters providing the encrypted message storage here: Diode ICP Zone Canisters going live

Diving into this technically the biggest lift here was to setup the right deployment strategy. After reading up here on the forum and some of the great work from the Cycleops people we decided to follow their Battery->Children pattern:

image
image875×363 14.7 KB

Again source code and architecture details are all available here:

The Good, (the open), the bad, the ugly

  • The factory pattern works great. Our app can autonomously create new children
  • Upgrading the children of the factory (and the factory itself) is not straight forward but ended up working great for us.

Open Questions

  • How to monitor a large fleet of canisters (1000s) effectively for resource usage patterns? What are other people using here?
    • Especially tracking stable memory consumption seems impossible from motoko or via status atm.
  • We ended up spawning new canisters with 1 trillion cycles (~1.30 USD) for every single canister - couldn’t make lower values work. This seems bit high for us especially knowing that many of these canisters will be in use only very shortly in cases when users churn out of the app. Can we reduce our losses here or recup the otherwise lost cycles?
  • Our factory is deployed on this subnet - does really mean that all children will be created there as well? That seems like an issue when we are going to create thousands of canisters. Should create multiple factories on different subnets and load balance ourselves or will the IC internals help us out?

Bad

  • It seems Denial-of-service (DOS) attacks on canisters are really easy to do because of the reverse-gas model. At least from motoko it didn’t seem like there is a good way of protecting against those. Specifically even checking for caller permissions costs gas and would be enough to run a DOS
  • There is really little documented information about the internals of the canister wasm upgrade process. We had learn a lot through trial and error, and many details are still unknown/unclear. (E.g. why is a motoko actor class constructor called again when doing mode=#upgrade?)

Ugly

  • Chain Fusion call-outs can timeout and many endpoints do not support IPv6 without additional work. We had to do some investigation and discussion with our endpoint providers to get it supported everywhere we needed.
  • ICP documentation often states “queries a free at the moment” but doesn’t go deeper into this. Were kind of afraid that this might change some day and suddenly render our solution non-viable from a cost perspective
  • Why are cycles pegged against XRP and not getting cheaper as hardware improves? E.g. ICP launched in 2021 and since the FLOPS/dollar has improved 8x but cycles cost stayed the same?
  • I once did a dfx cycles transfer <factory> <amount> instead of dfx cycles top-up <factory> <amount> and it took me quite some time to understand the difference and then be able to transfer them back and convert from “cycles ledger balance at the canister” to “cycles gas balance at the container” or whatever the right wording is for these two states. Also until now it’s unclear to me what magic the icp call dfx topup is doing to convert a cycles ledger balance to a cycles gas balance.
  • In order to use http out-calls it is necessary to supply a shared query as tranform function oracle_transform_function : shared query TransformArgs -> async HttpResponsePayload; – In motoko this kind of function is only possible on an actor (not inside a module) and so we had to expose this function on the most outer layer of our canister while the oracle and http interface logic itself is nicely encapsulated in a module. This felt like a really ugly leak of abstraction just because of a language/interface limitation

The new Elixir ICP Agent

Lastly during this last stage we released a new ICP agent in the Elixir programming language https://elixir-lang.org

You can fetch the new agent at GitHub - diodechain/icp_agent: Elixir ICP agent for interacting with the internet computer

That’s it for today, but feel free to ask any questions. I’m happy to answer here or on discord.

Cheers!

6 Likes
9

Answering your questions where I can

When you do dfx canister delete, dfx first installs a cycles wallet in the to-be-deleted canister. That cycles wallet is then used to extract as many cycles as possible to the user’s cycles storage (cycles wallet or cycles ledger). Only after that is the canister deleted. It’s not possible to extract 100% of the cycles, but if you first set the freezing threshold to 0 you can get out a decent amount

Depends on how you create new canisters. If you call create_canister on the management canister, then all your canisters will stay on the same subnet. If you call create_canister on the CMC, then you have more control. You can either let the CMC choose a random subnet with capacity, or you can specify a subnet that’s accepting new canisters explicitly.

Yes, that’s still not solved. Go read up on inspect_message, it will help mitigate some of the issues. Not all, but it’s a good start.

The documentation states this because it’s the best we can offer at the moment. Yes, queries are free, but it’s not possible to offer something for free forever. But we also don’t know yet what would be a fair price for queries. Once there are concrete plans there will be a lot of discussions here on the forum to discuss specifics

That’s just how it is, and it is nicely straightforward. When hardware improves it is still possible to lower the cycles cost of certain operations. This can still make operations cheaper, even with the fixed conversion rate.

Agreed, it’s not ideal. I was part of the cycles ledger development and we thought a lot about banning canisters from receiving cycles on the cycles ledger completely, but decided against it in the end. If you have any suggestions on how to improve wording I’d really appreciate it.

It uses this function on the cycles ledger

4 Likes
10

Especially tracking stable memory consumption seems impossible from motoko or via status atm.

You should be able to use some functions in the runtime system to do this:

Prim.rts_stable_memory_size: () → Nat
Prim.rts_logical_stable_memory_size: () → Nat

These report the size as number of wasm pages (64KiB).

There’s also a low-level, hidden query method “__motoko_runtime_information” that you can call to get this information (as self or a controller):

This test file show you how to access this programmatically from motoko:

test/run-drun/runtime-info/info.mo

We are planning to add a Runtime.mo library to base to make these functions more accessible.

It seems Denial-of-service (DOS) attacks on canisters are really easy to do because of the reverse-gas model. At least from motoko it didn’t seem like there is a good way of protecting against those. Specifically even checking for caller permissions costs gas and would be enough to run a DOS

To protect against external DOS attacks via ingress messages (calls from outside the network) you can author an “inspect” system method. Guarding against cross canister attacks is indeed not well supported yet.

Maybe we should offer something similar to inspect for inter-canister calls, either specifically just for Motoko or better, at the ICP level.

(E.g. why is a motoko actor class constructor called again when doing mode=#upgrade?)

This is because the fields of the actor can change after upgrade and their initial value can depend on the argument to the class constructor, which can also change on upgrade. So the initializer is always run, on install or upgrade, to initialize the local fields of the actor, apart from any inherited stable variables.

In order to use http out-calls it is necessary to supply a shared query as tranform function oracle_transform_function : shared query TransformArgs -> async HttpResponsePayload; – In motoko this kind of function is only possible on an actor (not inside a module) and so we had to expose this function on the most outer layer of our canister while the oracle and http interface logic itself is nicely encapsulated in a module. This felt like a really ugly leak of abstraction just because of a language/interface limitation

Yes, that really is more of a problem with the design of the management canister, that requires you to provide a public query method for the transform function. You could probably write the body of the query function as an ordinary function in your module, and then call it from the actual query function, to achieve better modularity.

3 Likes