[RFC] Canister Metadata Standard

Objective

The goal of developing Canister Metadata Standard is to improve the experience of integrating third party canisters. A service provider make its canister compliant with the standard. Then a consumer can easily pull the canister from the mainnet and set it up locally.

Background

What is Canister Metadata?

Canister is simply a Wasm module. Developer can add arbitrary data to a custom section of the Wasm module. Then the data can be fetched from the metadata path in system state tree. For more details, please check this section in IC specification.

Why we need a standard?

We plan to provide a complete workflow of third party canister integration for both service provider and consumer. Canister metadata is suitable for communication between the two parties.

The consumer can understand the provider only if they speak “the same language” which is the standard we are proposing.

How will the standard be executed?

In short, dfx will handle it.

A service provider explicitly opt in this standard by declaring necessary metadata value in dfx.json. Then dfx will write metadata in the standard format into Wasm module custom section. When install canister, dfx will check if the metadata are complaint with the standard in case the developer modify metadata outside dfx.

A service consumer, declare dependency canister ID in dfx.json, then dfx will request the metadata on-chain and setup the dependency canister locally.

Proposal

Before the standard details, I want to state several principles for it:

  1. Less is more
  2. No obvious interference with other metadata use cases
  3. Consider future evolvement

The canister metadata can be viewed as key-value pairs. For simplicity, we only allow valid UTF-8 text in both key and pair.

To avoid name conflicts, we prefix the keys with dfx:. And all metadata in this standard must have public visibility so that everyone can fetch them.

dfx:wasm_url

A URL to download canister Wasm module which will be deployed locally.

Service provider is responsible for hosting the resource online. When consumer pull the canister, dfx will check if downloaded wasm hash match on-chain version. If not, a warning message will be shown.

We expect that it will be possible to download canister Wasm directly from IC itself. Then this metadata won’t be required.

dfx:deps

A list of name:ID pairs of direct dependencies separated by semicolon. This metadata helps resolve transitive dependencies recursively.

For example, if the canister directly depends on:

  • ledger canister with mainnet ID bbbbb-bb;
  • dex canister with mainnet ID ccccc-cc;

Then the value should be:
ledger:bbbbb-bb;dex:ccccc-cc;

dfx:init

A message to guide consumers how to initialize this canister.

If it is simple to initialize, the message can be just the command to be executed. If it is complicated, the message can be a URL point to the page with detailed initialization process.

candid:service

The canister interface in candid.

dfx has supported automatically adding candid:service metadata. If service provider opt in this standard, this metadata is required to be public.

What next?

We look forward to any comments, questions or feedbacks from the community.

Please note that we are not proposing any new feature to Internet Computer itself. Instead, we are developing a recommended standard for the applications (canisters) on IC. And dfx will only write these metadata if the provider opt in this approach.

We will stick to current design of the framework. Any suggestion on the details are welcome.

After a short RFC period, we will put this standard in sdk repo and start implementation soon.

dfx.json preview

This is just a demo for how would provider/consumer write dfx.json to interact with canister metadata. It is not a part of the standard.

Provider:

{
  "canisters": {
    "provider_app": {
	  ...,
	  "pull": {
	    "wasm_url": "https://example.com/app.wasm.gz",
        "init": "https://example.com/readme.md"
      },
      "dependencies": [
        "ledger", "dex"
      ]
    },
    "ledger": ...,
    "dex": ...,
  }
}

Consumer:

{
  "canisters": {
    "app": {
      ...,
      "dependencies": [
        "provider_app"
      ]
    },
    "provider_app": {
      type: "pull",
      id: "ddddd-dd",
    }
  }
}
8 Likes

Thank you @lwshang.

Given that devs are not dependent on dfx to set metadata but can also use tools like ic-wasm to add custom sections, this would benefit from a public registry to register/announce namespaces.

3 Likes

Demergent Labs is very interested in this.

I would like to propose a cdk key-value pair. To keep it simple and avoid possible security implications with revealing too much information (such as a version), cdk could simply be the name of the CDK.

Or, cdk could be a prefix like follows:

cdk:name: The name of the CDK
cdk:version: The version of the CDK

One key use-case that we have in mind is for canister indexing services like icscan or the IC dashboard to allow totals and filtering of canisters by cdk name. Then we could know exact numbers of Rust, Motoko, Python, TypeScript, C++, etc canisters on the IC.

1 Like

@lwshang Are there any news about this and the integration in dfx?

We revisited this in a twitter thread as a way to link to the source code from a canister.

2 Likes

I would like to propose the following (updating my thoughts from a couple comments back):

cdk:name

This is the name of the CDK that was used to produce the canister’s Wasm module binary e.g. motoko, cdk-rs, azle, kybra, icpp.

cdk:languages

This is a list of strings indicating the source code languages that the CDK supports e.g. motoko, rust, typescript;javascript, python, c++

cdk:version

This is the version of the CDK that was used to produce the canister’s Wasm module binary e.g. 1.2.3, 0.16.2, 2.3

2 Likes

Any updates on pushing these standards forward?

Should these be ICRC standards?

Do we need a standard? AI will watch us all writing our code and tab completion will steer us towards a convention. :wink:

On a more serious note, maybe it just needs the cdk developers to reach a consensus on the cdk: metadata namespace. So, you, the motoko team and so on. Is there a IC cdk developers forum somewhere?

2 Likes

Personally I would be interested in expanding the detail of the init metadata. For me, the init args consist of feature flags and canister IDs. Can we have a convention by which init args are tuples and the second entry in the tuple is a map from canister names to IDs? That way the init argument can be made for all the canisters I work with in a standardized way. I am aware that this makes the rather crude assumption that for every name, there is just one canister ID, and that assumption doesn’t scale well. But this is a start.

Would love to see support for cdk:name

2 Likes

Another thing that would be nice to standardize: infrastructure:canister_name and infrastructure:path

E.g. if I have an app with 3 canisters, and a couple of deployments of this app, I would have 6 canisters in total, with deployment and names that might be:

  • name=my-dapp and path=test_deployment/my-dapp
  • name=backend and path=test_deployment/my-backend
  • name=my-archive and path=test_deployment/my-archive[0]
  • name=my-dapp and path=prod/my-dapp
  • name=my-backend and path=prod/my-backend
  • name=my-archive and path=prod/my-archive[0]

Regarding name: A key motivation here is to reduce the risk of deploying the wrong wasm to a canister. If it is really easy to get names, we reduce the risk of error. We might even have a pre-upgrade check that refuses an upgrade request if names don’t match.

Regarding ID: The motivation here is to have a structured way of distinguishing identical canisters, for the cases when there are multiple deployments of the same canister for any reason. Given two canisters with the same name, the deployment path, like a terraform path, makes it clear how that canister fits in to the rest of the infrastructure.

Regarding both: It would be nice to get feedback from infra people, who have a lot of experience managing large terraform, ansible, cloudformation and other deployments. Not everything from the centralized world will apply to blockchain but I expect that some lessons will be transferrable.

1 Like

For metadata to be useful, it helps for it to be widespread, and for it to become widespread, it helps for it to be there by default, and for a standard to be there by default it helps to have it in tooling, and tooling is built on standards.

Some things like cdk:name and deployment:name are information that tools such as dfx have already, so it feels as if the work needed to go from idea to widespread use is quite small.

2 Likes

@lastmjs hi, I would like to count Azle/Kybra canisters too on https://icp.zone (forum post)

Currently I check if motoko:compiler custom section is presented(although it is private).

What if Azle/Kybra will fill azle:compiler and kybra:compiler custom sections until cdk:name is standardized?

2 Likes

Is there a problem with just checking cdk:name even before it is standardized? Seems like it shouldn’t be too complicated?

I’m currently waiting for dfx extensions to be implemented and testable before really putting the metadata infrastructure in place.

2 Likes

This seems very exciting though, if we put the metadata in place for Azle and Kybra, how much effort/how long before icp.zone could pick everything up?

Yes, that’s even better. Will require only one request instead of many for each cdk

I don’t think it should be difficult, a few days maybe…

1 Like

Great, I’ll let you know then

1 Like

Same for icpp, the C++ CDK.
We have dfx integrated now as a default post processing step during a build and can use it too for creating the custom section.

3 Likes

I would love to count canisters on different languages to show on icp.zone.

Any progress on supporting cdk:name in Azle/Kybra @lastmjs?

In C++ CDK @icpp?

Would helpful if Dfinity also will add support for this metadata section for Motoko/Rust canisters too(idk who to mention @domwoe @Severin)
For now I make distinction between rust and motoko canisters by trying to get motoko:compiler section, which is hacky.

Also, this section should be filled for dynamically spawned canisters from the parent canister, right?

1 Like