Proposal to Adopt the Namespaced Interfaces Pattern as a Best Practice for IC Developers

by Austin Fatheree(@skilesare) with input from many others in the community. Special thanks to @wpb for laying the groundwork for healthy community debate.

ICDevs.org is launching a new initiative to help coalesce a set of languages around the Internet Computer. These are not programming languages. The “Languages Project” is an attempt by ICDevs.org to contribute to the adoption of the Internet Computer by creating generative, form, and pattern language to the community that provides a common set of communication tools that foster well-formed, productive, interoperable, and collaborative systems for innovation on and around the Internet Computer ecosystem.

These patterns are open-sourced and we would love community contribution. You can provide pull requests to our repo at GitHub - icdevs/Icdevs_fleeksite. and read more about the initiative at https://icdevs.org/language_project/index.html. We are just getting started, so excuse the lack of current content.

The first Pattern in our Pattern Language that we are proposing to the community is a concept called “Namespaced Interfaces”. This topic has been discussed a few times on the forum, specifically here.

This pattern is not a proposal that we can specifically enforce via code, but it is one that we can adopt as a community and all agree to abide by. The text of the pattern is below with more discussion below:


NSP. Namespaced Interfaces

…how do you provide INT - Interoperability across a broad set of interconnected services.

The Internet Computer will eventually host a broad set of interconnected services. Some of these services are predictable(such as the transferring of funds or tokens) while some applications will not be conceived of for decades.

When dealing with a broad set of computing endpoints in a diverse, growing, and dynamic system, it is common to run into confusing naming conflicts.

For example, both a fund transfer interface and a messaging system may have the concept of “send”. If a third, interoperable service is trying to combine the functionality of these two services it will have to make a decision about which “send” functionality it will want to expose and which it will want to abstract. This problem is further exacerbated by the Internet Computer because function addressing indexes only on the function name and not on the candid signature of the parameters as well. This seems to be an intended design decision although other blockchains, and specifically, have chosen to include the parameters in the hashing of an addressable function.

We have one example of this from the early days of the internet computer where an EXT NFT standard was created with a transfer function that takes the form:

type TransferRequest = {
  from : User;
  to : User;
  token : TokenIdentifier;
  amount : Balance;
  memo : ?Memo;
  notify : ?Bool;
  subaccount : ?SubAccount;
};
type TransferResponse = Result<Balance, {
  #Unauthorized: AccountIdentifier;
  #InsufficientBalance;
  #Rejected; //Rejected by canister
  #InvalidToken: TokenIdentifier;
  #CannotNotify: AccountIdentifier;
  #Other : Text;
}>;

transfer: shared (request : TransferRequest) -> async TransferResponse;

And a departure labs token that has the transfer function that looks like:

transfer(to : Principal, id : Text)

The issue arises that if you want to have your NFT service act as an interoperable service and to take advantage of other interoperable services, you are now stuck with a dilemma of picking the ‘right’ or ‘most valuable’ standard. You can either implement the EXT transfer or the Departure transfer but not both.

In order to eliminate this issue, all consumable services should implement namespaced functions. The EXT token should expose the same functionality at com_ext_nft_transfer and departure should expose com_departure_nft_transfer. By doing this an NFT builder can expose both functions and then any service that speaks either EXT or departure can consume, transfer, reference, and amplify the value of the developer’s NFT service.

While these namespaces are not “pretty” they do, if selected well, provide some essential information at the point of invocation at least provide improved context for the trade-off in readability.

We should be clear to define what a “consumable” service is. These are update calls or query calls that you expect external services to use or consume to interact with your service. These are functions at the edge of your system that you expect, want, and encourage 3rd parties to use. You do not need to use namespaces between your systems canisters that implement a larger system. In an ideal world, the signature of these namespaces will not change and they will not ‘go away’. An upgrade of a service should likely add a _v2 or some other version indicator to the function while leaving the former in place with any missing information handled via defaults. For example, com_ext_token_transfer(address, amount) could become com_ext_token_transfer_v2(address, amount, fee) where the initial version implements a default fee for backwards compatibility.

Further, developers should refuse to implement a ‘standard’ if it is not namespaced. Any existing services should be retrofitted with a set of namespaced endpoints and any services using unnamespaced endpoints should be upgraded to use the namespaced endpoints.

The sooner this pattern can be adopted and implemented the better. As more ‘blackeholed’ services are pushed onto the IC this will become a harder and harder pattern to implement.

We include the ICP ledger and NNS in this proposed pattern and suggest that any public functions should be exposed via a namespaced function. For example:

Ledger:

com_icpstandard_send
com_icpstandard_balance
com_icpstandard_get_blocks

NNS:

com_nns_manage_neuron
Etc

By setting an example at the root level, the DFINITY foundation can help extend this pattern throughout the ecosystem.

Therefore: When authoring standards and creation publicly consumable functions on the Internet Computer, use namespaced functions. When consuming services, refuse to use non-namespaced functions. Make and adopt a proposal to add namespaced interfaces to IC management canisters.


We’d like to propose that the community adopt this pattern and that the current dapps out there that are using/building interoperable services take the time now to update their interfaces before we have too many blackholed services that cannot be retrofit.

The main goal here is to create an interoperability environment on the IC that encourages Innovation while also supporting Integration. We may need to produce 15 NFT standards before we find the magic mixture and I’ve run across far too many projects that feel held back by the fear of launching because they don’t yet know what standard will “win out”. Using namespaced interfaces lets everyone run now and backfill later.

Issues for debate:

  1. What form of namespace should we use? We propose com_app_function_version, but we are open to other options as well. For example, would it be helpful to include if the function is a query or update call in some way?

  2. Should DFINITY retrofit the existing canisters(ledger, governance, etc) with namespaced interfaces that pass the call through to the current functions? Keep in mind that the initial function would stay in place, so nothing would break. Instead, we’d ask that docs, tutorials, and other material be updated with references to the proper functions to use. We propose that they lead and make these changes in the next update of system canisters.

A time period of one week will be provided for deliberation on this proposal in the forum. The final days will be used to wait for quiet on the deliberation. If new, insightful, and actionable comments continue in those final days, then the deliberation period will be extended. Otherwise, seven days after making the forum post the formal NNS proposal will be made on this topic. The exact content of that proposal will be shaped by the ideas presented in this article as well as actionable feedback that occurs during deliberation.

No matter how this vote turns out, our hope is that this proposal generates a high level of participation among IC stakeholders, translating into significant active voter turnout, and ultimately resulting in the IC community collectively learning something useful about decentralized governance on the IC. Please encourage participation in this process. Please vote to Approve or Reject according to what you believe is best for long-term governance of the IC. Please add your comments to this forum post so others will know your opinion. Thank you for your participation.

9 Likes

I guess the reason for this is that the IC distinguishes functions by name only.

I wonder if this can be solved by some sort of ENS on IC, where canisters can register for a human-readable, naturally-namespaced domain name from some registry canister? I suppose there would also need to be some underlying support from the IC system for this to work.

But that seems much more scalable than hoping that every developer renames their functions (which doesn’t seem enforceable anyways).

2 Likes

A good point to emphasize here is that this is about standards and the expectation of standards is that they will be used from a wide array of canisters with different domain names/canister IDs. The classic example here is the ERC20 standard on Ethereum. When they published the standard it instructed other devs to use functions named balanceOf, transfer, transferFrom, approve, etc and in return for using the standard you get “free” integration with a wide array of existing infrastructure. Lots of contracts on ethereum use this standard and as a result, it is super simple to add your token to metamask and other wallets, they just need to know the address and then they can 'speak the language of any token using that standard.

I believe DAB has been doing some of what you suggest by wiring up slightly different standards and mapping the translation of one kind of transfer call to another. Architecturally we probably don’t want to have to be routing our calls through a translations filter(two extra consensus cycles, ~4-5 seconds) and that won’t work for query calls right now anyway.

We could also suggest a major change to the IC and ask that the system index on Name + Candid signature, but that would likely break quite a bit and I think that ship has probably sailed for now.

1 Like

It reminds me of similar discussions we had within DFINITY, and yes, this unaesthetic namespace prefix in the method name was the least bad solution we found.

Maybe something prettier than a full domain could be used? Create a simple canister where you can register prefixes? Or leave it free-form and hope that clashes won’t happen? (But that seems not good enough)

One alternative in principle is to have proxying helper canisters if one has to provide incompatible interfaces, or possible (extending the system) a scheme where a canister is reachable under multiple IDs (like vhosts). But unfortunately such helper canisters don’t always work transparently (e.g. queries, although that lack of compositionality is in a way a telltale sign that our application model isn’t that great yet), and I am not sure the complexity of adding “virtual canister ids” will be worth it.

If only we had a capability-based model :slight_smile:

4 Likes

I’ve created a pull request for the EXT token standard to show what it would look like and what this would mean for developers:

A few notes from along the way:

  1. Perhaps we want namepspaced extensions? And a system like __extentions call?
  2. Candid makes typing fairly easy and it is nice that we don’t need to namespace types. This is enabled because ultimately all types must be stable and they distill down to base times at compile time.
  3. com_ is probably too much. We should simplify that. I just used ext_ for the namespace
  4. Code duplication is annoying, but a “feature” of motoko. See this playground for why we can’t just pass through functions from one top-level function to another: Motoko Playground - DFINITY. You also can’t passthrough query called from a namespace function to a legacy function because queries can’t call await. A more nuanced approach to encapsulation is necessary when you do your upgrade.
1 Like

This reminds me of something I had to deal with years ago with Objective-C where people employed a similar solution.

All classes in an Objective-C application must be globally unique. Since many different frameworks are likely have some conceptual overlap—and therefore an overlap in names (users, views, requests / responses, etc.)—convention dictates that class names use 2 or 3 letter prefix.

Even then, because this was a convention and not enforceable, Apple released new frameworks which conflicted with existing 3rd-party ones:

1 Like

Perhaps a better solution would be for Candid to allow duplicate names and disambiguate by type.

1 Like

@chenyan, is that feasible?

Maybe I’m missing something, but can’t we use variant type instead of namespaced-function names? In the transfer example, the proxy function can take variant { ext_nft: TransferRequest; departure_nft: (Principal, Text) }, then we can pattern match on the variant tag to call the corresponding service?

2 Likes

You could do this but then you are going to have to keep all users of your service in sync with the types that you use. If you update your type with a new service everyone will have to update their code to use the new types or face compile-time headaches. With namespaces, the services can add at their leisure and the type signatures shouldn’t change.

1 Like

If you update your type with a new service everyone will have to update their code to use the new types

No, with subtyping, none of the client code needs to update the type. For example, if the client only calls the ext_nft service, they can always send variant { ext_nft: TransferRequest } type to the proxy. With subtyping, the proxy can decode the value to variant { ext_nft: TransferRequest; nft2; nft3 } just fine. Adding new variant tag or changing the type of nft2 and nft3 doesn’t require update types in the client service that only uses ext_nft.

1 Like

The hope is not all lost yet. I’ve not been keeping track of host references in Wasm, but I think now as an external community member, you may have a better chance of making the capability proposal :slight_smile: @nomeata (as you can clearly tell it was not part of the 25 DFINITY proposals)

4 Likes

That is good to know! So I guess one option here is to make every public interface only have one variant parameter. How does that play? It could certainly simplify things!

1 Like

I think none of the public interface needs to change. Only the proxy service needs to use the variant type, and anyone calling the proxy service will be using the subset of the variant tags they are interested in. For example, if the client only uses the ext_nft, they can call proxy_canister.transfer(variant { ext_nft = ... }) without worrying about if the proxy_canister adds more variant tags or not.

1 Like

Hmm…But the whole point of having a public interface in a Standard is so that Clients know what to expect and all you have to change is the Address you are pointing to and it all “just works”. If a canister has to know that you are a “proxy canister” then interoperability is significantly more complex. I now need to tell you if the address I’m providing you is an DIP-20 contract or a DIP-20 Proxy canister.

From an integration standpoint, it is easier for me to tell you it is a DIP-20 address and for you to know that you should be calling dip20_transfer(). But this takes discipline from the standard developers.

I guess there is a generic function public interface that could be created for each canister with the signature:
call( {function: text; parameters: variant). That would abstract everything away but make for a big nasty nested switch statement.

1 Like

I see. Then it makes sense to add opt variant { my_nft } for each public method. If someone is sending variant { another_nft }, we get null instead of error on deserialization. The proxy canister will try to collect all different variant tags and do the proper dispatch.

I think variant tag and namespaced functions are essentially the same thing, but with variant, we get some type level guarantee, whereas with naming convention, you cannot really enforce it.

2 Likes

My understanding is that unambiguous naming is needed (only) for defining service interfaces that third-party services are supposed to implement (e.g., to support higher-order uses). Because if a given service wants to implement multiple different interfaces, that won’t work if there is a name clash between them. Hence, methods in such interfaces need to be distinguishable somehow.

This use case cannot be addressed with variants, I think. So the proposal makes sense to me.

@paulyoung, Candid cannot support overloading, because the IC doesn’t. And even if it could, it would be inadequate for multiple reasons:

  1. There is no reason to assume that all name clashes in the above scenario happen to have disjoint types, so it would not be a sufficient solution anyway.

  2. Overloading tends to be complex and difficult to specify. The presence of rich subtyping rules on arguments makes that much worse and generally ambiguous.

  3. It would interact badly with the upgradability mechanism for Candid interfaces, since a type refinement to a method might then turn an unambiguous overload downstream into an ambiguous overload, or worse, route unaware message sends to a different method unexpectedly.

In short, overloading is messy and error-prone. It would not be a good idea to introduce it in a fluid layer like this.

3 Likes

After having personally struggled with this I’d like to put forward the following official proposals for discussion. A more formal representation will be selected once we decide which one to move forward with.

Proposal A Require DFINITY to add a hash of the a standardized representation candid parameters of a function to the function signature to support function overloading.

This would allow a canister to support both .transfer(Principal, amount) (DIP20) and .transfer(TransferEventArgs) (ICP Ledger) and other similar namespace collisions. The one case this would not cover would be exact parameter matches from different standards. IE. A canister wants to support notifications from both a Ping service and a Broadcast Standard. Ping.notify(Text) tells a canister that that text is still active. Broadcast.notify(Text) broadcasts the text to a set of child canisters. Under A1 you could not integrate these two standards on the same canister.

Proposal B DFINITY will namespace all ledger functions with _ledger. DIP20 will add _dip20 to all functions, EXT will add _EXT to all functions, IS20 will add _is20, Dfinity Fungible standard will and _dfs. They shall do so or be mercilessly mocked and shamed. Future standard creators will be challenged to a dual by a randomly selected NNS participant if they should release a standard that is not namespaced. The NNS selectee will choose the weapon.

DFX will be updated to warn any user deploying a function without a _ in it that they may be mercilessly mocked and challenged to a dual and that they should only continue if it is a private canister that the public will never consider calling, and even then they should reconsider because they may have stumbled onto an amazing pattern that the border community may want to adopt…better safe than sorry. A link to documentation, justification, and best practices of sharing a global namespace environment should be provided.

Existing standards may keep existing endpoints for backward compatibility, but new namespaces functions should be created and the legacy functions should be removed from documentation. Failure to remove the legacy functions from documentation will result in taunting a second time. Namespacing fixes all namespace collision issues unless you are being deliberately hardheaded.

Proposal C Make no changes to the way standards handle namespace collisions and let DeFi flounder on the IC until a default standard emerges and everyone has to refactor their code or can finally move forward with the confidence that they aren’t wasting their time on a standard that could be replaced.


Note: Proposal B is the right solution even if is less enforceable. We need to police ourselves.

Note 2: Having one standard is great(unless it can’t evolve with the tech and we’ve only scratched the surface of what the IC can do). Having no standard is bad. Having a few standards that don’t collide is fine. Having conflicting standards is a nightmare.

Note 3: I’m open to more serious takes on B…I’m felling chippy after a couple months of trying to reconcile DIP20 and the standard ledger.

4 Likes

(A) does not work and never will, for the reasons explained in my previous post. I think we can make at least some progress by taking it off the table.

(B) sounds right, except that I still don’t understand the reason to use or even enforce name spacing everywhere. A canister already is its own name space. When exactly is cross-canister name spacing relevant, except where some form of informal “implements” relation has to be shared across unrelated canisters? The OP talks about “combining” services in an “interoperable” manner, but falls a bit short of explaining the scenario and assumed mechanism for such a combination. So I feel like I’m still missing a more precise problem analysis identifying concrete use cases to develop an informed opinion.

1 Like

Does a higher-order interface not work? For example, a method getDIP20 could return a record of methods that implement DIP20, and so on.