In the IC ecosystem, we constantly keep inventing new esoteric protocols/tooling with their own specialized syntax and rules that developers have to learn and completely bypassing industry standards that are already well established and have a large community around them. This is a huge barrier to entry for new developers and a HUGE productivity sink for existing developers
Exhibit A - Motoko vs Rust/TypeScript/Python
None of the significant canister code bases written/maintained by Dfinity use Motoko. All of them use Rust
Learning curve has been cited as another reason for the existence of Motoko. Most Rust devs disagree but for the sake of argument letās consider that to be true. Use TypeScript/Azle in that case. No other language is more prevalent/popular/well-known compared to JavaScript if learning curve is a concern
Not going to rant about the struggles with Motoko here. Already posted about it elsewhere
Exhibit B - IC System API vs WASI
IC invented the System API to provide access to OS exposed features like the clock, file system, randomness, etc. via the System API when it should just use the industry standard which is WASI
A wrapper for WASI support is being developed by an ecosystem builder with a grant when it should ideally have first party support and be built and maintained by Dfinity as it would significantly ease integration with the larger WASM ecosystem and all popular languages
This means that as more languages build their native integrations with WASM and choose their build target as wasm32-wasi, they would just work on the IC canisters without costly integration efforts such as Azle or Kybra. Nothing against Azle or Kybra, they are amazing projects with the brilliant @lastmjs at the helm
Exhibit C - IC Repl Test vs native testing via Pocket IC
There was a gap in testing tools in the ecosystem for quite some time before the main focus was on Pocket IC maintained by the amazing @mraszyk. Pocket IC came into being early this year
If you were trying to write automated tests before this, IC Repl is what you were expected to use but it was a NIGHTMARE to use. As a Motoko dev, you had no other option.
It required you to learn another new language/syntax
No syntax highlighting/autocomplete
Confusing error messages
Minimal to no documentation
The absolutely AMAZING and helpful @chenyan was your only way out if you got stuck
No serious Dfinity maintained canister codebase used this
All Dfinity maintained serious canister projects were using something called ic-state-machine-tests for which you had to use git dependencies to pull and download the entire IC repository into your project which led to your canister builds taking minutes instead of seconds, even on a beefy 16 core dev workstation. Shout out to the amazing @frederikrothenberger who helped us navigate this codebase
Internet Identity used this before
Us (Yral dapp) and OpenChat were using this before Pocket IC
Reason for this rant: Candid vs WIT/Protocol Buffers
Candid is another such frustrating tooling/language/protocol that the IC chose to create with minimal incremental benefits over existing industry standards like WIT or Protocol Buffers
New syntax to learn
Minimal tooling support via the Motoko Language Server (whole other can of worms)
So, you needed another storage serialization format like CBOR or MessagePack which bloats your canister WASM size
Till now, Candid was still bearable because you could just annotate your public endpoints with #[candid_method] and it would generate the Candid for you.
This slide from the June Global R&D summarizes this
This is an absolute nightmare for developers. They now have to learn a new language, a new syntax, a new tooling, and a new protocol for no significant benefits over existing industry standards.
The reasons for this move is cited in the first slide which are:
hard to maintain
lots of libraries
Both of these are abstracted away from the canister developer and leads to significantly more developer HAPPINESS.
This is an absolute step in the wrong direction and I hope the IC team reconsiders this decision and goes back to the drawing board to see how they can leverage existing industry standards and tools to make the developer experience better and NOT WORSE.
If you are considering making such a drastic change, why not move over to WIT completely and be done with it. Itās a more future-proof approach and will make IC canisters much more interoperable with the larger WASM ecosystem
Iāll wrap up this rant with this quote from Jurassic Park:
If DFINITY trains an AI LLM specialized for coding in Motoko and Rust by using prompts, would this remove the barrier to entry and increase productivity for the ICP ecosystem?
For Rust bindgen, we are NOT moving away from the Rust ā Candid export approach. This current approach will continue to work and be maintained. The global R&D demo is simply proposing an alternative option for the development workflow. The āhard to maintainā part on the slide was referring to the case where you are developing both backend and frontend. If you are purely a backend developer, I agree it doesnāt affect you much, and the existing approach can continue to work.
Speaking about industry standard, if you consider protobuf as one of the industry standards, protobuf also requires you to write type definitions in protobuf first and generate host language bindings, which is what we are proposing here. On the contrary, Iām not aware of any wire formats that exports type from the host languages. In this sense, we are moving towards the industry standard.
But again, the current approach continues to work. We are simply offering another option here. More options wonāt make thing worse at least.
Now letās talk about WIT. WIT is designed to allow interop between components within a Wasm binary. There is no concern about backward compatibility in their setting, because you control the whole codebase, and should be easy to upgrade all component in one go. In the blockchain setting, developers donāt own all the canisters, and some of canisters are even immutable (blackholed). We have to consider backward compatibility for these canisters to continue working when other canisters upgrade their interfaces. In essence, backward compatibility is the only thing Candid cares about. WIT doesnāt care about backward compatibility, they explicitly mentioned this in their spec.
On the plus side, if you compare the syntax of both Candid and WIT, they are extremely similar. I would even say they are the same. So at a future time when WIT is mature enough and supports subtyping, I donāt see much difficulty in switch/merging Candid and WIT. At itās current state, WIT is still a WIP, its spec and its implementation are still changing everyday. Are you aware of any projects using WIT? I would love to see how people use it at the moment.
If WIT doesnāt currently support a use case, wouldnāt it make sense to work with The Bytecode Alliance to implement the missing features into it and then use that instead of going harder on Candid?
Dfinity is a member of this alliance and should have some say on their roadmap. Itās a positive sum outcome for everyone involved
Fermyon Cloud is one provider that Iām aware of that is using this. I havenāt used their products in depth yet, so, canāt comment yet about its effectiveness
While I do see where youāre coming from, I think there might be some context missing for some of the cases you bring up.
Specifically, the System API was designed way before there was anything standardized (or even mature) around WASI. Even then, the first WASI standard was not very flexible and thatās why things are now moving into WASI 2 and the component model. We do have an internal exploration to see how WASI 2 could be integrated into the IC. Especially for things around WASM we are quite committed to follow along the community standards for maximum interoperability providing that the standards are mature enough for the use cases we need on the IC. WIT is somewhat similar in this aspect ā weāre not against exploring what it brings but certainly was not at a level that we could have seriously considered it 4-5 years ago.
About testing canisters: itās true that we delayed a lot in providing some standard way of testing thatās closer to what people are used to when building in the native languages. In the past, there simply wasnāt enough focus on developer experience but this has changed a lot over the past year or so I would say. There are multiple projects/features around improving developer experience and bringing canister development closer to building a traditional service, PocketIC being one of them.
I would say that we typically invent our own thing when existing industry standards are lacking or missing critical functionality/properties that we need on the IC. We can always try to work with the relevant organisations to add whatās missing in the standards but these processes take a long time and we need to balance providing at least something vs waiting until the standard adopts what we need.
I absolutely agree and see the need to create new things when nothing existing serves the use case
I am mostly urging that when we make decisions that affect developer experience, we at least do some rounds of feedback with teams who work with large canister codebases, both internal and external, and get their feedback on them before doubling down on a particular approach.
This is considering that we are at a stage that there are teams with significantly large codebases that deploy to canisters that they intend to maintain long-term as opposed to before where a lot of these codebases were smaller intended to be example or throwaway repos and things were still in their exploratory stages
I am mostly urging that when we make decisions that affect developer experience, we at least do some rounds of feedback with teams who work with large canister codebases, both internal and external, and get their feedback on them before doubling down on a particular approach.
I think we already do that, examples are canister logging and snapshots which we discussed on the forum and we plan to continue engaging with developers for such user facing features when it makes sense.
If this is about the Candid Type Selector, as Yan mentioned, this feature adds an optional path where someone first defines the Candid interface instead of first writing Rust or Motoko. If you want, you can keep using the old style. We considered this a net benefit as you can ignore it if you donāt like it but it greatly helps in other cases where you have canisters interacting with other third party canisters and you can more easily declare that dependency and get the generated types instead of having to copy them by hand.
This is overarching feedback in general. Not specifically a single thing.
And a documentation of some of the frustrations that one encounters while developing canisters on the IC.
Itās not targeted at any one specific feature in particular. Although the Candid changes is what brought this up.
Please consider this as feedback that is intended to provide an external team/developerās perspective on the general direction of what itās like to develop on the IC, from someone who looks at and tries multiple other offerings, both on and off chain on a continual basis.
Iām not calling for a stop to whatever it is you are doing to make things better. I am trying to add a perspective so that when you are building solutions targeted towards canister developers, you have some perspective about what canister developers find frustrating while building on the IC
I only have the utmost respect and love for what we are trying to achieve here
A note about languages. It is important to understand that none of the ones mentioned in the OP are comparable. They all make very different trade-offs and hence serve different use cases.
Rust is a low-level systems language. It is great if you need to optimise performance or have precise control over resources, which is why e.g. core IC canisters are written in it. OTOH, it requires much more hand-holding and boilerplate and generally is less productive than a more high-level language.
TypeScript is on the opposite end of the spectrum, it is JavaScript with some sugar coating: highly dynamic, highly complex runtime semantics. As such, expect 100x the gas cost or more for running an application compared to Rust. (And I wonāt even go into how error-prone it is, which IMO makes it a questionable choice for DeFi in particular.) Its main advantage is familiarity and the richness of the existing eco system. Python is in a roughly similar boat.
Motoko tries to strike a balance. Itās as high-level as TS, but without the same runtime overhead and most of the other technical disadvantages. As a result, it is not quite as efficient as Rust (Iād expect something like 2-3x, but that may vary widely depending on use case). It also has seamless native support for core IC features, such as actors and Candid. The big downside of course is that it is a much less baked and familiar language, with a tiny eco system.
But it is a feature of the IC that it letās you choose! And that it enables porting any other language, too. The ideal world is one where you can pick between dozens of tools, whatever is right for the job at hand, not one where you need to use the same hammer for every screw. ;)
Finally, to set the history straight, as @chenyan already hinted at, neither JS nor WIT nor WASI nor any other language besides Rust and C++ existed on Wasm when the IC started. Some of them are still largely WIP today, changing rapidly, and adopting them before they have sufficiently stabilised could be risky.
FWIW, Candid even had a non-zero influence on WIT. Iāve been involved with the former and have regular design discussions with the inventors of the latter.
Just wanted to say that I generally agree with the sentiment expressed in the OP, and I wanted to add some exhortations:
Please embrace Wasi and socket-based canisters to the fullest extent possible. The goal should be to allow developers to pull any package right off of crates.io, npm, or PyPI and have it execute without changes on ICP.
Using a socket-based approach it may be possible to implement many protocols for ingress messages and cross-canister communication, with messages of very large size. HTTP, SSH, and Candid RPC might be able to be implemented like this.
For example, I want to emphasize that even cross-canister calls should be performed with HTTP requests. This would provide a very simple developer experience. Ingress messages should also be performed with HTTP requests (all of this at least as an optional and maybe the preferred communication means).
Embracing Wasi and its socket APIs, and adding any other missing APIs as necessary, can help ICP become a more universal software platform with a better developer experience by embracing existing packages and application-layer communication protocols.
On behalf of the entire core PocketIC team (@michael-weigelt , @fxgst, and myself), we thank you for the acknowledgement! Weāre constantly improving the user experience and scope of canister testing with some cool features (e.g., canister https outcalls, snapshotting the state of PocketIC, using PocketIC library in tokio::test) coming out soon in the next PocketIC release.
Iāll just note on this that there is a significant contingent of us who have an assumption that āpull any package right offā is impossible and cannot be accomplished due to the different properties of distributed systems(which the IC is) and the unified systems approach that 99% of the software on those platforms assume. At scale, they break down which is why many are pursuing a bottoms-up approach to library creation and a de novo systems architecture for the IC. For smaller applications that donāt need to scale and donāt expect long-running processes, or that werenāt built with long-running processes inherent to their operation these are a great gateway for developers to enter into the ecosystem and the work Jordan has done is amazing.
Feature or bug? Let me count the number of attempted external contributions to these canisters. Oneā¦ ā¦ ā¦ ā¦ ā¦ ā¦ ā¦ ā¦ ā¦ ā¦
As a motoko dev Iāve written many tests and have had almost no interaction with ic-repl. The origyn nft has a massive suite of pure motoko tests. Pocket IC and Pic.js are certainly nice solutions, especially for testing against things like NNS canisters or multi-subnet components, but Iām still recommending that motoko devs write their tests in pure motoko because it makes you a better motoko programmer and helps you understand the async architecture of the IC at a fundamental level.(and then execute those test canisters with Pic.js).
Again, while interop would be great, most ecosystems are making very different assumptions about how their software will run than scalable, highly efficient IC code will need to make. (Iām not at all against using the standard as weāll get a bunch of tools for free, but I donāt expect to be able to start pulling a bunch of wasm code from other ecosystems into IC canisters without hitting significant roadblocks down the road and having to rewrite.)
As I said earlier, WIT and Candid are designed for different purposes. The term āinterfaceā in WIT refers to the interface of a component, whereas āinterfaceā in Candid refers to the interface of a canister/micro-service.
You can think of component as a third-party library in your project. When the interface of that library changes, you get compiler errors when your code tries to call that library with the old interface. As you have full control over your own code, you can fix these errors. Also, there is no rush to make these fixes, because the old library is linked in your final binary, your previously released code continues to work.
For canisters, we call other running canisters. When the interface of these running canisters change, your canister immediately stopped working, unless the serialization format is backward compatible and allows canisters call them with the old interface. Things can get worse if your canister is immutable that you cannot even update your code. Being backward compatible does mean that you are losing some flexibility in how you can update your interface (see the strange option rules in Candid). But in return, your canister continue to work without change, even though other canisters you depend on changed their interfaces.
None of the above mentioned problems exist in the component/WIT setting. Adding the subtyping rule to WIT would mean that they have to sacrifice some flexibility for a non-existed problem.
btw, if you look at how wit-bindgen currently works: GitHub - bytecodealliance/wit-bindgen: A language binding generator for WebAssembly interface types. It requires you to write a .wit file first and generate Rust bindings through a bindgen tool, which is exactly what we proposed in the demo, and exactly what you think to be the wrong direction. There is no way to generate a .wit file from your Rust code. So why do you think moving to WIT is an improvement to DX?
I apologize if my reasoning was unclear and got lost in the rant.
I am proposing that if we continue to use Candid, then we abstract it away and not force developers to learn new syntax only limited to the IC ecosystem with its own quirks and bugs but instead have it be autogenerated from source code (annotation). And that is what we focus our time and effort on.
If we have to force devs to learn new syntax, then it should be for industry accepted future proof syntax, like Protobuf or WIT.
Iām not sure youād like to read it as itās from September 2022 and as I was reading it, I realized a lot of it is outdated and some not true at all. However, for historical purposes, I can still put it up if you think Motoko devs and Dfinity engineers building the Motoko stack for the IC can benefit from that perspective.
Iād like to refrain from cross posting that here to keep the discussion focused on the original topic, however, if youād like me to post that, please DM me and Iāll create a separate post.
Having said that, what could be beneficial is if a current Motoko dev would summarize their frustrations/thoughts with the current state of things as that would be more relevant and up to date. We (Yral dapp) havenāt been using Motoko since Sep '22 and hence our views would be dated. Feedback from current Motoko devs would be much more valuable.
And then I can post my thoughts from back then for historical context and that would give you insight into how things have changed (mostly for the better) from back then
I understand the motivation, and thatās why we try to support Rust ā Candid export first, even though most mainstream serialization formats go in the opposite direction. Candid export will continue to be supported and improved. We recently discovered a new way to export candid type that can potentially fix some of the existing problems, but it introduces some other problems. We will see how it goes. This is just one example that we are still trying to improve Candid export, not abandon it.
Maybe counterintuitively, the Candid ā Rust bindgen approach also helps to abstract away Candid. When making an inter-canister call, all you have is a did file. If we donāt have a good Rust bindgen, you will have to write the type bindings manually and understand how Candid types map to Rust types, which is way harder than learning the Candid syntax. The bindgen tool abstracts away all these learnings, and gives you a Rust code you can use directly without looking at the did file.
DX is a very opinionated area. There are developers who prefer to use the Candid centric approach, just like other mainstream serialization libraries, such as protobuf and WIT. There are also developers who prefer to abstract away Candid. As a library, we are neutral on how people use it and should support both directions as much as we can.
