Long Term R&D: SDK (proposal)

1. Objective

Our SDK mission is to make IC smart contract tooling intuitive to use and flexible enough for any use case. For beginners, we want to cut down on the mental overhead. For intermediate developers, we want to add improved debugging tools and profiling. And for advanced developers, we want to offer rigorous tooling that will offer greater visibility to the low-level operations and memory management of a canister.

2. Background

Currently, the SDK team oversees several tools that allow developers to develop and interact with Internet Computer canisters. Those tools include the main SDK repo, the Rust Canister Development Kit (CDK), the Rust agent, and the JavaScript agent.

3. Why this is important

The IC has a great deal of complexity, and engineers from Web2 and other blockchain contexts have high expectations for the quality of their developer experience. For the IC to become the “world computer”, and attract talented developers at all levels of experience, we need to provide world-class tooling.

4. Topics under this project

  • Documentation
  • Improvements to debugging & profiling
  • Safe Upgrades and Backups

5. Key milestones (and technical solution if known)

Beginner-friendly features

Someone getting started on the platform has a lot of mental overhead, and we want to accommodate them by reducing the amount of Internet Computer context that someone needs to understand to get started. To that end, we have:

  • Expand the number of project templates we offer to include popular frontend frameworks and options that come with things like CRUD, authentication, and asset management
  • Simplify DFX, particularly around the Cycles Wallet and cache management
  • Provide tutorials for common use-cases
    • NFT’s
    • Tokens / ledgers
    • DeFi and payments

Intermediate Developer Features

For intermediate developers, we want to improve the quality of the development workflow, with a particular focus on developer tooling and safe, reproducible upgrades to live canisters. That will include:

  • Cycles cost estimation for Rust and Motoko
  • Debugging in the local canister execution environment
  • Logging in live canisters
  • Better IDE integration - code completion in Motoko, inline cycles cost insights
  • Ability to download state from a production canister
    • testing upgrades locally
    • Support for easy backups and restoration in DFX
  • Safe upgrades

Advanced Features

  • Profiling and scaling tools
  • Improve dfx for working with remote canisters with ambiguous interfaces
  • Tooling for automatic horizontal scaling

And finally, some features that everyone will enjoy:

  • React Native support
  • Improved asset / web functionality
    • Support for app push notifications
    • WebDAV or FS-UI support for asset canisters
    • Asset streaming through the serviceworker
    • New asset upload npm package
    • Overall improvements to managing large amounts of data on the IC

6. Discussion leads

Kyle Peacock

7. Why the DFINITY Foundation should make this a long-running R&D project

Tools under the SDK mandate are critical both to the internal and external stakeholders. Improved tooling makes it easier to discover and patch security issues on Canisters, to improve performance, and allow for safe upgrades and migration of data.

8. Skills and Expertise necessary to accomplish this (maybe teams?)

We have openings for experienced engineers with backgrounds in tooling, distributed systems, and anyone with solid experience in Rust.

9. Open Research questions

We have not settled on the best approach to run a local execution environment that supports multiple OS’s, debugging, and canister profiling.

10. Examples where community can integrate into project

All of our SDK codebases are open-source. Anyone can create issues, open pull requests, and collaborate with the SDK team. Get started at:

11. What we are asking the community

This will be the same for all:

  • Review comments, ask questions, give feedback

  • Vote accept or reject on NNS Motion

10 Likes

We are building a Python Agent library for the IC, now support basic query, update & read state calls.

9 Likes

We are implementing Java Agent for the IC. Already released several Beta versions, the first release candidate should be out in January, just finishing some features related to complex Candid types serialization/deserialization to Java objects.
We just received Dfinity grant to complete the first version with professional documentation, samples demonstrating various Java2IC scenarios (Android, Spring, integration with JDBC database). So this should be out early next year, probably March.
Any development team developing additional SDKs (Swift/ObjectiveC, C#, Python, Go), we will be happy to share our experience and knowledge. We should also sync up things like API (we used Rust Agent version as an inspiration, but added some Java specific goodies), documentation, test cases.

3 Likes

Proposal is live: Internet Computer Network Status

i tried the agent today and have to say it’s very nice to work with, thank you for implementing this!

3 Likes

We refactored our Java Agent libraries under new brand IC4J. It’s now packaged under org.ic4j.

This is our first Release Candidate, we finalized features for this release, added support for native Java object serializations/deserializations to Candid. Now we support Java POJOs, Jackson and Gson JSON objects, XML/DOM objects and JDBC ResultSet.
We also separated Candid package from Agent package. This will allow us in the future start working on Candid IDL related tools, like POJOs generated from Candid IDL, XML or JSON schema mappings to Candid. Eventually also plugins for Code, Eclipse, Android Studio.
In the next few weeks we are going to work on more complex demo application, Loan Flow, basically simplified Loan Broker app which will demonstrate some Java to IC scenarios (Android client, data integration with JDBC, Spring based front end).
We are also working on professional documentation (something similar what web3j has).

4 Likes

What is the progress regarding these two features? Is there a rough estimated time of delivery?

Good question. Let me ping SDK folks.

2 Likes

@chenyan has enabled cycles profiling on Motoko playground, as you can see here: Programmatically measure cycles consumption - #4 by gabe

I think were waiting on some features to get this running locally though

3 Likes

We’re only waiting on the next stable release. Will probably happen within the next two weeks, then we’ll have cycles accounting locally so you can estimate pretty well on a local machine.

4 Likes

Wow, this is great news! Would you like to elaborate on which features that will be added?

I’m not going to go into detail here as we’ll have a proper announcement prepared once the next release is stable. We’ll also post it in the forum, but I’ll also make a note to come into this thread and link it manually.

If you want a sneak-peek you can go to the changelog of the dfx repo, subsection feat: Configure subnet type of local replica.

1 Like

Understandable. Perfect, I will check out the changelog. Thanks!

Will the local environment simulate and include e.g Compute Percent Allocated Per Second, Ingress Byte Reception, and GB Storage Per Second from this table in the profiling, allowing us to get a complete picture of cycles consumption?

The local environment should not just simulate it. It should be the same environment since the binary is even the same. If those values are not considered, it would be a bug. So yes, you should get the full picture of cycle consumption (assuming you can get similar data to your local canister).

Got it. That is awesome!

When it comes to the profiling though, will we be able to see the “full picture” of cycles consumption as in including ingress message dynamic and fixed costs, inter-canister dynamic and fixed costs, allocated compute percent costs, as well as storage costs? I am asking because from the profiling currently available in Motoko Playground it is only possible to deduce the dynamic execution cost of update calls AFAIK.

We’re starting with what is available in Motoko Playground. We will continue to improve our tooling to be more transparent and user friendly to get as much info in your hands as we can access

1 Like

Makes total sense. Cycles cost estimation is high up on my priority list so I’m excited to see what’s to come! Great work, I’m looking forward to the upcoming stable release :slight_smile:

1 Like

Just found out that there’s dfx release 0.10.0-beta.1 available if you want to try it yourself

2 Likes

dfx 0.10.0 is available: Exciting changes coming to your local development experience - #15 by Severin

2 Likes