Long Term R&D: SDK (proposal)

1. Summary

This is a motion proposal for the long-term R&D of the DFINITY Foundation, as part of the follow-up to this post: Motion Proposals on Long Term R&D Plans (Please read this post for context).

This project’s objective

Significant investment into the developer experience as the SDK is the main tool and interface used by developers when building on the IC. The SDK shall provide a better debugging and testing environment, allow profiling of dapps, ensure reliable canister upgrades and canister state retrieval. The SDK shall come with improved documentation and example code for more use cases, frameworks and languages.

2. Discussion lead

Kyle Peacock

3. How this R&D proposal is different from previous types

Previous motion proposals have revolved around specific features and tended to have clear, finite goals that are delivered and completed. They tended to be measured in days, weeks, or months.

These motion proposals are different and are defining the long-term plan that the foundation will use, e.g., for hiring and organizational build-out. They have the following traits and patterns:

  1. Their scope is years, not weeks or months as in previous NNS motions
  2. They have a broad direction but are active areas of R&D so they do not have an obvious line of execution.
  3. They involve deep research in cryptography, networking, distributed systems, language, virtual machines, operating systems.
  4. They are meant to match the strengths of where the DFINITY foundation’s expertise is best suited.
  5. Work on these proposals will not start immediately.
  6. There will be many follow-up discussions and proposals on each topic when work is underway and smaller milestones and tasks get defined.

An example may be the R&D for “Scalability” where there will be a team investigating and improving the scalability of the IC at various stages. Different bottlenecks will surface and different goals will be met.

3. How this R&D proposal is similar to what we have seen

We want to double down on the behaviors we think have worked well. These include:

  1. Publicly identifying owners of subject areas to engage and discuss their thinking with the community
  2. Providing periodic updates to the community as things evolve, milestones reached, proposals are needed, etc…
  3. Presenting more and more R&D thinking early and openly.

This has worked well for the last 6 months so we want to repeat this pattern.

4. Next Steps

Developer forum intro posted
1-pager from the discussion lead posted
NNS Motion proposal submitted

5. What we are asking the community

  • Ask questions
  • Read 1-pager
  • Give feedback
  • Vote on the motion proposal

Frankly, we do not expect many nitty-gritty details because these are meant to address projects that go on for long time horizons.

The DFINITY foundation’s only goal is to improve the adoption of the IC so we want to sanity-check the projects we see necessary for growing the IC by having you (the ICP community) tell us what you all think of these active R&D threads we have.

6. What this means for the existing Roadmap or Projects

In terms of the current roadmap and proposals executed, those are still being worked on and have priority.

An intellectually honest way to look at this long-term R&D project is to see them as the upstream or “primordial soup” from which more baked projects emerge from. With this lens, these proposals are akin to asking, “what kind of specialties or strengths do we want to make sure DFINITY foundation has built up?”

Most (if not all) projects that the DFINITY foundation has executed or is executing are borne from long-running R&D threads. Even when community feedback tells the foundation, “we need X” or “Y does not work”, it is typically the team with the most relevant R&D area that picks up the short-term feature or project.

Please note:

Some folks gave asked if they should vote to “reject” any of the Long Term R&D projects as a way to signal prioritization. The answer is simple: “No, please, ACCEPT” :wink:

These long-term R&D projects are the DFINITY’s foundation’s thesis at R&D threads it should have across years (3 years is the number we sometimes use internally). We are asking the community to ACCEPT (pending 1-pager and more community feedback of course). Prioritization can come at a separate step.

Maybe we can list the SDKs we have right now, and see how community can fill the gaps.

2 Likes

love it. great idea.

1. Objective

Our SDK mission is to make IC smart contract tooling intuitive to use and flexible enough for any use case. For beginners, we want to cut down on the mental overhead. For intermediate developers, we want to add improved debugging tools and profiling. And for advanced developers, we want to offer rigorous tooling that will offer greater visibility to the low-level operations and memory management of a canister.

2. Background

Currently, the SDK team oversees several tools that allow developers to develop and interact with Internet Computer canisters. Those tools include the main SDK repo, the Rust Canister Development Kit (CDK), the Rust agent, and the JavaScript agent.

3. Why this is important

The IC has a great deal of complexity, and engineers from Web2 and other blockchain contexts have high expectations for the quality of their developer experience. For the IC to become the “world computer”, and attract talented developers at all levels of experience, we need to provide world-class tooling.

4. Topics under this project

  • Documentation
  • Improvements to debugging & profiling
  • Safe Upgrades and Backups

5. Key milestones (and technical solution if known)

Beginner-friendly features

Someone getting started on the platform has a lot of mental overhead, and we want to accommodate them by reducing the amount of Internet Computer context that someone needs to understand to get started. To that end, we have:

  • Expand the number of project templates we offer to include popular frontend frameworks and options that come with things like CRUD, authentication, and asset management
  • Simplify DFX, particularly around the Cycles Wallet and cache management
  • Provide tutorials for common use-cases
    • NFT’s
    • Tokens / ledgers
    • DeFi and payments

Intermediate Developer Features

For intermediate developers, we want to improve the quality of the development workflow, with a particular focus on developer tooling and safe, reproducible upgrades to live canisters. That will include:

  • Cycles cost estimation for Rust and Motoko
  • Debugging in the local canister execution environment
  • Logging in live canisters
  • Better IDE integration - code completion in Motoko, inline cycles cost insights
  • Ability to download state from a production canister
    • testing upgrades locally
    • Support for easy backups and restoration in DFX
  • Safe upgrades

Advanced Features

  • Profiling and scaling tools
  • Improve dfx for working with remote canisters with ambiguous interfaces
  • Tooling for automatic horizontal scaling

And finally, some features that everyone will enjoy:

  • React Native support
  • Improved asset / web functionality
    • Support for app push notifications
    • WebDAV or FS-UI support for asset canisters
    • Asset streaming through the serviceworker
    • New asset upload npm package
    • Overall improvements to managing large amounts of data on the IC

6. Discussion leads

Kyle Peacock

7. Why the DFINITY Foundation should make this a long-running R&D project

Tools under the SDK mandate are critical both to the internal and external stakeholders. Improved tooling makes it easier to discover and patch security issues on Canisters, to improve performance, and allow for safe upgrades and migration of data.

8. Skills and Expertise necessary to accomplish this (maybe teams?)

We have openings for experienced engineers with backgrounds in tooling, distributed systems, and anyone with solid experience in Rust.

9. Open Research questions

We have not settled on the best approach to run a local execution environment that supports multiple OS’s, debugging, and canister profiling.

10. Examples where community can integrate into project

All of our SDK codebases are open-source. Anyone can create issues, open pull requests, and collaborate with the SDK team. Get started at:

11. What we are asking the community

This will be the same for all:

  • Review comments, ask questions, give feedback

  • Vote accept or reject on NNS Motion

10 Likes

We are building a Python Agent library for the IC, now support basic query, update & read state calls.

9 Likes

We are implementing Java Agent for the IC. Already released several Beta versions, the first release candidate should be out in January, just finishing some features related to complex Candid types serialization/deserialization to Java objects.
We just received Dfinity grant to complete the first version with professional documentation, samples demonstrating various Java2IC scenarios (Android, Spring, integration with JDBC database). So this should be out early next year, probably March.
Any development team developing additional SDKs (Swift/ObjectiveC, C#, Python, Go), we will be happy to share our experience and knowledge. We should also sync up things like API (we used Rust Agent version as an inspiration, but added some Java specific goodies), documentation, test cases.

3 Likes

Proposal is live: Internet Computer Network Status

i tried the agent today and have to say it’s very nice to work with, thank you for implementing this!

3 Likes

We refactored our Java Agent libraries under new brand IC4J. It’s now packaged under org.ic4j.

This is our first Release Candidate, we finalized features for this release, added support for native Java object serializations/deserializations to Candid. Now we support Java POJOs, Jackson and Gson JSON objects, XML/DOM objects and JDBC ResultSet.
We also separated Candid package from Agent package. This will allow us in the future start working on Candid IDL related tools, like POJOs generated from Candid IDL, XML or JSON schema mappings to Candid. Eventually also plugins for Code, Eclipse, Android Studio.
In the next few weeks we are going to work on more complex demo application, Loan Flow, basically simplified Loan Broker app which will demonstrate some Java to IC scenarios (Android client, data integration with JDBC, Spring based front end).
We are also working on professional documentation (something similar what web3j has).

4 Likes

What is the progress regarding these two features? Is there a rough estimated time of delivery?

Good question. Let me ping SDK folks.

2 Likes

@chenyan has enabled cycles profiling on Motoko playground, as you can see here: Programmatically measure cycles consumption - #4 by gabe

I think were waiting on some features to get this running locally though

3 Likes

We’re only waiting on the next stable release. Will probably happen within the next two weeks, then we’ll have cycles accounting locally so you can estimate pretty well on a local machine.

4 Likes

Wow, this is great news! Would you like to elaborate on which features that will be added?

I’m not going to go into detail here as we’ll have a proper announcement prepared once the next release is stable. We’ll also post it in the forum, but I’ll also make a note to come into this thread and link it manually.

If you want a sneak-peek you can go to the changelog of the dfx repo, subsection feat: Configure subnet type of local replica.

1 Like

Understandable. Perfect, I will check out the changelog. Thanks!

Will the local environment simulate and include e.g Compute Percent Allocated Per Second, Ingress Byte Reception, and GB Storage Per Second from this table in the profiling, allowing us to get a complete picture of cycles consumption?

The local environment should not just simulate it. It should be the same environment since the binary is even the same. If those values are not considered, it would be a bug. So yes, you should get the full picture of cycle consumption (assuming you can get similar data to your local canister).

Got it. That is awesome!

When it comes to the profiling though, will we be able to see the “full picture” of cycles consumption as in including ingress message dynamic and fixed costs, inter-canister dynamic and fixed costs, allocated compute percent costs, as well as storage costs? I am asking because from the profiling currently available in Motoko Playground it is only possible to deduce the dynamic execution cost of update calls AFAIK.