Long Term R&D: SDK (proposal)

1. Summary

This is a motion proposal for the long-term R&D of the DFINITY Foundation, as part of the follow-up to this post: Motion Proposals on Long Term R&D Plans (Please read this post for context).

This project’s objective

Significant investment into the developer experience as the SDK is the main tool and interface used by developers when building on the IC. The SDK shall provide a better debugging and testing environment, allow profiling of dapps, ensure reliable canister upgrades and canister state retrieval. The SDK shall come with improved documentation and example code for more use cases, frameworks and languages.

2. Discussion lead

Kyle Peacock

3. How this R&D proposal is different from previous types

Previous motion proposals have revolved around specific features and tended to have clear, finite goals that are delivered and completed. They tended to be measured in days, weeks, or months.

These motion proposals are different and are defining the long-term plan that the foundation will use, e.g., for hiring and organizational build-out. They have the following traits and patterns:

  1. Their scope is years, not weeks or months as in previous NNS motions
  2. They have a broad direction but are active areas of R&D so they do not have an obvious line of execution.
  3. They involve deep research in cryptography, networking, distributed systems, language, virtual machines, operating systems.
  4. They are meant to match the strengths of where the DFINITY foundation’s expertise is best suited.
  5. Work on these proposals will not start immediately.
  6. There will be many follow-up discussions and proposals on each topic when work is underway and smaller milestones and tasks get defined.

An example may be the R&D for “Scalability” where there will be a team investigating and improving the scalability of the IC at various stages. Different bottlenecks will surface and different goals will be met.

3. How this R&D proposal is similar to what we have seen

We want to double down on the behaviors we think have worked well. These include:

  1. Publicly identifying owners of subject areas to engage and discuss their thinking with the community
  2. Providing periodic updates to the community as things evolve, milestones reached, proposals are needed, etc…
  3. Presenting more and more R&D thinking early and openly.

This has worked well for the last 6 months so we want to repeat this pattern.

4. Next Steps

Developer forum intro posted
1-pager from the discussion lead posted
NNS Motion proposal submitted

5. What we are asking the community

  • Ask questions
  • Read 1-pager
  • Give feedback
  • Vote on the motion proposal

Frankly, we do not expect many nitty-gritty details because these are meant to address projects that go on for long time horizons.

The DFINITY foundation’s only goal is to improve the adoption of the IC so we want to sanity-check the projects we see necessary for growing the IC by having you (the ICP community) tell us what you all think of these active R&D threads we have.

6. What this means for the existing Roadmap or Projects

In terms of the current roadmap and proposals executed, those are still being worked on and have priority.

An intellectually honest way to look at this long-term R&D project is to see them as the upstream or “primordial soup” from which more baked projects emerge from. With this lens, these proposals are akin to asking, “what kind of specialties or strengths do we want to make sure DFINITY foundation has built up?”

Most (if not all) projects that the DFINITY foundation has executed or is executing are borne from long-running R&D threads. Even when community feedback tells the foundation, “we need X” or “Y does not work”, it is typically the team with the most relevant R&D area that picks up the short-term feature or project.

Please note:

Some folks gave asked if they should vote to “reject” any of the Long Term R&D projects as a way to signal prioritization. The answer is simple: “No, please, ACCEPT” :wink:

These long-term R&D projects are the DFINITY’s foundation’s thesis at R&D threads it should have across years (3 years is the number we sometimes use internally). We are asking the community to ACCEPT (pending 1-pager and more community feedback of course). Prioritization can come at a separate step.

Maybe we can list the SDKs we have right now, and see how community can fill the gaps.

2 Likes

love it. great idea.

1. Objective

Our SDK mission is to make IC smart contract tooling intuitive to use and flexible enough for any use case. For beginners, we want to cut down on the mental overhead. For intermediate developers, we want to add improved debugging tools and profiling. And for advanced developers, we want to offer rigorous tooling that will offer greater visibility to the low-level operations and memory management of a canister.

2. Background

Currently, the SDK team oversees several tools that allow developers to develop and interact with Internet Computer canisters. Those tools include the main SDK repo, the Rust Canister Development Kit (CDK), the Rust agent, and the JavaScript agent.

3. Why this is important

The IC has a great deal of complexity, and engineers from Web2 and other blockchain contexts have high expectations for the quality of their developer experience. For the IC to become the “world computer”, and attract talented developers at all levels of experience, we need to provide world-class tooling.

4. Topics under this project

  • Documentation
  • Improvements to debugging & profiling
  • Safe Upgrades and Backups

5. Key milestones (and technical solution if known)

Beginner-friendly features

Someone getting started on the platform has a lot of mental overhead, and we want to accommodate them by reducing the amount of Internet Computer context that someone needs to understand to get started. To that end, we have:

  • Expand the number of project templates we offer to include popular frontend frameworks and options that come with things like CRUD, authentication, and asset management
  • Simplify DFX, particularly around the Cycles Wallet and cache management
  • Provide tutorials for common use-cases
    • NFT’s
    • Tokens / ledgers
    • DeFi and payments

Intermediate Developer Features

For intermediate developers, we want to improve the quality of the development workflow, with a particular focus on developer tooling and safe, reproducible upgrades to live canisters. That will include:

  • Cycles cost estimation for Rust and Motoko
  • Debugging in the local canister execution environment
  • Logging in live canisters
  • Better IDE integration - code completion in Motoko, inline cycles cost insights
  • Ability to download state from a production canister
    • testing upgrades locally
    • Support for easy backups and restoration in DFX
  • Safe upgrades

Advanced Features

  • Profiling and scaling tools
  • Improve dfx for working with remote canisters with ambiguous interfaces
  • Tooling for automatic horizontal scaling

And finally, some features that everyone will enjoy:

  • React Native support
  • Improved asset / web functionality
    • Support for app push notifications
    • WebDAV or FS-UI support for asset canisters
    • Asset streaming through the serviceworker
    • New asset upload npm package
    • Overall improvements to managing large amounts of data on the IC

6. Discussion leads

Kyle Peacock

7. Why the DFINITY Foundation should make this a long-running R&D project

Tools under the SDK mandate are critical both to the internal and external stakeholders. Improved tooling makes it easier to discover and patch security issues on Canisters, to improve performance, and allow for safe upgrades and migration of data.

8. Skills and Expertise necessary to accomplish this (maybe teams?)

We have openings for experienced engineers with backgrounds in tooling, distributed systems, and anyone with solid experience in Rust.

9. Open Research questions

We have not settled on the best approach to run a local execution environment that supports multiple OS’s, debugging, and canister profiling.

10. Examples where community can integrate into project

All of our SDK codebases are open-source. Anyone can create issues, open pull requests, and collaborate with the SDK team. Get started at:

11. What we are asking the community

This will be the same for all:

  • Review comments, ask questions, give feedback

  • Vote accept or reject on NNS Motion

10 Likes

We are building a Python Agent library for the IC, now support basic query, update & read state calls.

9 Likes

We are implementing Java Agent for the IC. Already released several Beta versions, the first release candidate should be out in January, just finishing some features related to complex Candid types serialization/deserialization to Java objects.
We just received Dfinity grant to complete the first version with professional documentation, samples demonstrating various Java2IC scenarios (Android, Spring, integration with JDBC database). So this should be out early next year, probably March.
Any development team developing additional SDKs (Swift/ObjectiveC, C#, Python, Go), we will be happy to share our experience and knowledge. We should also sync up things like API (we used Rust Agent version as an inspiration, but added some Java specific goodies), documentation, test cases.

2 Likes

Proposal is live: Internet Computer Network Status

i tried the agent today and have to say it’s very nice to work with, thank you for implementing this!

1 Like