How to write a minimal CDK?

twan · October 5, 2020, 9:22pm

This is quite a broad question, and I’d appreciate it if the answer wasn’t that more information will be published soon.

Back in the day, I released a minimal viable EOS SDK to develop smart contracts with assemblyscript, a subset of typescript that compiles to wasm. I’m interested in doing the same with dfinity, but instead of assemblyscript I’d like to use tinygo (golang that can compile to wasm). The value in this is that Golang is much more accessible and readable then Rust (all though I believe Motoko has these properties as well).

A basic SDK only needs two functions: Communicate with the environment (eg. print debug information, send messages, get caller metadata…) and storage

Communication:

How is communication done with the environment?
I assume communication is done using standard wasm exports, if so, is there a definition of a list of functions that should be imported. Is this that list?
Storage:
How is storage persisted?
In the talks it is said that memory pages are simply persisted. Can you elaborate? Does this mean we have to have some garbage collection/memory management or can we be more specific about which memory should be persisted?

A nice and long explanation post addressing these two topics, with links to the rust cdk for example, would be very valuable to the community and developers of upcoming cdks. I’m looking forward to it!

twan · October 5, 2020, 9:28pm

Oh yeah and full disclosure: I may ask for some renumeration from the beacon fund before opensourcing this if the golang cdk is actually really useful

cryptoschindler · October 7, 2020, 2:11pm

tagging for visibility @stanley.jones @chenyan @hansl @enzo

enzo · October 7, 2020, 4:59pm

Here are the host functions provided to the Internet Computer in the Sodium release:

/**
 * File       : ic0.h
 * Copyright  : DFINITY USA Research LLC
 * License    : Apache 2.0 with LLVM Exception
 * Maintainer : Enzo Haussecker <[email protected]>
 * Stability  : Experimental
 */

#ifndef IC0_H
#define IC0_H

#define WASM_IMPORT(module, name) \
  __attribute__((import_module(module))) \
  __attribute__((import_name(name)));

#define WASM_EXPORT(name) asm(name) \
  __attribute__((visibility("default")));

#include <stdint.h>

uint32_t ic0_msg_arg_data_size()
  WASM_IMPORT("ic0", "msg_arg_data_size");

void ic0_msg_arg_data_copy(uint32_t dst, uint32_t off, uint32_t size)
  WASM_IMPORT("ic0", "msg_arg_data_copy");

uint32_t ic0_msg_caller_size()
  WASM_IMPORT("ic0", "msg_caller_size");

void ic0_msg_caller_copy(uint32_t dst, uint32_t off, uint32_t size)
  WASM_IMPORT("ic0", "msg_caller_copy");

uint32_t ic0_msg_reject_code()
  WASM_IMPORT("ic0", "msg_reject_code");

uint32_t ic0_msg_reject_msg_size()
  WASM_IMPORT("ic0", "msg_reject_msg_size");

void ic0_msg_reject_msg_copy(uint32_t dst, uint32_t off, uint32_t size)
  WASM_IMPORT("ic0", "msg_reject_msg_copy");

void ic0_msg_reply_data_append(uint32_t src, uint32_t size)
  WASM_IMPORT("ic0", "msg_reply_data_append");

void ic0_msg_reply()
  WASM_IMPORT("ic0", "msg_reply");

void ic0_msg_reject(uint32_t src, uint32_t size)
  WASM_IMPORT("ic0", "msg_reject");

uint64_t ic0_msg_funds_available(uint32_t src, uint32_t size)
  WASM_IMPORT("ic0", "msg_funds_available");

uint64_t ic0_msg_funds_refunded(uint32_t src, uint32_t size)
  WASM_IMPORT("ic0", "msg_funds_refunded");

void ic0_msg_funds_accept(uint32_t src, uint32_t size, uint64_t amount)
  WASM_IMPORT("ic0", "msg_funds_accept");

uint32_t ic0_canister_self_size()
  WASM_IMPORT("ic0", "canister_self_size");

void ic0_canister_self_copy(uint32_t dst, uint32_t off, uint32_t size)
  WASM_IMPORT("ic0", "canister_self_copy");

uint64_t ic0_canister_balance(uint32_t src, uint32_t size)
  WASM_IMPORT("ic0", "canister_balance");

void ic0_call_new(
  uint32_t callee_src,
  uint32_t callee_size,
  uint32_t name_src,
  uint32_t name_size,
  uint32_t reply_fun,
  uint32_t reply_env,
  uint32_t reject_fun,
  uint32_t reject_env
) WASM_IMPORT("ic0", "call_new");

void ic0_call_on_cleanup(uint32_t fun, uint32_t env)
  WASM_IMPORT("ic0", "call_on_cleanup");

void ic0_call_data_append(uint32_t src, uint32_t size)
  WASM_IMPORT("ic0", "call_data_append");

void ic0_call_funds_add(uint32_t src, uint32_t size, uint64_t amount)
  WASM_IMPORT("ic0", "call_funds_add");

uint32_t ic0_call_perform()
  WASM_IMPORT("ic0", "call_perform")

uint32_t ic0_stable_size()
  WASM_IMPORT("ic0", "stable_size");

uint32_t ic0_stable_grow(uint32_t new_pages)
  WASM_IMPORT("ic0", "stable_grow");

void ic0_stable_write(uint32_t off, uint32_t src, uint32_t size)
  WASM_IMPORT("ic0", "stable_write");

void ic0_stable_read(uint32_t dst, uint32_t off, uint32_t size)
  WASM_IMPORT("ic0", "stable_read");

uint64_t ic0_time()
  WASM_IMPORT("ic0", "time");

void ic0_debug_print(uint32_t src, uint32_t size)
  WASM_IMPORT("ic0", "debug_print");

void ic0_trap(uint32_t src, uint32_t size)
  WASM_IMPORT("ic0", "trap");

#endif /* IC0_H */

enzo · October 7, 2020, 5:00pm

You can implement a canister method called foobar like this:

#include <ic0.h>

void foobar() WASM_EXPORT("canister_update foobar");
void foobar() {

  // Read Candid-encoded arguments via ic0_msg_arg_data_size and
  // ic0_msg_arg_data_copy.

  // Do something...
  char str[18] = "Doing something...";
  ic0_debug_print((uint32_t)str, 18);

  // Write Candid-encoded results via ic0_msg_reply_data_append.

  ic0_msg_reply();
}

If your canister method does not modify any memory, or if the modified memory can be discarded, then you can change canister_update to canister_query.

If you want to execute some initialization logic that runs at installation, then you will need into implement a routine called canister_init like this:

void canister_init() WASM_EXPORT("canister_init");
void canister_init() {

  // Initialize...
  char str[15] = "Initializing...";
  ic0_debug_print((uint32_t)str, 15);
}

With respect to persistence, the Internet Computer persists the process memory of the WebAssembly programs that run on it. WebAssembly has a very simple linear memory model. When a function exported from WebAssembly program is invoked, and if that invocation modifies memory, then at the end of execution, the platform takes a snapshot of the memory. The snapshot is derived from a transaction log of modified memory pages. The platform uses the snapshot to instantiate the WebAssembly program when the next invocation occurs.

The process memory does not persist across canister upgrades by default. You’ll need to implement routines canister_pre_upgrade and canister_pre_upgrade for this by copying state into stable memory and back.

enzo · October 7, 2020, 5:29pm

Go will be tricky. You will need to implement the relevant runtime and syscall compiler modules so that the user can run GOOS=ic GOARCH=wasm go build. Those modules will require some minimal implementation of Candid.

skilesare · October 8, 2020, 5:36pm

Enzo,

Thanks so much for posting this. It is very helpful. As someone not super great with c compilers, do we just need the .h file? Does the dfinity sdk shim in the actual .c as a compiled binary at some point?

Do any of these let you get the Principal making the call? Or is that a different header file? What about randomness?

I’m trying to do something similar here but using duktape to get javascript compiling inside the cannister and this info is really helpful.

Thanks!

enzo · October 8, 2020, 9:41pm

Depends what you’re doing, but you’ll probably want more than just that header file yeah. You’ll probably want some basics from libc. Most are using Musl for this as it has pretty good support for compiling to WASM. The DFINITY SDK does not do anything special here, just produces a standard WASM binary with ic0_blah_blah_blah in the WASM import section.

Yes, the principal is available in that header file via ic0_msg_caller_size and ic0_msg_arg_data_copy. Randomness is available too, but right now it comes from the management canister ic1. I believe this is subject to change pending some design considerations. Personally, I would prefer to see it in the ic0 system API.

I haven’t used duktape, but it sounds like an interesting project.

.

twan · October 8, 2020, 10:49pm

Go is actually no problem! It will run with overhead however (so if memory and execution cycles are expensive, it might be relatively pointless). You would use the tinygo package which allows for proper wasm bindings and allows for gc-less execution.

The only challenge is see is getting the generic persistence to work accross upgrades, but perhaps that is something that can be figured out after the proof of concept.

Anyways, thank you for the clear explanation. Please do release a commented header file with the api! That would be great.

@skilesare, webassembly actually has the concept of imports and exports, you can consider these functions as remote procedure calls with the environment it’s an awesome feature for language interoperability!

enzo · October 8, 2020, 11:19pm

Using a garbage collected language isn’t really a problem. Motoko is garbage collected (though great care has gone into making it efficient for the IC). With respect to tinygo, I’m not sure how much mass-market appeal this has got. It certainly has a user base, but seems, at least superficially, to be peanuts compared to the compiler provided by Google, which has tens of thousands of developers installing it, and already provides support for WASM. It seems to me like the Google compiler just needs to be extended. We could push back channels to get a PR though.

These are good points about storage. Orthogonal persistence does not lend itself well to upgrades, and conveying the relevant bits to the end user is tricky, but could be abstracted behind a library as we did with Rust.

twan · October 9, 2020, 11:28pm

Both tinygo and native wasm compilation have their advantages. The tinygo compiler serves a subset of golang so there is no problem in targetting both.

However, as dfinity does not use a standard ABI format, do you still support wasi (or a subset of it)?

twan · October 10, 2020, 12:30am

I did an MVP test but am sadly enough getting an error:

$ dfx canister call print hello --query
Error deserializing blob 0x
Invalid data: Invalid IDL blob: An error occured:
CouldNotSerializeIdlFile(
   Deserialize(
       "wrong magic number [0, 0, 0, 0]",
       "Trailing type: []\nTrailing value: []\nType table: []\nRemaining value types: []",
   ),
)

The essence of my test code:

//export hello
func Hello() {
  log("Hello, world.")
  Commit()
}

func log(msg string) {
  DebugPrint(*(*uint32)(unsafe.Pointer(&msg)), (uint32)(unsafe.Sizeof(&msg)))
}

chenyan · October 10, 2020, 5:05am

Nice! If you call dfx canister call print hello --query --output raw, you should get no error.

The missing part is the Candid serialization format: https://github.com/dfinity/candid/blob/master/spec/Candid.md#binary-format.

Basically, if the function has no return value, dfx expects the raw bytes to be “DIDL\00\00”. If your function takes arguments, it also needs to follow the same serialization format. Your function actually receives “DIDL\00\00” as arguments sent by dfx, as the input argument is also empty.

twan · October 10, 2020, 10:04am

Very nice advice! I’ve just tried out the raw output, will now try appending DIDL\00\00 to the message reply to see how that goes.

[Canister 75hes-oqbaa-aaaaa-aaaaa-aaaaa-aaaaa-aaaaa-q] Hello, World!

Seems like the naive approach with the reply bytes did not work:

$ dfx canister call print hello --output raw --query
4449444c5c30305c3030

func commit() {
	msg := `DIDL\00\00`
	MsgReplyDataAppend(*(*uint32)(unsafe.Pointer(&msg)), (uint32)(len(msg)))
	MsgReply()
}

twan · October 10, 2020, 10:15am

Update, I was just being stupid with my golang string escaping:

func commit() {
	msg := "DIDL\000\000"
	MsgReplyDataAppend(*(*uint32)(unsafe.Pointer(&msg)), (uint32)(len(msg)))
	MsgReply()
}

yields:

$ dfx canister call print hello 
()

Important:
I would like to propose that the dfinity vm implements a subset of the WASI specification to allow for easier incorporation with languages that compile to WASM. This way, stuff like printing to stdout will continue to work as expected without any magic. WASI is the standard interface for languages to connect with the capabilities of the browser/host, like logging to console or accessing files. Not a lot of the API is probably relevant but implementing the logging part is very valuable as many libraries don’t adhere to the “don’t log anything by default” principle.

twan · October 12, 2020, 1:44pm

Ok, couple of steps further now. The lack of the second feature in this feature request is currently quite blocking.

To start with the data format of input and output, if I understand it correctly, dfinity does not adhere to the traditional wasm specification of exports (eg. using the function parameters of an export and types like u32). Instead we have to use ic0_msg_arg_data_copy to read in the parameters using the candid specification as described in the serialization section.

enzo · October 12, 2020, 11:22pm

We have no plan to support WASI. My recommendation would be to implement the relevant runtime and syscall compiler modules so that the end-user can type GOOS=ic GOARCH=wasm go build.