Completed: ICDevs.org - Bounty #19 - CBOR and Candid Motoko Parser - $3,000+

Developers Bounties
1

This is the sixth bounty in a series of bounties we are releasing this week in the run-up to Motoko Bootcamp. Winners of the Bootcamp’s Intermediate level will get the first crack at selecting one of these bounties to complete.

CBOR and Candid Motoko Parser - #19

Current Status: Discussion

  • Discussion (02/27/2022)
  • Ratification
  • Open for application
  • Assigned
  • In Review
  • Closed

Official Link

Bounty Details

  • Bounty Amount: $3,000 USD of ICP at award date - $3000 USD of ICP Match Available
  • ICDevs.org DFINITY Foundation Grant Match Available: $3000 USD of ICP at award time - (For every ICP sent to d0e3be904a939ff752979aaa53cfb66fac8d75b85ce4fa878e147d60706be6ed, ICDevs.org will add $125 USD of ICP at award date to the bounty, up to the first 24 ICP donated, After 24 ICP, donations to the above address will add .25 ICP to this issue and .75 ICP to fund other ICDevs.org initiatives)
  • Project Type: Single Contributor
  • Opened: 02/20/2021
  • Time Commitment: Days
  • Project Type: Library
  • Experience Type: Intermediate - Motoko
  • Issue Type: Motoko Library

Description

This bounty gives the opportunity to

  • learn how CBOR encoding works
  • learn how candid works
  • learn how call_raw works in motoko

The goal of this bounty is to produce two motoko libraries.

More info about CBOR can be found at:

https://cbor.io/

An example of cbor library can be found at:

https://github.com/hildjj/node-cbor/tree/main/packages/cbor/lib

DFINITY uses CBOR in the agent-js library as well. The implementation can be found at https://github.com/dfinity/agent-js/blob/main/packages/agent/src/cbor.ts

Github user Nomeata has done some cbor(only on binaries) at https://github.com/nomeata/motoko-certified-http which may demonstrate some good paterns for parsing and constructing CBOR.

We need a CBOR library in motoko because the new call_raw functionality allows a motoko canister to receive Binary Represented data from a server. A wallet may want to filter this call and decide if it wants to pass it on to another IC service by data inside the encoded representation. The cbor plug in will allow a wallet author to inspect the payload and make decisions about the call.

The opposite is also possible, where a wallet may want to be able to construct a payload for call raw using an uploaded candid definition for a service that did not exist when the wallet was minted.

Once CBOR is decoded it needs to be synced with the cadid representation of the service it was encoded for. If you want to be able to tell what property is what in your function call package you need to know the candid that was used to encode the package.

The Candid parsing logic can be found for the agent-js library at: https://github.com/dfinity/agent-js/blob/main/packages/candid/src/candid-core.ts

These libraries will often be used together to parse and encode binary data.

The developer is welcome to attempt to refactor the Candy Library at https://github.com/aramakme/candy_library to host the candid parser as that was one of the initial intents of that library.

A full set of tests should be implemented.

The package should be deployed as a vessel package.

To apply for this bounty you should:

  • Include links to previous work writing tutorials and any other open-source contributions(ie. your github).
  • Include a brief overview of how you will complete the task. This can include things like which dependencies you will use, how you will make it self-contained, the sacrifices you would have to make to achieve that, or how you will make it simple. Anything that can convince us you are taking a thoughtful and expert approach to this design.
  • Give an estimated timeline on completing the task.
  • Post your application text to the Bounty Thread

Selection Process

The ICDevs.org developer’s advisors will propose a vote to award the bounty and the Developer Advisors will vote.

Bounty Completion

Please keep your ongoing code in a public repository(fork or branch is ok). Please provide regular (at least weekly) updates. Code commits count as updates if you link to your branch/fork from the bounty thread. We just need to be able to see that you are making progress.

The balance of the bounty will be paid out at completion.

Once you have finished, please alert the dev forum thread that you have completed work and where we can find that work. We will review and award the bounty reward if the terms have been met. If there is any coordination work(like a pull request) or additional documentation needed we will inform you of what is needed before we can award the reward.

Bounty Abandonment and Re-awarding

If you cease work on the bounty for a prolonged(at the Developer Advisory Board’s discretion) or if the quality of work degrades to the point that we think someone else should be working on the bounty we may re-award it. We will be transparent about this and try to work with you to push through and complete the project, but sometimes, it may be necessary to move on or to augment your contribution with another resource which would result in a split bounty.

Funding

The bounty was generously funded by the DFINITY Foundation. If you would like to turbocharge this bounty you can seed additional donations of ICP to d0e3be904a939ff752979aaa53cfb66fac8d75b85ce4fa878e147d60706be6ed. ICDevs will match the bounty 5:1 for the first 24 ICP out of the DFINITY grant and then 0.25:1 after that. All donations will be tax deductible for US Citizens and Corporations. If you send a donation and need a donation receipt, please email the hash of your donation transaction, physical address, and name to donations@icdevs.org. More information about how you can contribute can be found at our donations page.

General Bounty Process

Discussion

The draft bounty is posted to the DFINITY developer’s forum for discussion

Ratification

The developer advisor’s board will propose a bounty be ratified and a vote will take place to ratify the bounty. Until a bounty is ratified by the Dev it hasn’t been officially adopted. Please take this into consideration if you are considering starting early.

Open for application

Developers can submit applications to the Dev Forum post. The council will consider these as they come in and propose a vote to award the bounty to one of the applicants. If you would like to apply anonymously you can send an email to austin at icdevs dot org or sending a PM on the dev forum.

Assigned

A developer is currently working on this bounty, you are free to contribute, but any splitting of the award will need to be discussed with the currently assigned developer.

In Review

The Dev Council is reviewing the submission

Awarded

The award has be been given and the bounty is closed.

Matches

DFINITY Foundation Grant: - $3000 USD of ICP at award date

Other ICDevs.org Bounties

5 Likes
2

Hey, I looked into the call_raw method, and it seems the Blob data is the binary representation of candid and not CBOR as the candid blob maps its data types to this format, which is different from the types specified for CBOR.

Instead of producing two libraries, I think there could be a single one that encodes the data from the candy_library types to Blob and then parses the Blob back to the candy_library types

1 Like
3

Hey @skilesare, I would like to apply for this bounty!
I plan to write the encoding and parsing functions in the candy_library repo.

I have a few questions:

  • Currently, the #class type is used to identify general types with a field and a value. This is a problem because the variant and record types would be mapped to the #class type when parsing. To differentiate between these types, I would be replacing the #class type with two new types: #variant and #record. I’m worried that this will break the existing code. What do you think would be a good way to go about this?

  • How can I convert Floating types to bytes in Motoko? I can’t seem to use bit operations on Float types, but I can do so with Int and Nat types by converting them to Int64 and Nat64.

The estimated timeline for the project is one month.

4

Let’s discuss the replacing of class. I’m not sure I understand. Do you mean that candid has a #class? What would be your proposal for that structure? Sounds like some great features!

I think from some discussions we may not need cbor, and some of this may be coming in motoko as a native library. On Tuesday I can look at it more.

I have been looking at adding a dictionary using stableRBtree.

5

I’ve been using this and it works very well Release 0.6.28 · dfinity/motoko · GitHub

Here is the PR Operations `to_candid` and `from_candid` as surface syntax. by matthewhammer · Pull Request #3155 · dfinity/motoko · GitHub

1 Like
6

Is this still up for grabs? I’ve been looking for a project to get deeper with motoko.
I did similar work with Candid/Cbor with GitHub - Gekctek/ICP.NET: ICP libraries for .NET
Most of it was candid implementation with a cbor library, but after doing so much manual binary decoding, I think im ready for some more.

I have something i just started up to create a motoko libarary. GitHub - Gekctek/motoko_cbor

My thought is I was going to create a couple libraries, one for cbor, one for candid.

Are there any use cases I should work toward besides the call_raw? Im thinking just a very generic library for reading/writing bytes will be good for all cases.

7

Yes! Those would be great….I have another bounty that could some of this as well. As soon as I get it approved I’ll let you know.

One interesting use case is to have a canister produce the cbor that needs to be relayed to the network from an t-ecdsa signed message. Once we have http requests, a canister would be able to send signed messages to the replica endpoint itself.

1 Like
8

I should have something usable within the next few days. Im almost done with basic cbor reading right now, but trying to solve a few issues like Float creation from bytes and a good serializer pattern.

Its fun to work in code that has almost no libraries, get to write everything from scratch. The wild west of coding right now lol

9

I was hoping to have something to have reviewed this weekend but my wife is going into labor for our first child, so i doubt I’ll have time lol

1 Like
10

Ha! See you in a couple of months. :grinning: Enjoy this moment and congrats!

1 Like
11

Had a few moments between nap times (mine and his) and was able to get the basic functionality done.

I don’t know how this bounty process works but this would probably be a good time for a initial review. I have implemented bytes -> cbor object and cbor object -> bytes but there is no custom type serialization, its all a manual process.
I have put basic information in the README along with some TODOs

Im curious if anyone has thoughts on how to implement the custom seriliazation. My world has been .NET and that was much different than this. I don’t see any concept of reflection, so im trying to figure out the best way to go forward.

1 Like
12

We’ve discussed this before at Improving Motoko's programmability - #14 by skilesare

I really wish it was natively in the motoko runtime as it would make tasks like this much easier. This library will get us a long way there. It is going to be integral to some of the upcoming bounties as well.

3 Likes
13

// TODO is there a way to convert a Nat to an Int directly?

A Nat is a sub-type of Int so you don’t have to convert it. :slight_smile:

1 Like
14

Any ideas for a good integration test? Are we at the point where we could encode something like a balance request and check the cbor generated against the cbor in a network request from agent.js?

Or better yet…maybe we could just grab one of those and decode them. Maybe I’ll set up a motoko playground later and try it out.

1 Like
15

I love this structural typing concept in Motoko but its taking a minute for my mind to adjust. Keep thinking in terms of inheritance and casting

1 Like
16

Yes and no. The way to do it right now requires a manual step of converting motoko types/classes into a ‘cbor value’ which is just a variant of all the cbor major types. Once its in the form of the ‘cbor value’ it can encode/decode just fine
My next step is to remove that manual process, so ill take a look at the candy_library you linked and see what i can do

1 Like
17

I think you might find this helpful. My attempt at writing a cbor library . I didn’t complete it, but I was able to write methods to encode most of the candid_library types to cbor except for Float.

18

Nice. Ty.
Ya floats were a pain

2 Likes
19

@skilesare
Alright I think I need clarification on a few things because im confusing myself

  1. Is the CBOR encoding a custom thing vs the HTTPS interface?
    From what I understand the CBOR is encoded/decoded in the boundry nodes and doesn’t touch the canister itself. From the tests/examples I have seen for call_raw, the encoding is just the raw candid, not cbor. Is there going to be CBOR inside the candid in blob form?

  2. Looking at the candy_library i get what is happening, allowing the ability to look at properties/meta data of the data itself. BUT what i don’t get is how it goes from a motoko value to a candy value. It seems like its just a manual process, similar to what im doing with Cbor values, where its just a variant storing motoko types with some meta data. Am I understanding it correctly or am I missing something.

  3. How exactly do we want the code to look like with converting to and from cbor in the call_raw method. I see in the description we want to be able to inspect whats inside or build cbor based on a definition, but I think Im having troubling picturing it. For construct a payload for call raw using an uploaded candid definition what does the definition look like? Is it a *.did file? If so is the idea to parse that file, build some kind of descriptor data for it, then use that for validation and creation of candid models? Thats also where im confused because im not sure what that has to do with CBOR, seems like a candid thing.

Any help would be great because im going in circles.
I think some sort of pseudo code of the call_raw function of what you expect/want would be helpful as well.

1 Like
20

Starting the candid portion in a different project

Will work similarly to have I have the CBOR where I will take the Blob and create a candy_library like model holding the candid value/type meta data

1 Like