Assigned: ICDevs.org - Bounty #45 - File Uploader Pattern - JS, Rust, Motoko - $10k

File Uploader Pattern - JS, Rust, Motoko - #45

Current Status: Discussion

  • Discussion (01/09/2023)
  • Ratification: (01/09/2023)
  • Open for application: (01/09/2023)
  • Assigned
  • In Review
  • Closed

Official Link

Bounty Details

  • Bounty Amount: $10,000 USD of ICP at award date.
  • ICDevs.org Bounty Acceleration: For each 1 ICP sent to 88159b4fcc365576856c3d9c12a8d5b0e9614660dd30d507894466dabd58e92b, ICDevs.org will add .25 ICP to this issue and .75 ICP to fund other ICDevs.org initiatives.
  • Project Type: Individual
  • Opened: 01/09/2023
  • Time Commitment: Weeks
  • Project Type: Library
  • Experience Type: Intermediate - Motoko; Intermediate - Rust; Intermediate - JS;

Description

Uploading files to the IC has been difficult up to this point. The root cause of this is that the IC has an ingress limit of 2MB. Therefore, if you need to upload a file, you need a custom integration that chunks the files into 2MB chunks and handles the processing and repeated calling of the IC to get the file into your canister.

This bounty call for the developer to create a best-practices tutorial, motoko implementation, rust implementation, and agent-js handler to simplify this function.

The example canisters should have the following interface:

com_uploader_chunk({bytes : [Nat8], file_id: ?Nat, chunk : Nat}) -> Nat; 

This function should assemble files into a Map of <Nat, [[Nat8]]> or <Nat, Buffer<[Nat8]>]>.

The canister author is free elsewhere to handle the association of the file_id with a file name.

Code elsewhere in the library should be able to retrieve the files in chunked format from this state variable.

The developer should add a function to the dfinity agent that auto handles the uploading of a byte array into this function. Under the hood there is likely much code you can repurpose from @dfinity/assets - npm. you should adapt this library so that its not as specific for the asset canister and returns/notifies of the file id. The following offers another approach mintjs/stage.ts at 3cba559c5cd20e233aec211651cc42c59f9504d3 · ORIGYN-SA/mintjs · GitHub.

Create a package in agent-js/packages at main · dfinity/agent-js · GitHub called fileUploader. This should take an actor and an identity. You should be able to pass it a file and await the upload of that file:

let file_id = await fileUploader(actor, bytes);
let result = actor.notify_of_file(file_id, "myfile.txt");

This bounty gives the opportunity to

  • learn about Motoko
  • learn about Rust
  • learn about IC File uploads

To apply for this bounty you should:

  • Include links to previous work writing tutorials and any other open-source contributions(ie. your github).
  • Include a brief overview of how you will complete the task. This can include things like which dependencies you will use, how you will make it self-contained, the sacrifices you would have to make to achieve that, or how you will make it simple. Anything that can convince us you are taking a thoughtful and expert approach to this design.
  • Give an estimated timeline on completing the task.
  • Post your application text to the Bounty Thread

Selection Process

The ICDevs.org developer’s advisors will propose a vote to award the bounty and the Developer Advisors will vote.

Bounty Completion

Please keep your ongoing code in a public repository(fork or branch is ok). Please provide regular (at least weekly) updates. Code commits count as updates if you link to your branch/fork from the bounty thread. We just need to be able to see that you are making progress.

The balance of the bounty will be paid out at completion.

Once you have finished, please alert the dev forum thread that you have completed work and where we can find that work. We will review and award the bounty reward if the terms have been met. If there is any coordination work(like a pull request) or additional documentation needed we will inform you of what is needed before we can award the reward.

Bounty Abandonment and Re-awarding

If you cease work on the bounty for a prolonged(at the Developer Advisory Board’s discretion) or if the quality of work degrades to the point that we think someone else should be working on the bounty we may re-award it. We will be transparent about this and try to work with you to push through and complete the project, but sometimes, it may be necessary to move on or to augment your contribution with another resource which would result in a split bounty.

Funding

The bounty was generously funded by the DFINITY Foundation. If you would like to turbocharge this bounty you can seed additional donations of ICP to 88159b4fcc365576856c3d9c12a8d5b0e9614660dd30d507894466dabd58e92b. ICDevs will match the bounty $40:1 ICP for the first 125 ICP out of the DFINITY grant and then 0.25:1. All donations will be tax deductible for US Citizens and Corporations. If you send a donation and need a donation receipt, please email the hash of your donation transaction, physical address, and name to donations@icdevs.org. More information about how you can contribute can be found at our donations page.

FYI: General Bounty Process

Discussion

The draft bounty is posted to the DFINITY developer’s forum for discussion

Ratification: (01/09/2023)

The developer advisor’s board will propose a bounty be ratified and a vote will take place to ratify the bounty. Until a bounty is ratified by the Dev it hasn’t been officially adopted. Please take this into consideration if you are considering starting early.

Open for application

Developers can submit applications to the Dev Forum post. The council will consider these as they come in and propose a vote to award the bounty to one of the applicants. If you would like to apply anonymously you can send an email to austin at icdevs dot org or sending a PM on the dev forum.

Assigned

A developer is currently working on this bounty, you are free to contribute, but any splitting of the award will need to be discussed with the currently assigned developer.

In Review

The Dev Council is reviewing the submission

Awarded

The award has been given and the bounty is closed.

Other ICDevs.org Bounties

1 Like

I have experience with Motoko and Javascript in my previous projects. I am interested in taking on this bounty. However, I was wondering if it would be possible to complete the work for Motoko and Javascript at a reduced cost, since I am already familiar with those languages. While I am willing to learn Rust, I believe it would be more efficient if someone who is already proficient in Rust could handle that part of the work. Could we consider splitting the work in this way?

Below, I have provided a brief example of my previous implementation, but I would like to improve it, add documentation, and expand it to meet the requirements of this bounty.

This code doesn’t show the integration and unit tests I have for this implementation. You can also see a project that uses this to scale file assets across many canisters at the following:

https://kqlfj-siaaa-aaaag-aaawq-cai.ic0.app/
https://kqlfj-siaaa-aaaag-aaawq-cai.ic0.app/snap/7GWF5PA4PHG55MC2R2RSVXN41D?canister_id=lrr5x-jaaaa-aaaag-aatzq-cai

Dependancies I will use:
mo:base

Dependancies I might consider:
aviate-labs/ulid.mo to generate id (although might not be necessary)

I would not use any dependancies unless it is absolutely necessary. For this task I don’t think anything other than base is necessary.

  • chunking files into 2MB chunks and handling the processing and repeated calling of the IC to get the file into the canister.

    solution: to some extent this is done via the following
    GitHub - cybrowl/upload-file: uploading file assets

    I think if we want scaling I can build an approach to that as well. The description asks for one canister.

  • create a package called fileUploader in agent-js/packages that takes an actor and an identity, and allows to pass a file and await the upload of that file. Additionally, the developer should adapt existing libraries such as @dfinity/assets and mintjs/stage.ts to make them more general and to return/notify of the file id.

    solution: my understanding is that the actor would contain both the canister_id and the identity to upload the file. Adding this package shouldn’t be a problem. Looking at @dfinity/assets - npm. It seems like much of what we want to use is there. I think adding great tests to all this will best showcase how it works.

This will take me a couple of weeks to complete. Most likely 2 weeks. There are still a couple of questions that I would like clarification on.

I don’t understand what this is supposed to do:

let result = actor.notify_of_file(file_id, "myfile.txt");

What is result?

For sure. If you can take care of the JS and Motoko we can do 2/3 of the bounty and attempt to recruit a Rust dev to help you for the other 1/3.

I’ve asked for approval and I expect it to pass:

1 Like

Hi!
I already have such loaders. There are several options in my project. One of the advanced forms shows progress. The other one just loads the file and shows completion. It is possible to check the integrity of the file by CRC and without it. Similarly, file acceptance is implemented in motoko.

Link github https://github.com/fury02/Difibase
Web https://7wwjw-5iaaa-aaaan-qbguq-cai.ic0.app/

I can do it. Implemented in TypeScript/JS and Motoko

Safik…this looks super cool. The bounty is currently assigned to @cyberowl. Please check with him. If you guys want to coordinate and share the bounty it would be up to him. It looks like you’d basically just need to extract the relevant pieces into their own repo that could be easily included with vessel or mops along with an example. Then wire up the TS into the agent. Seems pretty straightforward, but I’ll leave it to cyberowl on how to proceed.

Ok @skilesare. Let’s not get in the way @cyberowl. Then let’s let him do it alone. If my examples help, let him put stars on github :slightly_smiling_face:

@skilesare I have a request for you. Consider me for this task. Definitely I want to solve it.

Hello @Safik, I hope you are able to claim bounty #39. I have already reached out to @LightningLad91 and we are planning to meet on Friday to discuss the design for file uploader. I have suggested splitting it between the two of us.

I was not aware that I had already been assigned the bounty, proposal appears to be active. Regardless, this is the current status. I will update you after our meeting.

1 Like

You can develop everything from scratch. I hope this will be the best solution. But there is a great opportunity to focus (when solving) on my project. I don’t mind if you use my modules. But, like any code, it requires refactoring. In any case, need there is work on the design of the code.

1 Like

So I am a bit confused how this process works. How do I know I got approved and can proceed with the work. Thanks @skilesare

Doh…sorry…let me repost. We need to do a better job voting as a board. I’ll bug people about it. Please keep working.

1 Like