Thanks for your answer, this time I clearly understand what I should do!
Hi @heyuanxun, any updates?
I just want to provide a progress report for the Motoko side of file uploading. I closely followed the interface from @dfinity/assets. However, I simplified the interface by not creating two distinct methods for uploading files. Previously, there was one method for files smaller than a certain byte size and another for chunking larger files. Now, I have designed it to chunk all files, regardless of size. Many tests demonstrate how this works, and I plan to create a video explaining its usage. I would appreciate feedback from the community regarding any additional features or improvements they would like to see.
Chunking is required over about 2MB. What do you have the chunk size at?
I have chunking size at 2MB
I wrote above. I did the same in my project. If possible (preferably) make it possible to adjust the size of the piece.Since the user needs (not always) to count the hash or checksum. It doesnât always succeed in a round.
I just didnât understand why a developer would choose anything less than 2MB chunking size. I guess in extremely poor or unstable internet connections it might be useful. Going to add the retry to the client, should be easy enough. Anything else that you think might be good to have let me know.
So I added retry logic. I think one thing missing is checksum. However, going to think about this and then add it.
We are MetaBox Teamďźwe will start working on this bounty.
In MetaBox, once we have fulfilled the requirements mentioned above, we will abstract these functions.
MetaBox : app.metabox.rocks
Hey C-B-EliteâŚthe bounty is currently assigned to Cyberowl.
1 Like
Can you take a look at the code above and offer any suggestions that you would find helpful for MetaBox. Thanks
1 Like
Seems like checksum is a good idea here are my approach to doing it:
#Checksum Verification
Client-side (using JavaScript):
a. Compute the checksum of the file before uploading, using a JavaScript library that supports the desired checksum algorithm (e.g., MD5, SHA-1, or SHA-256).
b. Send the computed checksum along with the file to the server.
Server-side:
a. Form File from all the chunks
b. Compute the checksum of the received file using the same algorithm as the client.
c. Compare the server-generated checksum with the client-generated checksum.
If the checksums match, it is highly likely that the file was uploaded correctly and without errors. If they donât match, there might be an issue with the uploaded file, such as corruption or an incomplete transfer.
I thought maybe it would be good to add a sync file mech as well, but that will probably take more time. However, I think a simple checksum will be good for file integrity in case some chunks are injected in some way while uploading the file.
1 Like
I implemented CRC-32 checksums for file uploads. This ensures that if any chunk is missing when committing a batch, an error will be triggered. Additionally, checksums are performed at both the chunk and file levels, allowing for potential synchronization of different chunks in the future. For instance, updating specific parts of an executable file could be made possible. While I havenât tested this functionality yet, itâs an experiment I plan to explore next. Another aspect Iâm interested in trying is implementing gzip compression and decompression on the client side.
2 Likes
Is there any particular reason to why you chose to use transient memory instead of ZhenyaUsenkoâs stable hashmap? Afaik it is more performant than the one used in the base Motoko library despite using stable memory and it would lift the storage limit per canister to 48GB.
I donât think ZhenyaUsenkoâs uses stable memory, it is just a module. There is no particular reason, just not brining in more dependancies than necessary. I still donât understand why we donât support improvement of base hashmap vs creating additional deps. However, if the community prefers that I can change it.
1 Like
Taking into consideration your feedback @Zane I have updated refactor: use dep hashmaps from ZhenyaUsenko ¡ cybrowl/upload-file@904a3e8 ¡ GitHub. to move towards hashmap dep. I think it makes code easier to read as well. Anyway, let me know if you have any other suggestions.
That is great, but just to be sure Iâd also ask @skilesareâs opinion on this one, the additional dependency argument you pointed out makes sense an since he is the bounty issuer he deserves the decide if itâs a worth compromise for more storage capacity.
Are more granular permission per asset planned?
Also remember to update this line to match the new domain, ic0.app wonât be used anymore for new canister after the 20th April: upload-file/utils.mo at main ¡ cybrowl/upload-file ¡ GitHub
1 Like
Yeah I just saw a tweet about the domain name change. I donât think the hashmap dep adds any increase in storage capacity but will add support for stable storage as well next since it seems like a good thing. There are limitations though and I will outline them in the readme. Permission per asset can only be done via token generation and is def something to explore.
Initially, I wrote the @dfinity/file-upload package in JavaScript. I chose this approach because I didnât want to learn TypeScript and didnât see the need for a small library. However, to maintain consistency with agent-js, I decided to rewrite it in TypeScript. I also updated all my tests to be compatible with Jest. I created a new repository to manage changes while testing the package at GitHub - cybrowl/agent-js-file-upload: fileupload pkg. After testing, it can be more easily merged into agent-js. Currently, there is a minor issue I couldnât fix with agent-js, where the Jest tests are not working for me. It seems to be a configuration issue related to the monorepo structure.
The next step is to start using the library in a project, and I am considering building a simple UI for people to see what they can do with the project. Domwoe also suggested that I should add it to Internet Computer Loading, which I plan to do soon.
1 Like
Using Evgineyâs hashmap is great. It is fast and performant. It doesnât do stable storage but takes advantage of the stable migrations provided by motoko so you shouldnât need to do or/post/upgrade.
A stable version that gets you up to 48GB wold be great, but I think we need to let @matthewhammer burn in the stable region stuff(is that ready to go Matt?)
2 Likes