Releasing the source of the BigMap PoC demo

Hello everyone. I’m writing to let the community know that the DFINITY foundation has released the source code for the BigMap PoC demo.

Some time before launch, a group of developers at DFINITY had worked on a PoC for BigMap and this was showcased at the Sodium release.

Currently the DFINITY foundation is not actively working on the BigMap project. We are releasing the source code to serve as an inspiration for the community to build similar projects. As such, the project is not maintained, no support is provided, and we will not be accepting pull requests on it. Instead we encourage prospective developers to fork the project to build their own solutions.

17 Likes

awesome, I will keep focus on this project
Here is a similar project I wrote recently (motoko) : GitHub - Di-Box/Storage

4 Likes

Someone asked on Reddit why the foundation is not working on BigMap anymore.

I want to post what I wrote there:

It’s no longer working on this design version of BigMap… there are plans for a new one. We realized there are issues with the posted Proof of concept that required a different design. There are still plans for BigMap implementation (but with different designs).

I cannot speak to what was lacking in this design (I will ask the team to expand more), but this brings up an excellent point: to build the Internet Computer, the foundation has built many, many, many prototypes, proof of concepts, demos, designs, and thrown them away. I have seen in my 3 years at least 3-4 versions of the IC before actual Genesis. It’s been a lot of dead ends combined with hard work of saying “that doesn’t work. Let’s try again.

I will ping team to see if folks can expand what were the design or implementation points lacking in this version.

I hope that’s a helpful answer.

@akhilesh.singhania Do you think that is a fair assesment of the situation?

7 Likes

Incredible, that was a very fast turnaround. Thank you for getting this done.

1 Like

Thanks for the nice context Diego. I would like to add some more of my personal opinions here as well.

As much as the foundation would like to work on various different projects, we do have limited resources in terms of engineers and we have to work on the most urgent and highest priority projects first. Having an actual production network means that a huge amount of resources are needed to maintain and stabilise the network. A lot of this work happens in the background and the end users may not see the immediate benefit of it. However, the benefits are significant in terms of reducing the occurrences of high severity issues; improving monitoring and observability of the network; addressing technical debt so we are more confident about the correctness of the code and can build new features easily; etc. In many cases, we are forced to prioritise such works over working on new features.

Further, my opinion is that the project of increasing the stable memory size allowing canisters to soon hold up to 300GiB of storage may have potentially helped further reduce the urgency of BigMap. I have not seen many canisters in production with a huge state so suspect that it will be some time before we see canisters that need to store more than 300GiB of state.

And finally, I would love to see the community actually build / experiment with the different designs of BigMap. I don’t think anyone can claim to know for certain what precisely is the best way to build BigMap that can span span 100s of subnets and manage petabytes of storage. So we need to explore different ideas.

7 Likes

Agreed… till the Internet Computer has a streamlined solution for enabling TiB of storage, it will struggle to support such applications.