So the IC can't store files well either?

I’ve noticed dscvr and dsocial don’t actually store post content on ICP, they use IPFS and arweave respectively.

It seems terrible to me that the IC requires thousand+ dollar nodes with multi-terabyte data storage all in a blockchain of only 13 nodes and it’s still seemingly inferior to hosting on arweave and ipfs.

It seems like the IC is going to be beat out by more decentralized solutions as it’s monolithic design seemingly hasn’t yielded any benefits over more decentralized ones.

3 Likes

lol Arweave

We’re building out a multi-terabyte DynamoDB type solution for our upcoming game, Dragginz. Every single part of this game (database, assets, logic) is on the IC.

I think dscvr and dsocial were a bit too early to take advantage of all the progress the IC team has made since they launched, but I wouldn’t be surprised to see them move back when a few of the issues have been ironed out.

10 Likes

Can you clarify a little bit more about how are you scaling it? The biggest constraint to me is that IC currently does not support intercanister query calls, so, how do you scale terabytes in database without awating multiple seconds for a simple query?

1 Like

Well, inter-canister query calls would be nice, and yes parts of our roadmap are held up as a result. We’re trying to design around that as best as we can.

We’re using CanDB, the first scalable NoSQL Database on the Internet Computer, but modified as our data model is frankly ridiculous so I have to use Go to generate all the code.

We’re not using the “auto-scale” function of CanDB just yet because it would just make things overly complex in our system. What we’re doing is splitting up the data tables into different microservices, and then looking up the canister by a modulo of the ID.

The main canister creates the other canisters required in the schema, and has an endpoint that tells you the canisterID of a record+ID combination (which the client should really cache).

Each service manages the CRUD API calls for each associated Record.

It’s probably overly complex for now but it’s working and we’ve got a mostly-clear roadmap for the next 6 months.

That’s where we’re at. API calls that hit many different canisters are going to be a problem, but we’re not in a rush to launch anything.

6 Likes

DSCVR will use IC as our file storage. As @borovan said, lots of progress has happened recently to reduce the development friction for us to be able to do this and we will simply migrate to them.

7 Likes

For sure not the same scale as these amazing projects above but, my personal project Papyrs does already use the IC as file storage. If you upload images, they are saved in custom asset canister.

Speaking of, regarding scalability, each of my users get personal smart contracts - i.e. canisters are generated on the fly for the users. So in that sense, I think there are ways to make the architecture scalable.

Thanks for clarifying,that was a smart solution !

I think the comment about arweave is a bit derisive, but I guess it is expected as a decentralized DB. Where is the reason to ridicule arweave, where arweave can’t go as far as hosting? I would like to know the difference between IC and arweave.

Didnt Solana have to switch saving its transaction history from Arweave to AWS because Arweave was too expensive?

This might be relevant as well:

There was also an effort to create specialized storage subnets, but my feeling is that has stalled indefinitely.

Here at Distrikt we are effectively fully on chain from the start. So in order to achieve this we use two different approaches as we have need for horizontal and vertical scalability.

  • Our assets are stored using a bigmap fork that was built by Dfinity and it works fine as long as you do no maintenance (moving stuff around when adding a new bucket for example) because there is too much instructions to process. So we use a fixed amount of bucket canisters and offload / upload all the assets when we want to add more buckets.

  • As of now all our BE data is in one canister but we are currently working on our new backend infra that will split our components into canister pools (vertical scaling) and those pools will auto scale over time (horizontal scaling) by spawning clone of themselves (state excluded) in order to store new data.

6 Likes

Projects on the IC need to advertise themselves appropriately. You can’t claim fully on chain if you are using IPFS or even Arwewave. The biggest advantage of the internet computer is the ability to run fully decentralized applications, preferably on chain on the IC. If not you might as well use Lens protocol for socials.

1 Like

There was a question on the forum months earlier that was never addressed by Dfinity. Would be good to get an answer to it.

1 Like

MODCLUB stores all data on the chain. We auto-scale our storage buckets and delete data after a grace period. We store audio, video, images etc…

4 Likes

Seachan doesn’t store files on-chain either
embarrassing

Where do they store files? Seachan is super slow.