Infinite Storage on the IC - Some Ambitious Ideas

Dear Ladies and Gents of the IC Blockchain,

I am in the process of pitching a decentralized application for storage, a way to create unlimited space for decentralized apps and regular web2 apps.

Something that has some similarities to STORJ https://www.storj.io/ but is built with newer and more reliable technology.

In short, the layer2 would enable:

  • An S3 like fast layer for object storage of files
  • An S3 like way to store, retrieve, and maniuplate storage
  • The IC nodes would operate as the storage network

Is this feasible using the current version of the IC?

    1. Technically feasible using Rust, and canisters?
    1. Cost efficient enough to S3 storage?

The benefits would be huge for IC Apps and also for other decentralized apps.
But is it possible? What do you think?

Joseph Hurtado
Founder
Granata Consulting LLC

4 Likes

Interesting. I would love to know if you could build something like this on IC as well. If we are to build everything on the blockchain then this is something that will be needed.

1 Like

Asking some of the experts I know in the IC perhaps they can chime in. Can you ladies and gents check the post I did above, would the IC be appropriate for a decentralized storage network today, with the current node infrastructure?

1 Like

Is the data that is being stored accessible directly to other canisters? And is the plan that other canisters make use of it? Or are the consumers outside the IC (web2)?

I think the main benefit of storage on the IC is that other canisters have access. If that benefit is gone then it is probably not cost effective for large volume of data. For small volume it may not matter because the convenience may dominate over cost.

Keep in mind that storage cost on the IC is subsidised by other fees such as computation and data transfer in inter canister calls. If those things don’t happen, if the only access to the data is by query calls from outside the IC, then the IC cannot collect any fees except for the storage cost itself (maybe $5 per GB per year). If you occupy a whole subnet with data and the subnet can’t do anything else then it will be underpaid. If you store terabytes then the IC will have to do something about it (raise fees).

This may change with dedicated, cheaper storage subnets in the future.

I have been thinking about this for HPL and the transaction archive. 10k transactions per second are in the order of 1 billion per day so that would be 10-100 GB per day. We would be filling up one subnet (700 GB) per week with transaction history. That is not sustainable. So the only solution for that kind of volume is off-chain storage (with hashes on chain).

3 Likes

Timo, thanks for your answer I will reply after your quotes:

  • Yes, our first intention is to create an S3 like Storage Object Store that will serve DFINITY developers, they are one of our main audiences.
  • The ambition was to not stop there, and reach out other developers, primarily those in the Blockchain space, but since those work mostly off-chain they qualify as Web2 developers.
  • Compute would be used in the IC for the main app measurement of traffic being stored and served, so that the cost to the user is fair, and the network can be paid as well using ICP.
  • Compute would also be used to aggregate storage canisters into clusters, so that we can have large units of storage, not just small canister level ones.
  • Current network demands on the IC are not heavy on storage, mainly because it is very hard to use, plenty of code is needed to use it. Our bet is that building the infrastructure to make it easy will unlock this resource so first the IC developers can use it, and next other developers outside the IC.
  • The fact that fees can go up is a risk I do see, and we are thinking ways to address that if it becomes a real showstopper. A way to limit growth would be to just focus on the IC, but this seems a very small market. Our compromise vision is to target only developers, not end users. That market is smaller, but needs great developer experience, and that could be the main advantage, especially given the IC support of identity, BTC and ETH.
  • Off-chain storage is a possibility we are exploring too, where the solution is a hybrid. The IC providing Identity, BTC and ETH support (wallet, transaction level) and the rest being run in off-chain storage in clusters we run, and may get investors to run.

Overall Timo, what do you think, is this a project DFINITY would welcome? We are in the initial stages of a VC search.

I think I don’t understand your application well enough. Do you have an example? Who would use it and how exactly?

1 Like