Interesting. I would love to know if you could build something like this on IC as well. If we are to build everything on the blockchain then this is something that will be needed.
Asking some of the experts I know in the IC perhaps they can chime in. Can you ladies and gents check the post I did above, would the IC be appropriate for a decentralized storage network today, with the current node infrastructure?
Is the data that is being stored accessible directly to other canisters? And is the plan that other canisters make use of it? Or are the consumers outside the IC (web2)?
I think the main benefit of storage on the IC is that other canisters have access. If that benefit is gone then it is probably not cost effective for large volume of data. For small volume it may not matter because the convenience may dominate over cost.
Keep in mind that storage cost on the IC is subsidised by other fees such as computation and data transfer in inter canister calls. If those things don’t happen, if the only access to the data is by query calls from outside the IC, then the IC cannot collect any fees except for the storage cost itself (maybe $5 per GB per year). If you occupy a whole subnet with data and the subnet can’t do anything else then it will be underpaid. If you store terabytes then the IC will have to do something about it (raise fees).
This may change with dedicated, cheaper storage subnets in the future.
I have been thinking about this for HPL and the transaction archive. 10k transactions per second are in the order of 1 billion per day so that would be 10-100 GB per day. We would be filling up one subnet (700 GB) per week with transaction history. That is not sustainable. So the only solution for that kind of volume is off-chain storage (with hashes on chain).
Timo, thanks for your answer I will reply after your quotes:
Yes, our first intention is to create an S3 like Storage Object Store that will serve DFINITY developers, they are one of our main audiences.
The ambition was to not stop there, and reach out other developers, primarily those in the Blockchain space, but since those work mostly off-chain they qualify as Web2 developers.
Compute would be used in the IC for the main app measurement of traffic being stored and served, so that the cost to the user is fair, and the network can be paid as well using ICP.
Compute would also be used to aggregate storage canisters into clusters, so that we can have large units of storage, not just small canister level ones.
Current network demands on the IC are not heavy on storage, mainly because it is very hard to use, plenty of code is needed to use it. Our bet is that building the infrastructure to make it easy will unlock this resource so first the IC developers can use it, and next other developers outside the IC.
The fact that fees can go up is a risk I do see, and we are thinking ways to address that if it becomes a real showstopper. A way to limit growth would be to just focus on the IC, but this seems a very small market. Our compromise vision is to target only developers, not end users. That market is smaller, but needs great developer experience, and that could be the main advantage, especially given the IC support of identity, BTC and ETH.
Off-chain storage is a possibility we are exploring too, where the solution is a hybrid. The IC providing Identity, BTC and ETH support (wallet, transaction level) and the rest being run in off-chain storage in clusters we run, and may get investors to run.
Overall Timo, what do you think, is this a project DFINITY would welcome? We are in the initial stages of a VC search.