Make local development experience similar to the mainnet

Summary

Local development with dfx differs from how the things actually work on the mainnet quite a bit. The proposal is to make local dfx replica as close to the real deal to provide better development experience.

Note

I do only elaborate on missing cycles consumption mechanics on the local replica and encourage the community to extend this topic with any addition they see important. I remember there were some proposals to incorporate critical mainnet canisters into the local replica also - feel free to elaborate on that topic as well.

1 Cycles on dfx are ignored completely

Problem A - cycles consumption estimation

There is no way for me to estimate how many cycles does my application consume, except to deploy it to the mainnet, which is not cheap and not always easy. Especially in scenarios with multiple canisters deployed dynamically (since I need to write some additional logic to reuse these deployed canisters) - which is both: expensive and not so easy sometimes.

Cycles usage estimation is a critical functionality IMO since it provides us with a quickly way to check if the chosen algorithm performs well or there are still optimizations to be done, especially for people who came from other platforms and don’t know the IC’s economy capabilities yet.

Problem B - cycles consumption limit per message

There is no way to check message boundaries in terms of cycles consumption per message, except to deploy the canister to the mainnet and to perform an extensive testing. For those who don’t know, there are two kinds of limits per each message you send to a canister: message size limit (~2mb) and cycles consumption limit (the actual value doesn’t mean anything since there is no way for you to check it locally rn).

While the first limit is checked correctly locally (this number actually differs on the mainnet and on the local replica - the latter is about 3.5mb on dfx 0.8), the second limit is ignored completely which leads to unexpected bugs in production. This problem is not hard to solve, but the solution implies limiting the size of payload you send from frontend in one go, which could lead to some serious work you need to do, to always be sure you only send the right amount of data for canister to process at once.

Problem C - you don’t need a wallet canister to deploy locally

We recently did an experiment with on-the-fly canister deployment straight from the frontend code. Locally everything worked fine - call provisionally_create_canister_with_cycles() (which actually works only locally, in production you would call create_canister()) and you’re done. You don’t need a wallet canister for that.

But in reality this is not how it works. For a canister deployment you have to hold some cycles, but your keypair-based principal can’t hold them, you need a canister that would do it for you. It could be a wallet canister or any other canister that would pay for your create_canister call. The thing is that this is what we should have see at the beginning of our experiment locally, but only did on the mainnet.

Solution

Implement all cycles-related mechanics on local replica with an ability to switch them off on demand.

8 Likes

This makes sense to me. Of course, this is easy for me to say since I would have to do no work for this. They are the ones actually doing the hard work and also the real experts.

Let me ping SDK team…

4 Likes

Your work is also a great value, Diego :wink:

1 Like

+1000 Cycle management simulation would be huge. I know that mainnet is ‘cheap’…but it isn’t that cheap when you start doing things that threaten the cycle limit. When you couple this with the fact that people can find your canisters and start pinging them with data when you’re just trying to test something it makes a testnet really, really desirable. Being able to simulate locally would be a big step in the right direction.

I don’t even know if it has to be perfect, but being able to run a process that overruns the cycle limit by a factor of 500 locally is not a good developer experience when you’re just learning and get to the point where you push to mainnet and your world devolves into a fiery armageddon of code patching, pseudo logging, and praying that you can identify the rogue loop before the world discovers what tripe your dev skills are. Not that that has ever happened to me. But I could see it happening.

3 Likes