The Internet Computer has a limit of 2MiB on the size of ingress messages on most subnets. This limit also applies to install_code messages when they are sent as ingress messages to install or upgrade canisters. In many cases, this is enough but we are slowly seeing more and more cases where this limit is too low and bigger Wasm modules cannot be installed. There is also the ability to install compressed modules but it only helps up to some extent since in some cases canisters might not compress well enough (or are simply too big even after compression). This poses a big limitation to canister developers and is considered a high priority item to fix from our side.
Sketch of a high level approach
The high level idea is relatively straightforward. We can allow canister installation to happen in chunks where the Wasm module is uploaded over multiple ingress messages (of up to 2MiB) and the installation proceeds once all chunks have been received. This will likely require changes across the stack, from replica to dfx tooling, to allow for a smooth user experience.
Action for the community
We would like the community to provide their feedback on how useful they find this feature and additionally, if they have specific requirements in terms of:
Use cases for large Wasm modules
Wasm module sizes to be supported
Desired interface to allow chunked installation
The feedback will be of great help as we begin scoping the feature in the following weeks.
Tagging some people for visibility (feel free to tag more that you think they might be interested): @lastmjs (who originally asked for this) @dfx-json@mikhail-turilin from the SDK/product side
The certified assets crate does something along these lines for static assets, and I think @peterparker may have also written something for dynamic assets.
I’ve seen a few solutions specifically for Wasm modules. I know @skilesare came up with one.
If possible, I’d love it if this could address the more general problem and not result in a solution that is specific to Wasm modules.
I think having this be backed by general purpose abstractions provided in agent_rs and cdk_rs would be great.
To make sure it’s clear to everyone: our intent here is to provide a protocol level solution for chunked Wasm uploads. I’m aware there are some cases where people have done some form of chunking on the application level for other use cases (e.g. to upload videos to the IC) but that was not in the scope we had in mind.
If possible, I’d love it if this could address the more general problem and not result in a solution that is specific to Wasm modules.
That’s a fair point. My hope is that starting with a solution for Wasm modules can inform us on the viability of the approach and we can then extend to any ingress message, not just wasm installations. It sounds like most people are currently trying to find their own solutions for large wasm installations and this is what we want to address in a more general way on the protocol side as a first step.
That said, if there is a lot of demand for a solution that covers any ingress message in this thread, we can consider expanding the scope of the work, but given that expanding scope can add delays, we probably want to be careful before doing so and maybe iterating in smaller steps to get faster incremental wins is a better strategy.
I echo this sentiment. If we add support for chunking any ingress message at the protocol level, it is well worth the wait.
Almost any serious application that deals with blob data or large transactions in general has to write boilerplate logic to partition data for transmission for this in their application. This is unnecessary and error prone.
Even the exchange rate example that could just have been a SIMPLE example of polling an API with the https outcall API had to implement this additional chunking logic to not exceed the ingress message limit.
Finally, the IC is supposed to be a full stack alternative to traditional cloud and hosting, so having a restrictive limit on networking and having to come up with application level partitioning logic and management is quite frustrating once you start hitting these limits for your use case
Not sure it is what you meant @paulyoung but yes, I use @skilesare approach to upload the code of canister that are spawned at runtime within the canister that will create these. Regardless if Motoko or Rust, I just chunk the wasm (in NodeJs ) an upload these to an update endpoint that append everything to an array of bytes in stable memory (e.g. Motoko).
I want to list some of the issues that having a small Wasm binary limit has lead to for us:
Can’t deploy most of the Python stdlib with Kybra
The main source of compilation time in our Kybra/Azle build process seems to be from ic-cdk-optimizer, which is only necessary to use because of the Wasm binary limit (our compilation times are very long, but could be ~5-10 secs without ic-cdk-optimizer)
gzip is required to deploy Kybra/Azle canisters, adding extra plumbing to our build processes and requiring the user to change their dfx.json
ic-cdk-optimizer has installation problems on various people’s machines when using Kybra/Azle
ic-cdk-optimizer removes the Wasm metadata section, requiring us to also use ic-wasm to re-insert the metadata section after running ic-cdk-optimizer. We are thus required to force the user to install ic-cdk-optimizer and ic-wasm
We can’t get rid of ic-cdk-optimizer because ic-wasm doesn’t compress the Wasm binary enough (at least for Kybra)
As an early Kybra user I also want chunked Wasm uploads. There are many Python dependencies that Python devs will want to include in their canisters that will easily go over the 2Mb limit right now. And I don’t see dependencies being able to be supported (standard lib or external dependencies) without this.
I also agree that implementing a general ingress message chunking solution at the protocol level would be helpful, as it is annoying to implement right now and would simplify large asset transfer.
I’m biased toward focusing on the Wasm solution (as proposed) initially as this is a major blocker for all Kybra projects, and it isn’t too crazy to implement a chunking solution for assets using http streaming.
I am also biased towards focusing on removing the Wasm binary limit ASAP for the great detriment it is to most of what Demergent Labs is working on currently.
Is this feature currently under development? When could it be expected?
I would like to install certain wasm Blobs using the management canister install_code method but in some cases even the gzipped Blob does not seem to fit within the 2 MiB ingress message limit.
Hi @brady, no the development for this hasn’t started yet. It’s hard for me to give a good estimate at the moment but I think some time in the first half of the year is a doable target.
For transparency, the team that can work on this is wrapping up some last work items on BTC integration and then we still need to start working on some other high priority projects (like a solution for safe canister upgrades which is a big pain point for canister developers). This all might push this one a bit later in the first half since unfortunately we have limited engineering resources.
In the hopes of giving more motivation to prioritize this: the small Wasm limit is quickly becoming the #1 issue holding back especially the Kybra Python CDK but also Azle.
We can’t allow developers to use the Python stdlib and thus almost no PyPI packages, and our build/compilation processes are very long mostly because of dealing with optimizing the binary and the ensuing issues.
This problem is becoming very potent. When discussing Kybra we often have to explain the limitations with the stdlib and PyPI packages.
Oh, and we also can’t upgrade Kybra to use async/await instead of generators because we can’t use the stdlib, something we’d like to do to get Kybra closer to its final API rather than keep changing it on everyone.
@lastmjs Thank you for sharing this information. I will get back to the team internally and share this feedback and see what we can do and if we can push this up somehow.
There were some items which are important but we had decided to not block launch for them and tackle them shortly after. For example: more robust testing of upgrading the BTC canister on CI, cleaning up the old replica implementation that was not used anymore, using stability count for counting confirmations instead of longest chain, a couple minor fixes after a security review that was conducted internally and some other minor improvements for the BTC canister in general now that’s fully productionised.
I thought that work was essentially completed with the launch of ckBTC?
Note that the team working on ckBTC is different than the one working on the core BTC integration which includes the BTC canister which is what I am referring to above and the follow ups we worked on.
Are there security concerns for large wasms? I had always assumed the small size was so that an attacker couldn’t attack a subnet via this method. I believe that the current cross-subnet communication limit(~3.x MB) is what restricts the wasm installation since you have to send the wasm from a staging canister to the management canister and then out to the destination subnet.
If the wasms get “big enough” I’d guess they could be used to choke up a subnet or two with data processing. I have no idea what these costs might be as I’d imagine it would be an expensive attack, but it seems like something that should be addressed in general.
A few more issues in favour of removing the Wasm limit and its impact.
If someone wants to transfer ICP or deploy canisters dynamically or use Internet Identity in their project, developing locally gets quite involved. That’s partially due to having to setup the canisters locally (getting the wasm files, understanding the required deploy arguments, switching candid files before and after deployment and having a separate scripts to deploy in different environments). To bypass the wasm limit they have to deploy a deployer canister and managing deploy scripts and add to an already elaborate process.
After updating to dfx 12 (from dfx 11) some of the wasm files got around 30% bigger (from 1.6mb to 2.1mb) and went over the Wasm limit. Although the change log states that the actual optimisation hasn’t changed in practise it did. About the update refactor-optimize-from-ic-wasm. Probably related to make-ic-wasm-as-capable-as-ic-cdk-optimizer.
@skilesare Maybe we need to clarify a couple things.
First, the limit that you refer to about cross-subnet communication is related mostly to how much data we can put in a consensus block (eventually all cross-subnet messages get in a block at the destination subnet so they can be delivered to the upper layers of the IC and get processed). And yes, indeed, given that such cross-subnet messages that are trying to install canisters would be adhering to the same limit as other inter-canister messages, they cannot be more than 2MiB.
The proposed solution in this thread would add the ability to upload a large Wasm module by chunking it in smaller pieces that fit within the allowed limit for ingress messages (which are typically the same as for inter-canister calls). We wouldn’t increase the limit for individual messages, either ingress or inter-canister. Rather, we’d use multiple (and specifically ingress messages, the solution was not targeting inter-canister calls) to upload a larger Wasm.
Note however that a canister can already send a Wasm over multiple cross-subnet calls (by chunking it up) to another canister which would then install the module on a canister in the target subnet but I don’t see any issues there personally.
There are of course some considerations when allowing larger Wasm modules:
If we allow uploading in chunks, this would mean that we would need to keep the intermediate chunks until we receive all of them and can reconstruct the Wasm before installing it. This is some extra overhead but I believe we can incorporate this to the memory used by the target canister which the canister pays for anyway.
We would need to ensure that a large Wasm module does not result in unreasonable compilation times.
Final note: for new features specifically we always consult with the product security team to get a sense of the risk of the change and such security concerns are usually revealed in this process. So, even if I’m missing something now rest assured that there will be a thorough review of any design to ensure we are not introducing new attack vectors.
Thanks for the feedback @LiveDuo! Your second point sounds like a more specific feedback for the SDK team and the regression from dfx 11 to 12. Afaik, the team is aware of the switch from ic-cdk-optimizer to ic-wasm resulting in bigger modules and they are working on getting this fixed.
Sure thing @dsarlis. A temporary solution at this moment probably is running ic-cdk-optimizer manually if there isn’t any issue feeding the optimised wasm from ic-wasm. I hope the team comes up with a solution soon.
About the first point I’ll create another topic at some point to get more into details and possible solutions.
) an upload these to an update endpoint that append everything to an array of bytes in stable memory (e.g.