It’s the weekend now, how about the progress?
We will submit the proposal to allow canisters to transfer ICP on Monday November 22, 2021 at 15:00 UTC.
This proposal implements community motion 20588 to enable all principals including canisters to transfer ICP.
To vote YES means you agree to adopt the proposal.
To vote NO means you disagree with or want to delay the proposal.
Voting: The IC uses a wait-for-quiet mechanism such that if there is a stable preference for adoption or rejection and a 3% threshold of votes is met the vote can be decided after 24 hours. If no stable preference appears, wait-for-quiet keeps extending the deadline up to a maximum of 48 hours. The DFINITY Foundation will abstain from early voting to let the broader community’s preference be expressed.
Performance: During testing we saw around 7 seconds latency in ICP transfers.
Integrity: We are not aware of any ways to do a Wasm jailbreak or otherwise break the integrity of ICP on canisters. We believe that without mitigation there would be high risk of a Wasm jailbreak within a 12 months period as in the past there has been a bug of the sort that could potentially assist an attacker (the IC never used the affected version). We are implementing sandboxing of canisters to guard against Wasm jailbreaks with an ETA at the end of December.
Availability: In our security analysis we looked at the deterministic replicated state and identified some general risks, which I have earlier referred to as ‘congestion’. The impact of those could be e.g. be a delay in a canister transferring ICP or a user input not being delivered to a canister smart contract. We have mitigated the highest priority concern, which was that inter-canister messages were prioritized over external ingress messages and hence flooding a subnet with inter-canister messages could cause a DoS of external users, by implementing a round-robin schedule between different types of senders. We have a mitigation for the second highest priority which we expect to roll out in a few weeks, and a mitigation for the third highest priority in the works. The risk with lowest priority is not yet resolved. Given we have found several risks it is reasonable to assume that additional unknown risks affecting availability exist.
28 Likes
I am not sure about how this categorization of “high risk” in Integrity came from. I think that the reasoning must be made transparent because of attribution, in part, to a bug (see below).
The bug, in question, specifically refers to “a potential sandbox escape in a Wasm program”. Not sure about how sand-boxing,as a mitigation strategy, would solve that.
2 Likes
Thanks for the update.
A couple questions:
-
Can you explain where the 7s latency comes from? Regular update calls are advertised to complete in <2s, so I’m curious what the source of the remaining 5s is.
-
Is the prioritization of inter-canister messages over external ingress messages a universal problem across all subnets, or does it only affect the ICP canister and/or the NNS subnet?
-
You mentioned a couple other bugs, including a “second highest priority”, a “third highest priority”, etc. Can you briefly describe what they are?
-
This would be a perfect use case for a fully open sourced IC codebase, as suggested by this post. If this proposal is indeed high risk, I think it would probably be best if we all could take a look at the PR before voting.
Thanks!
4 Likes
The bug, in question, specifically refers to “a potential sandbox escape in a Wasm program”. Not sure about how sand-boxing,as a mitigation strategy, would solve that.
Yes, I should clarify there are two layers of sandboxing we are talking about. WebAssembly provides sandboxing to ensure the code cannot be modified and that data is contained within allocated memory. The risk we want to mitigate is jailbreak from this Wasm sandbox.
Sandboxing of canisters is another layer. The idea is to use process isolation. Each canister will run in its own process. So even if malicious canister code manages to break out of the Wasm sandbox, it is still only affecting its own isolated process, i.e., captured within the canister sandbox.
We have a thread for discussing canister sandboxing here.
5 Likes
- Can you explain where the 7s latency comes from? Regular update calls are advertised to complete in <2s, so I’m curious what the source of the remaining 5s is.
There are at least two rounds of consensus involved. A canister sends a transfer request to the ledger, which need to go through consensus on the NNS subnet to reach the ledger (and the NNS subnet runs at a slower speed than other subnets to prioritize stability). Then the response from the ledger needs to go through consensus on the canister’s subnet before the canister knows whether the transfer succeeded.
- Is the prioritization of inter-canister messages over external ingress messages a universal problem across all subnets, or does it only affect the ICP canister and/or the NNS subnet?
It affects all subnets.
- You mentioned a couple other bugs, including a “second highest priority”, a “third highest priority”, etc. Can you briefly describe what they are?
I’d prefer not to not discuss them until resolved. But generally: affecting availability, related to deterministic replicated message routing and execution, and not specific to ICP on canister.
- This would be a perfect use case for a fully open sourced IC codebase, as suggested by this post. If this proposal is indeed high risk, I think it would probably be best if we all could take a look at the PR before voting.
Totally agree. For ICP on canisters, note that the proposal only affects the ledger canister (open source) slightly; it upgrades the ledger canister by removing a condition that canisters transferring ICP must be whitelisted. We do not see much risk in the ledger upgrade by itself. The risk is whether there could be problems in the interaction between the ledger (open source) and the replica. But the replica (the platform itself on which the canisters run) does not change during the upgrade of the ledger canister.
3 Likes
Not sure that I completely follow your reply below.
However as i understand it, this high risk is perceived is from layer 1 risk, but the canister sandboxing is addressing a layer 2 risk. So currently there is no current mitigation for the layer 1 risk. Is this correct understanding?
1 Like
Perfect! So when should we expect to see the fully open source IC codebase?
1 Like
They’re addressing the same risk. The 2nd layer, process isolation, is a backup in case the 1st layer of defense, Wasm sandbox, fails to contain the attacker.
2 Likes
Is there a way to vote from inside NNS Dapp?
1 Like
When can we hold ICP in ordinary canister
2 Likes
Voting on enabling canisters to transfer ICP is now open
9 Likes
Is there a way to vote from inside NNS Dapp?
Yes, if you have staked ICP in a neuron with long enough dissolve delay in the NNS Dapp, or you have staked outside the NNS Dapp but assigned a hotkey in the Dapp
7 Likes
I’m voting no again, basically the same reasoning as before: Enable Canisters to Hold ICP - #77 by lastmjs
3 Likes
Earlier discussion previously here leads me to the decision of voting yes!
TL;DR: Due to an ongoing incident, the foundation will vote “reject” on the currently open proposal for “ICP on canisters” (proposal 30496). Once the incident is resolved, a new proposal will be submitted.
Hello, there is an ongoing incident with one of the application subnets (subnet “pjljw”). The subnet hosts a canister for which execution of messages results in different states on different replicas. Multiple teams are investigating the cause for this non-determinism right now.
The Internet Computer has a fail-safe built in for such a case causing the subnet to suspend sending messages to other subnets so that the problem can be isolated and fixed. However, this may prevent canisters on other subnets from being upgraded. The reason for this is that, to upgrade a canister, the canister needs to have processed all outstanding messages - otherwise replies might not be processable anymore by the upgraded canister due to change of code.
Unfortunately, there is thus a possibility that the Ledger canister may not be able to upgrade and stall: if there are outstanding messages from the Legder canister to the Cycles Minting Canister to top-up or create canisters on the “pjljw” subnetwork, these messages cannot obtain a reply and the Ledger canister will not be able to fully stop, which is a prerequisite for upgrading.
We view this as a high risk. Therefore, to avoid another potential incident, the DFINITY Foundation has chosen to vote “reject” the ledger upgrade proposal 30496 enabling ICP for canisters.
Please note that the incident with subnet “pjljw” is NOT related to the “Canisters can hold ICP” feature. Once the incident with subnet “pjljw” is resolved, the Foundation will resubmit the proposal upgrade the ledger canister and to enable ICP on canisters.
16 Likes
sad news, “pjljw” subnetwork What’s happening?
Outstanding responses preventing a ledger upgrade - it needn’t be that way.
7 Likes
Was the break in determinism just discovered? I know pjljw has been a hotspot for issues, but I wasn’t aware of something this big.
4 Likes