Hi everybody! In the recent weeks, we’ve seen the demand for computation on ICP significantly increase. This is of course great news, but also leads to some challenges. We now see that there are quite some subnets that are running at or almost at maximum compute capacity. This means that canisters have to wait for their turn to compute and therefore messages are processed with much higher latency. You may notice ingress messages get rejected with an error saying “timed out waiting to start executing”. The main affected subnets are lspz2 fuqsr 6pbhf e66qm bkfrj 3hhby nl6hn opn46 lhg73 k44fs You can see which subnet a canister is on by looking at the canister’s page on the dashboard, eg https://dashboard.internetcomputer.org/canister/ryjl3-tyaaa-aaaaa-aaaba-cai. What can you do now If you are a canister developer experiencing issues from being on such a busy subnet, there are two main things you can do. migrate your canister to another subnet, by creating a new canister on a more quiet subnet (you can see the load of subnets on the dashboard, key things to look at are canister count, million instructions executed per second, and state), and migrating the state of your canister over to the new subnet (which requires support on your canister, there currently is no built in support for this). Set a compute allocation. A compute allocation is a way for a canister to reserve compute capacity. They are expressed in percentage point, so setting a compute allocation of 5 means your canister will be guaranteed run 5% of the rounds, so at least once every 20 blocks, even if there are many other canisters that want to compute. The cycles cost is roughly 35$ per month per percent point for a compute allocation on a 13-node subnet. Are there improvements to the protocol coming? Yes, DFINITY is working on two short term improvements. the scheduler, which decides which canisters get to run when, can be improved. The scheduler currently only guarantees fairness in the sense of being the first to get scheduled in a round. However, the canister that gets the first chance to run may not need the full round duration, and in that case other canisters can do more work, but this is not factored into future scheduling. This means that under certain workloads, some canisters get significantly more compute time than others. We plan to upgrade the scheduler to slightly change the definition of fairness, which should reduce latency for canisters on busy subnets. This change may already be included in the replica version that dfinity proposes later this week. The canister execution layer limits the amount of work it does every round such that it ends in a timely fashion (it aims for at most 1 second). However, it looks like for certain types of workloads, in particular with many distinct canisters processing small messages, it is too conservative in the work it schedules, meaning it could actually do a lot more work without exceeding the desired time limit. That means that with some adjustments to the bookkeeping, the compute throughput of a subnet could significantly increase, alleviating the congestion. We hope such an improvement can come in the coming weeks. Are there other ideas for improvements? Yes, there are many ideas. Subnets can be split, we can imagine further improvements to the scheduler, we could add other ways for canisters to request priority on busy subnets, we could further increase the compute capacity of subnets by using more cores, make migrating canisters to a different subnet easier, etc. However, for now I believe that it’s best to focus on the two short term improvements mentioned above, and then see where we stand. Please feel free to ask questions here, and we’ll keep this thread up to date on developments wrt these busy subnets.

Is it planned to use more than 4 cores per subnet? That’s the cheapest way to scale if execution rounds being full (by other canisters) is the bottleneck.

It is definitely an option. We didn’t do it because (i) there’s already high contention due to our setup (e.g. all mmap() calls following a page fault go through a single kernel thread; disk encryption is slow and IIRC it also goes through a single thread); and (ii) more concurrent executions means higher peak memory usage (and, believe it or not, between replicated executions, queries, multiple versions of the replicated state, P2P artifact pools and the various overhead, we could easily exceed 512 GB of RAM and if the kernel kills any of our sandboxes, it’s good bye deterministic execution). There are people looking into all of this, it’s just that it’s not as simple as bumping a constant.

The idea was essentially to make compute allocation more flexible. If I understand correctly, we are currently paying a fixed price regardless of the subnet’s load. I believe that sometimes you can’t get the allocation if the resources are oversubscribed. Ideally, that shouldn’t be the case.

A compute allocation is a hard guarantee. It’s quite expensive, but you will get that percentage of a CPU core regardless of subnet load (e.g. if you pay for 50% compute allocation, you get scheduled at least 1 round out of every 2).

@free @Manu thank you for being frank here :pray: I know that the (DFINITY) system dev/integration team is very experienced and knowledgeable but still (because IMO this is the most important topic for cloud services provider - the abstracted functionality (most likely) won’t be more effective and reliable than the underlying layer/system) - from my experience more heads (with different background) know more - mainly diversity in experience/approach can be very convenient for solving such complex problems - I want to ask if is there any way how community members can get involved in order to (try to) help in this matter. E.g., there is a Technical Working Group DeAI (and other groups for specific topics), is there any Technical Working Group Base System?

[image] rbole: how can I find a subnet with low loads? I think the best metric to see if a subnet has a lot of computation load or not is to look at the “Million Instructions Executed Per Second” metric on the subnet page of the dashboard. Many of the busy subnets are > 5 MIEPs, while some others are < 1MIEPs.

[image] free: I apologize for getting so excited and combative but, as said, either you pay for guaranteed performance (which is very expensive, because you are basically blocking off a large proportion of a subnet for yourself); or you don’t get guarantees. It’s as simple as that. And this is not a shortcoming of the IC (although there are plenty of those). It’s simply impossible to provide “predictable” performance to arbitrarily many canisters with what is necessarily limited resources. The current high loads have really brought to the surface lots of interesting facts on how IC operates under the hood. Going forward, we must now internalise this information and let it affect what types of applications we try to build. I believe the general perception is that you can build and host “anything” on IC, that it is a platform to replace any AWS serverless function or Cloudflare worker. This, clearly, is not the case. Which is ok. It is just different. In some cases this difference is a deal breaker, in many cases you can work around it. Two examples: Login using SIWE. Signing in with Ethereum requires using threshold ECDSA and thus makes update calls during the login process. If it takes 20 seconds to login that is a 100% dealbreaker. Conclusion: Make sure you app runs on a subnet with low load or make a compute allocation or both. The smallest compute allocation, 1%, costs ca $35 per month, right? Which unfortunately also is a 100% dealbreaker for small open source projects. So, either make sure you have a revenue model from day 1 or try to find a subnet with low load. A visual note taking application. While the user edits, the changes to the document are saved to the backend canister. In this case you can work with optimistic updates in the UI and do the saves in the background. It is not the end of the world if a save takes 20 seconds as long as you have planned for this wait. To avoid making app developers really sad, I think the unpredictability needs to be more clearly communicated. There was some change made to the main website just recently, but it would be wise to often (and in many channels) remind app devs that they cannot expect update calls to finish in 2 seconds.

We badly need more transparency on subnet stats. The current stats MEIPs and TX/s aren’t enough. Usually TX/s is a rather flat line of updates per second sometimes in the hundreds. The fact that it is a flat line shows that it must be a high number of heartbeats drowning out the number of ingress messages. Ingress message rate would be expected to follow some recognizable fluctuation pattern over the 24 hour and 7 day periods. We need to be able to see that. Any additional information would be helpful such as ingress message rate, size of induction pool, percentage to which the execution rounds are filled, etc. That will help developers understand when, for how long and why their canister’s user experience might have been impacted. The question is also why we need a dashboard for this. Why can we not query nodes directly and have them export these stats?

[image] kristofer: The smallest compute allocation, 1%, costs ca $35 per month, right? Which unfortunately also is a 100% dealbreaker for small open source projects. It should be possible to get that funded from the community by donations. Doing that would be easier if there was a way to pay directly for compute allocation of another canister instead of sending cycles to the other canister first and then have that canister buy the compute allocation with it. Because it involves two steps and the other canister can potentially withdraw the cycles or do something else with it. I am mostly interested in the problems that were described in this thread that can not be solved by compute allocation. There seem to be two kinds: I want a rather small (affordable) compute allocation, say for example 10%. I am worried about latency for my user’s ingress messages, but my own executions are rather short. That is a common scenario for most user facing dapps. Say I have multiple noisy neighbours on the subnet which fill up ingress messages, heartbeats, timers and run long executions. Now let’s look at the ingress latency for my users. Over three consecutive execution rounds the scheduler prioritizes ingress, heartbeats, timers and then starts with ingress again. So ingress is prioritized every 3rd round. If I have 10% allocation my ingress messages will get prioritized every 30 rounds, or around 15 seconds. That is waiting time. The execution time and reading the response is still added to it. That is clearly too long. Even 100% compute allocation will prioritize my ingress messages only every 3rd round which can still be too slow. I think to support low latency user facing dapps with short executions better we need specialized subnets. We can optimize for the absence of heartbeats/timers and for short executions. For example we can prioritize ingress 9 rounds out of 10 and we can limit the execution time to a lower bound than 2 billion instructions. So that multiple canisters can run per core per round. My own canister uses long executions. That is fundamentally contradicting low latency responses from my canister. Because a canister is an actor and cannot process two message in parallel. If that is the scenario then I have to move to a multi canister architecture and possibly also multi-subnet architecture. If I have such high execution load I might also have to give up on low latency. And I should be prepared to pay for the required compute allocation or even rent an entire subnet. The complaints in this thread seem to come from apps in scenario 1. And the root cause of the problem seems to be that we have one-fit-all subnet parameters that simply cannot deliver what these dapps need. Trying to deliver low latency and long executions spanning full rounds in the same subnet is not going to work.

Subnets with heavy compute load: what can you do now & next steps

Developers

free October 28, 2024, 7:24am 152

There’s a Scalability & Performance Technical Working Group. More community involvement is always appreciated, the attendance isn’t all that high usually.

Topic		Replies	Views
LAMENT: A tale of constant struggle of what it's like trying to scale on ICP Developers	73	4913	November 3, 2024
Suggested measures to reduce latency and improve ICP scalability Developers	47	1802	November 4, 2024
Path forward for subnet splitting and protocol scaling Developers	19	606	October 17, 2024
Evaluating Compute Pricing in Response to Increased Demand on the Internet Computer Protocol Developers	51	2611	April 7, 2025
High User Traffic Incident Retrospective - Thursday September 2, 2021 Developers	50	9796	October 30, 2021

Subnets with heavy compute load: what can you do now & next steps

Related topics