Welcome to the forums and IC development 
Here’s a doc with all of the previous meeting recordings.
If you want to recommend a discussion topic for the next month’s meeting, feel free to do so in this topic → Technical Working Group: Scalability & Performance
I’ll push back a bit here. The problem isn’t the number of canisters, it is the number of canisters that are competing for scheduling slots. If you use an architecture that only gets pushed to and your canister is only doing work when it is necessary then things should work rather well. Right now the scheduling algo only has one kind of ham-fisted mechanism(compute allocation) right now(not a criticism…that’s all we’ve needed until now and I’m all for gsd and not over-engineering before its time…but its time now 
).
The scheduler can get a lot smarter! And we can get better handles on it. Here are some things I’d like to be able to say with my code:
If you are using a timer right now you don’t have a ton of options. I mentioned in a thread last week that the @PanIndustrial Timer module helps with some of this because it can batch up a bunch of timers into only one scheduling request. It would be awesome if I could also spend a bit more to get that prioritized at that time…and if I don’t/can’t at least it will process missed timer hits.
The event utility WG’s work on ICRC-72 has been focused on this fan-out distribution of data across the IC. If I write my dapp with this event-based architecture then I really want to know when I’ve gotten a message and react to it quickly. The rest of the time I can just kind of chill in the round-robin pool.
I’d be interested in building some of this stuff at the application level, but if it is going to be moved to the replica layer there is less motivation to do so. And sliding a lever up and down on thousands of canisters a second likely isn’t the best engineering solution either.
At a high level, folks should be backing off of timers. Especially heartbeats. That wasn’t great architecture before and it is even worse now. I’ve been saying for a while that the IC is a new kind of computing platform and it requires going back to the base metal and building up new kinds of software that take its unique properties into account. Can you imagine trying to run a dynamic web server like Express under these conditions? Or trying to get timely HTTP outcall responses? It just wasn’t designed to support that kind of software at scale with serious reconfigurations of our architectural assumptions.
Another concern is that big chunky canisters have the same problem but will just experience it a bit later than small ones because of the artifact of the current scheduler algo. The resources are restricted on a subnet for either no matter what algo you choose. Once you get above 3 big chunkers that want 100 percent compute you’re right back here.
The Yral canisters are all running heartbeat?
If that’s the case then no wonder there’s an issue because they’ll all be trying to run on every round.
I understand what I’m about to say is very idealistic and wishful thinking, but I would like to urge those with interest in the protocol layer to attempt to make this vision a reality:
The fact that we’re even talking about subnets (and maybe even canisters) feels not ideal and unfortunate. The concept of a subnet should be entirely abstracted away from the developer. We shouldn’t have to think about these kinds of things.
I hope someday that the protocol can be significantly rearchitected with the absolute best developer experience in mind.
Developers should write a program and deploy it. It should be able to seamlessly communicate with any other program. It should be able to utilize any networking protocol to communicate within and without the network. It should choose its level of security (replication). Load balancing should be automatic. Programs should simply scale in heap and storage and throughput as long as they can pay for it. Heap and storage should be virtualized and infinite.
It’s astounding the level of complexity that exposing subnets and other concepts has caused.
Due to the enormity of the issues with building scalable applications on ICP, I wonder if some fundamental redesigns are necessary.
Hi and thanks for the comment, I believe that this would be the ultimate goal, but to get there it might take very/too long (considering mainly existing projects which want to scale and run), to improve current projects/developers experience regarding scaling issues I’d suggest to consider (also) following as minimum:
It went to profile dropdown. Also, it’s currently broken and dfinity team says that they will push a fix only on Monday if NNS proposal goes through.
The reason for choosing MMC is probably - planning to go for wapps in the future. Where users install modules in their personal canisters. MMC, Hearbeats and Timers imo are some of the best features of the IC.
However, when you use them all together combined with a scheduling system that seems to be prioritizing MMC, because of the sheer number of canisters, other architectures around perform poorly.
Additionally, there are xnet messaging limits per canister and the MMC architecture can also take over that, not only the computation.
In a way massive multi-canister architectures now get reserved-like compute allocation without paying to reserve it.
How about a simpler scheduler change: Something like - the top 10 canisters by cycle balance get 20% of the non-reserved computation power. Next 50 - get 10%. The rest get 70%.
Priority by Stake (PoS)?
Actually it reminds me of how EOS auctions its ram space. We probably need another fundamental token that is not cycles, and its value should be decided by market. Canisters that hold such tokens will get compute priority, calls attached with such tokens will get message priority, and so on.
So the DAOs with the bigger treasuries get to buy more cycles and get priority?
There is no cost barrier entry for a developer on web2 to get a reasonable performance. I was sold infinitely scalable at near web speed for $5 per GB per month.
I kinda like the balance thing. You still get your promised prices, but you get priority by showing commitment. Those cycles don’t get burned in @infu 's proposal and you still get use them on compute/storage, you just have to pay earlier if you want priority.
It is a decent short-term solution if the scheduled has easy access to cycles balances.
Why not using $ICP? Why we need separate token? it already becomes too complex
If everyone dumps a lot of cycles into their canister to get priority such that all canisters maintain a 1:1 ratio of cycles with each other wouldn’t we be back in the same situation? Of course this perfect equilibrium doesn’t exist and perhaps more naturally it will create a bidding war where devs are constantly dumping cycles into their canisters to get top priority
In a way it’s good that BOB and Yral are bringing these issues to light sooner rather than later.
If the priority gets figured and doesn’t depend on how many canisters one has spawned. Yral probably won’t feel a thing and it will still utilize the same amount of subnet resources, while other canisters are responsive like before, executing ahead of the swarm. Easier said than done.
Yes in the grand scheme of things it’s better we see this issue now, it’s just a shame its going to last for a couple more weeks maybe 
It is almost as if the two biggest decentralized chains have a user pays gas/fee model for a reason. 
Reverse gas is great to the extent that it works, but service providers notoriously search for the cheapest path to deliver their service as soon as margins get thin. Where we are now is that we have service providers up against paying a base carrier rate to continue to provide base level functionality without the expectation, funding, or desire to do so.
It is likely imperative to have a way for users to pay for priority in the future as they may want to consume a service where the dev has become disinterested in the product or it is under a blackhole/sns that doesn’t have the capacity or timely ability to deal with usage spikes. Is there even a way for SNS controlled app canisters to up their compute allocation right now?
So if a canister doesn’t have enough cycles to be high priority enough does that mean they’ll likely see timeouts? If so, i wouldn’t support that. Sure, service providers should be compensated accordingly but not at the expense of degrading the run time of mom and pop’s canister. I think I would be happier with a distribution of different types of subnets that vary in terms of cycle cost and the max number of canisters that can live on them but this relies on being able to migrate a canister easily and so maybe it’s not feasible. With the different types of subnets you’ll have some that have a higher cycle cost but a guarantee that there won’t be as many canisters on it. It’s not too dissimilar from choosing a VPS with fewer people consuming resources in a loose sense.
I agree this would be ideal, but I think that is a longer-term solution. Right now the choices are to let people pay for priority or let DFINITY pic favorites or maintain the status quo and I think only one of those is acceptable in the immediate term.
But keep in mind that you can’t keep this from happening for canisters that aren’t willing to pay for computing. A subnet costs about $15k / month. If you want to be guaranteed to not have time out you either need to pay that or a third of it at least or cross your fingers and hope everyone shares nicely(they won’t).
If cycle balances get used to boost the priority and a canister doesn’t have a lot of cycles, then for them it will work the same way like now.
One of my canisters was just delayed scheduling by 12 hours before it finally processed the work it was meant to do. I know we want to prioritise canisters in some manner but giving a 12 hour delay to those on lower priority seems… not optimal. people just won’t bother to develop on ICP