Thanks for this thoughtful reply @levi!
Before going into the two scenarios, I would like to make it clear that I wouldn’t say everything is fully done when we have canister migration. My point is that I believe canister migration is the most valuable thing we can do now to make scaling ICP easier. You can still imagine many things that would be helpful after that, eg
- give canisters a better way to deal with temporary surges of activity on a subnet. Right now, the only option is allocations, but this does not fit all use cases. Maybe there can be some option for canisters to express they’re willing to pay more whenever things are busy to get more priority.
- Some form of automatic load balancing. So developers can choose to migrate themselves, but there is also some automatic thing that coordinates load balancing on ICP (perhaps opt-in?).
These ideas are clearly not fully baked yet, but are just examples of things that could still be improved.
So coming back to your scenarios: i agree that scenario 1 could happen, although i also think it wouldn’t be so bad. Different canisters will likely have different latency requirements, so some would migrate earlier than others. Also more subnets would likely help: if we have eg 50 subnets, and some are suddenly busy, while all others are not heavily loaded, then it seems likely to me that the load would spread. So in other words, the bigger ICP is, the less impactful small bursts of usage are, and the easier it is to spread the load. In the long term, perhaps this automatic load balancing idea that I mentioned above could help here. However, also for that the first step we’d need is the ability to move canisters from one subnet to another.
Wrt Scenario 2: i also agree that this could happen. As mentioned above I think some better mechanism that allows a canister to make sure it remains highly responsive even when the subnet is busy could be helpful. And the proposed cycles cost changes would at least make it expensive to DoS.
So in summary, I’m not claiming everything is perfect once we have canister migration, but I do think it would be a huge step in the right direction and that it’s the first thing we should focus on. Does that make sense, or do you see it differently?