Proposal: Create a 3-node Subnet for Real-Time Multiplayer Game Canisters

Objective

This is a governance motion proposal describing a solution for efficiently running a real-time game server using canisters. The team of PLETHORA proposes that the NNS should create a subnet specifically designated for real-time multiplayer game canisters. This subnet would consist of 3 nodes rather than the usual 13 node application subnets.

This is a motion proposal and does not directly result in changes to the IC. Rather, voting to support the proposal is an indication that work should begin on creating this gaming subnet. If it passes, then we will work with DFINITY and other parties to create a 3-node subnet.

Background

In order to have a multiplayer game with real-time player movement, it is necessary to have a server that records the positioning of each player. This data needs to be updated frequently in order to prevent lagging of players. In most online games, this requires a ping of <100ms.

PLETHORA is focused on building 100% on-chain games, meaning we’ve built a real-time multiplayer server in a canister. We leverage the Unity game engine and the ICP.NET agent (thanks @Gekctek !) to communicate directly with canisters.

And we made a video to show it to you!

In this video:

  • Every 300ms each player updates their position in the world to the canister
  • Every 25ms each player queries the canister for positions of other players in the world
  • We then estimate the animation states and use smoothing techniques to render the characters running/jumping around the 3D world

Here’s the link to the video demo:

However, we’ve ran into two main issues:

  1. Canisters have >2000ms ping because canister state changes must go through consensus. The more nodes in a subnet, the longer it takes to reach consensus. This is awful lag for real-time multiplayer.
  2. Three canister update calls are sent per second per player to record player positions. This huge volume of calls cost an exorbitant amount of cycles. The more nodes in a subnet, the more expensive the update calls become.

By lowering the amount of nodes in a subnet to three nodes, we can reduce both latency and cost for running a real-time multiplayer game canister.

Another consideration we want to propose is creating 2 subnets of 3 nodes each. The first subnet would be populated with 3 nodes based in North America. The second subnet would be populated with 3 nodes based in Asia. Therefore, we could route our Western players to the North America subnet and our Eastern players to the Asia subnet. This will result in much lower latency from the game client to subnet, but also lower latency between the nodes communicating for consensus. We would love to hear your feedback on this.


But what about the security of these subnets?

In its first iteration, the movement in PLETHORA’s games will be client-authoritative, meaning it is purely for visual purposes. For this reason, it’s not a major concern that a subnet could be subverted by two colluding nodes. There’s no incentive to do so because the multiplayer movement does not have important game logic associated with it.

In the future, we can take small steps to enforce server-authoritative movement in a canister to prevent clients from cheating their movement. At that point in time, the security of the subnet may become a bigger concern. But we firmly believe the tradeoff with latency and cost is worth the reduction of nodes for client-authoritative movement.

Why This Is Important

Gaming will be the next major use case of crypto, and gaming economies will eventually overtake economies of entire countries. We have conviction that in order to build games that will rise to that scale, everything needs to be fully on-chain. When a game is fully enforced by smart contracts, it presents the greatest opportunity for humanity to improve how we interact and create value for each other.

And when we say 100% on-chain, we mean it. This includes real-time multiplayer game servers. Adhering closely to this strict standard will enable composability like never before, and will empower games to be DAO-governed through the SNS.

No other blockchain can run a complex game with tremendous amounts of logic and data 100% executed and stored on-chain. We need to show the world what the Internet Computer is capable of accomplishing.

What We Are Asking The Community:

Provide feedback on this proposal within this thread.

Vote accept or reject on NNS Motion

Participate in technical discussions as the motion moves forward

Additional Cycle Costs Calculations:

Let’s calculate the cost savings of switching to a 3-node subnet. These are just estimates, the actual amounts will vary depending on how many bytes are sent in the ingress messages.

If we send 2 update calls per second per player, that’s roughly 7.2K update calls per hour.

On a 13-node subnet, 7.2K update calls with position/rotation/scale/animation data uses roughly 14B cycles.

Let’s assume we have 10,000 concurrent players for 24 hours on a 13-node subnet.

On a 13-node subnet, it would cost around 3400T cycles.
That’s roughly $4,530 per day or $136,000 per month

If we do the same math for a 3-node subnet, it would cost around 785T cycles.
That’s roughly $1,070 per day or $32,000 per month

As you can see, the cost of cycles is significantly reduced with less nodes. This will become increasingly important as games scale to millions of users.

It’s harder to calculate latency when reducing the node count of a subnet, but we expect it will significantly decrease as well.

23 Likes

Congrats on the project!

It’s admirable that you’re not just juicing up some KPIs and doing an SNS sale for 1.5M ICP like a few others, but a stable ping of around 30ms is required to play most of the dynamic online multiplayer games. Will this be possible on the IC at its current state?

3 Likes

This seems like a very well researched proposal with a lot of benefit to all game apps. It makes a lot of sense to take this approach. I’m looking forward to following the deliberation and seeing the proposal hit the NNS. Thanks for outlining the problem and offering a solution.

4 Likes

If I’m understanding correctly all you are asking for is a subnet with low replication. I was under the impression that DF was already working on something like this for their storage subnets.

Would you ask the foundation to work on this request as a focused effort or do you think you could use a storage subnet to achieve the same thing?

2 Likes

How did you decide on the number 3? The IC’s consensus can handle at most f < n/3 faulty nodes (source here). Note the < instead of <=. This would mean that even a single malicious node could break the whole subnet. In this case one could go even down to 2 or just 1 node. If you want to tolerate one faulty/malicious node then you’d have to have at least 4 nodes in the subnet

11 Likes

30ms ping would not be possible considering that blockchain updates must go through consensus of nodes around the world.

We’re limited by the speed of light, because fiber optic cables take roughly 500-800ms to transport data around the world.

Video game companies solve this by placing game servers in every geographic region of the world. In contrast, blockchain nodes are distributed around the world and require consensus, meaning extra latency.

We’ve proposed a solution for 2 geo-located subnets. 1 subnet in North America and 1 subnet in Asia. This would reduce latency considerably and we view it as the perfect compromise of decentralization and speed.

We believe that fully on-chain games will invent new genres. Not all game genres or game mechanics need e-sports latency.

Unfortunately, games that require competitive gameplay based on server-authoritative movement are not possible in their current form. But we believe these games can be redesigned to be about strategy and consistency instead of lighting-fast reflexes.

Some examples of games that would benefit from on-chain multiplayer even if it was slow ping:
RPG: Pokemon, Breath of the Wild
TCG: Hearthstone, Marvel Snap
Puzzle: Candy Crush, Gardenscapes
Word: Wordscapes, Scrabble GO
Strategy: 4X games, Clash of Clans, Clash Royale
Sandbox: Minecraft, Roblox
Idle Games: Raid Shadow Legends, Coin Master

The list goes on and on. And these are just the things I can think of off the top of my head. But what I’m excited about are the game genres that don’t exist today that will be invented by fully on-chain games!

7 Likes

I’m not sure the status of DFINITY’s storage subnets implementation, thanks for bringing this up. I assume there might be additional logic layered onto those subnets catered towards storage. They also may require more security than ours.

Also, we would like to have geo-located subnets (eg. North America subnet and Asia subnet) to further reduce latency. Node collusion is a very low concern for us because there’s no benefit to subverting a real-time game server subnet.

4 Likes

We were under the assumption that consensus could handle f <= n/3 faulty nodes. But looks like we were mistaken.

In that case, we will revise our proposal to be about creating either a 4-node subnet or a 1-node subnet.

I could see pros and cons in both.

Since this subnet is purely for visual rendering and does not store any persistent game state, I’m leaning towards a 1-node subnet. And if the subnet becomes malicious, we can just move our canister to another subnet or remove that malicious node via NNS.

I’m curious to know what you think would be better considering cycle costs/latency/decentralization?

Also if a 1-node subnet becomes malicious, can it affect other subnets? Can it attack the IC network?

5 Likes

I see you know what you’re talking about!

Will keep a close eye on your future projects!

I’ll also try to vote in favor of any governance proposals you make if they help materialize your vision for gaming on the IC!

Great job writing such a high quality proposal @atomikm!

I think running something on a single node is a bit questionable wrt it being “on chain”. Of course it technically is, but having no fault tolerance at all makes me wonder what the advantage is of running it on chain. So I think I would prefer 4-node subnets as minimum.

Yes, there are some challenges wrt subnets that become overall malicious. One example is wrt cycles that can be sent across subnets, and another is that a malicious subnet may simply not reply to calls from canisters on other subnets, which in turn could prevent them from upgrading. There is some work ongoing / planned in DFINITY for both of those, but I think that would be engineering work that has to be done before such subnets can be created.

Some last point of feedback: have you thought about how you would actually want to use this? How would a user indicate in dfx which subnet to install on? Currently, you can specify a subnet-type when creating a canister (docs), that allow you to install on the “fiduciary” subnet (pzp6e). How would a user in dfx express that they want to install on eg a small US-based subnet? We need to find a way that also works if in the future there are eg also small but not geographically colocated subnets, standard size subnets that are geographically colocated, etc.

8 Likes

I support the overall proposal. A 4 node subnet is preferable since you get some fault tolerance and still be on chain. A single node is asking for trouble as if that single stop working then the game will be down for a while.

4 Likes

Is there a way to have the bulk of the game on the 4 node subnet but to the extent there’s anything or significant value in the game - in app wallets, etc - those can be held on a different subnet with more nodes?

3 Likes

Great proposal! The additional benefit of such a low-replication subnet is that it would lay the groundwork for GPU subnets running AI models in the future.

4 Likes

Great proposal. We (Eimolad) have already raised this issue on the forum earlier. We believe that 1 or 2 nodes are enough. Also, it is necessary to remove the artificial limit on the number of calls, if it exists.

1 Like

Interesting Porposal. Will offer precende to further projects like on-chain AI.

1 Like

Can you clarify which limits you mean? AFAIK all limits are negotiable, but we don’t want to open any DoS attack vectors

1 Like

Hello Manu! Thanks for your valuable feedback.

Yes that makes sense, having at least 4 nodes would qualify it as on-chain with fault tolerance. And it would help prevent a malicious subnet wreaking havoc.

It would also probably be enough security that we could enforce movement constraints in our canister. Thus, smart contract logic could be tied to a player’s position in a world. A primitive version of smart-contract-authoritative movement could be possible in the future!

I will need to redo my cycle cost calculations to account for a 4 node subnet. Are there any cycle cost savings if the nodes are geo-located close to each other?

Regarding DFX commands, I think makes sense to add another subnet-type called small or game (or whatever you deem fitting) that has 4 nodes. Then we would also need another argument like subnet-region where you could specify north-america or asia. This way you could combine the different arguments to achieve the subnet of your desired size and geography.

2 Likes

Hey Davie!

Yes, this is exactly the structure we are planning. Essentially, this real-time multiplayer subnet is only responsible for recording player position data and storing it temporarily. In the future, it may enforce smart contract logic to constrain movement to ensure clients don’t cheat their movement.

All canisters that hold valuable game/player data and NFT/ICP/ICRC-1 assets are located in other higher security 13-node subnets.

In the future, we could have canisters in the 13-node subnets query the canisters in the 4-node subnet to validate player positions. Therefore, enabling important smart contract logic to be executed based on a player’s position in a world.

5 Likes

As it was said at the heart of the topic, the multiplayer canister should be able to answer hundreds of calls per second. Otherwise, it is impossible to achieve a comfortable gameplay. It is probably necessary to find other ways to avoid DDOS attacks. Maybe there is a way to grant access to call initiation only to certain users. We are now discussing the subnet, not the main network.

We don’t need a highly reliable subnet for multiplayer, because we don’t store critical data there. The whole game is built in the main chain. The multiplayer canister provides interaction only with streaming data. I will try to give an example: User 1 changes coordinates and User 2 changes coordinates. Both users should only exchange coordinates. For example, User 1 left the game. The current coordinate of User 1 is written to the main IC network. Thus, the data about its coordinates are stored securely.

1 Like

If a 4-node subnet is the practical minimum cluster size and lowest possible latency and cost is the goal for an on-chain multi-user real-time gaming server, then we are already talking about a low-Nakamoto co-efficient cluster of nodes. So let us consider another part of the equation.

I am assuming the other main factor relevant here is to have very low-latency network links between the 4 nodes. So if the 4 nodes are all located in the same data centre or in difference data-centres which are geographically nearby (maybe with dedicated inter-data-centre network links) then I expect the IC update query consensus between the localised cluster of nodes would be optimised. It should be easy to test with a 4-node local testnet hooked up to the same switch I would think.

Obviously this kind of IC subnet is special purpose because of its low level of effective decentralisation (# of nodes, # of DCs, # of NPs) but could it still be much more resistant to attacks and outages than a single node. And if we have one city or data centre (DC) with more than one Node Provider (NP) in the one place that would be obviously better. In the near future as more IC node providers on-board you may find one location in EU or USA where you can connect 4-6 replica node machines in a low-latency network configuration that includes 3 DCs and 3 NPs, thereby maintaining a low but still effective Nakamoto co-efficient for this specialised subnet.

You could label these special subnets as an L3NC subnet-type , where:

  • L3NC == LLLNC => Low-Latency, Low Nakamoto Co-efficient (subnet)
    or equivalently
  • L3NC == LLLNC => Low-Lag, Less Nodes Cluster (subnet)

OR just use some real word for the subnet-type instead, like a normal person would :wink:

And this L3NC subnet-type would support applications other than gaming that have similar low-latency optimisation requirements. Then (as @atomikm said above) the standard application pattern would be to make cross-chain calls to pull “permanent” data back into standard 13-node application subnets for security.

Also, a L3NC subnet would need 1 or 2 colocated boundary nodes that clients connect to directly to guarantee the minimum latency.

PS: there is a highly applicable discussion thread here What is the theroretical number for txns per second on Internet Computer right now about transaction throughout related to geo localised and reduced node count subnets.

1 Like