Rollback Netcode, Explained
The speed of light is not as fast as we would like.
When players are far away from each other, there is a delay between when the player sends an action and when another receives it. This delay is called latency, and it can make games feel unresponsive. Imagine pressing the jump button and your character not jumping immediately. Rollback netcode is a technique that makes a game feel responsive even when players are far apart.
What is rollback netcode?
Normally, a game would have to wait for all inputs to arrive before simulating a frame. The waiting time is what causes the latency. You press jump, but you can't see the result because your computer is still waiting for your opponent's inputs to arrive.
Rollback netcode allows your game to move forward without waiting. The game predicts what the missing inputs are likely to be, then simulates the next frame using those predicted inputs. This lets it move forward. If later the actual inputs arrive and they differ from the predicted inputs, the game rolls back to the point in time where the prediction was made, and then re-simulates the game from that point forward using the actual inputs, correcting any misprediction errors.
There may be a little blip when the game corrects itself, but it can be smoothed over using rubberbanding and normally the correction is so small and quick that the player does not even notice it.
Rollback netcode lets the game appear to respond immediately to player inputs, even when there is significant latency between players. It is commonly used in fast-paced player-vs-player multiplayer games where the milliseconds matter.
Why does rollback netcode work?
Generally players only provide input a few times a second, while the game is simulated 60 ticks per second, and so most frames can be predicted accurately without needing to wait for all inputs to arrive. That is why rollback netcode is able to hide latency so well in practice.
Why is rollback netcode hard to implement?
Rollback netcode requires two things:
- The game must be able to snapshot and restore a previous game state, so that it can roll back to a previous point in time and re-simulate from there.
- The game must be deterministic. That is, given the same inputs, it must always produce the same outputs, on every computer, no matter what.
These requirements must be fulfilled 100% correctly across your entire game's codebase. If even one line of code is not deterministic/snapshottable, it will cause desynchronization bugs where different players see different game states, ruining the multiplayer experience.
Most game engines cannot support rollback netcode
When you are coding everything yourself, you can ensure that everything you write is deterministic and snapshottable. But the problem is that most games are built on top of other people's code - normally a game engine, a game framework, or a physics engine. These will not be 100% deterministic and snapshotable unless they were specifically designed to be that way.
That is why most game engines cannot support rollback netcode. When people use rollback netcode in a game engine that was not designed for it, they do it by reimplementing certain parts of the engine themselves, which can be a lot of work.
A rollback programming language
There is an easier way!
Bake the rollback netcode into the programming language itself, and then anything you write in the language will automatically be rollback netcode-ready. This is the approach that Easel takes.
In Easel, multiplayer has been built into the fabric of the Easel programming language, which means that anything you write in Easel is automatically multiplayer-ready. All the hard work is done for you, so you can just code as if all your players are in one shared world, like a singleplayer game, and without any additional effort from you, Easel makes your game multiplayer, completely automatically. It's like magic!
Easel's state-of-the-art rollback netcode implementation
Easel has one of the most advanced rollback netcode implementations of any game engine. This section details the unique features of Easel's rollback netcode.
Incremental snapshotting and rollback
Many rollback netcode implementations must snapshot the entire game state at every tick. Sometimes this takes longer than executing the game simulation itself. In Easel, only the changes are snapshotted and rolled back, not the entire game state, which improves performance dramatically. We call this incremental rollback netcode.
Because multiplayer is built into the Easel programming language, the change detection necessary for incremental rollback netcode happens transparently and reliably without any special effort on your part.
True peer-to-peer
Easel uses peer-to-peer networking because this means every packet can take the shortest, most direct route between players, minimizing latency.
A common design of peer-to-peer games is for one of the player's computers to act as the host. This gives the host an unfair zero-latency advantage, while doubling the latency for all other players as they must wait for a full roundtrip to the host and back.
In Easel, there is no host player. Inputs are sent to all other players equally, as quickly as possible, and every client forms its own view of the game whenever the inputs arrive. Latency is distributed fairly to all players with no one getting an unfair advantage, and no one suffering doubled latency.
True peer-to-peer with no host is only possible because of rollback netcode. The need for a host comes from the need for an arbitrator to decide the one true input sequence, which comes from the fact that in most games, clients cannot recover from simulating the wrong input sequence. Rollback netcode allows clients to recover from mispredictions, eliminating the need for a host. Every player can form their own independent picture of the game state, and it will eventually converge to the same state for all players.
Perfectly-located servers
In traditional multiplayer games, the servers are in a fixed location, and they are never quite perfectly placed to the current set of players that are online right now. Easel's peer-to-peer architecture is like there is a virtual server effectively located at the exact midpoint between all current players, giving them the best possible latency for the players right now.
Because the peer-to-peer virtual server can sometimes be "placed" in geographically impossible locations, like the middle of the ocean, Easel is able to achieve lower latency than traditional multiplayer architectures could ever hope to achieve.
Flexible regions
Traditional multiplayer games have fixed servers in fixed locations, and so players must select a server region to play in. This subdivides the player base, making it harder for the game to reach a critical mass of players.
As Easel games are peer-to-peer, it can match players more flexibly. Easel uses geolocation to determine which players are close enough to each other to play together, regardless of if they are on different continents. For example, if one player is in New York and another in Paris, they might be on different continents, but they are actually close enough to play together on Easel with good latency. This means Easel can help your game reach a critical mass of players more easily.
See Regions to learn more about how Easel matches players together.
Relayed through Cloudflare
Peers connect to each other through their nearest Cloudflare servers, which protects their IP address, and also ensures that players behind NATs and firewalls can still connect to each other. Cloudflare has an international network of 400+ data centers, with most people on Earth being within 10ms of a Cloudflare data center, and so this gives excellent performance for most players while protecting their privacy.
Fan out by Cloudflare, not client: Normally in a peer-to-peer architecture, if a player must connect to 10 other peers, it must send 10 copies of the same message, which can cause congestion on the player's network connection and affect their latency. In Easel, the message is sent just once to Cloudflare, and then Cloudflare's server fans it out to the other peers. The Cloudflare servers are connected to each other through high-bandwidth links so they have no trouble handling the load.
Fair latency distribution
Sometimes not all latency can be eliminated by rollback netcode. In these cases, Easel assigns the latency so that each player experiences the latency they contribute to the game. If one player is far from the others, they themselves experience their own high latency and the other players are unaffected.
This is the fairest way to handle latency in multiplayer games. Other rollback netcode implementations are only able to assign latency to all players equally.
Automatic rubberbanding
When a rollback occurs, the predicted game state may differ from the actual game state. Easel uses rubberbanding to smooth out prediction errors, making them less jarring to players. This happens automatically and most game developers won't even know it is happening!
Server-authoritative
Even though messages are sent peer-to-peer, the Easel server still has ultimate authority over the input sequence. If there is any disagreement between peers, the server's version of the truth wins, meaning it is impossible to cheat.
Lag spike protection: One of the biggest problems with peer-to-peer is one player can introduce lag for everyone else, and if there is a lag spike, it is difficult to confirm which side is at fault. As Easel uses a server-authoritative peer-to-peer model, the server is able to detect when one particular player is causing lag for everyone else, so it is able to mitigate lag spikes from individual players, ensuring a smooth experience for everyone.
Continuous time synchronization
When packets arrives late, Easel must decide whether the difference is due to either latency or clock differences, and so good time synchronization is essential for rollback netcode to work well.
A common way to do time synchronization is to measure the round-trip time of packets. The packets need to return as fast as possible to get an accurate measurement, which means the network must be uncongested. This is not practical - in games we need to be downloading assets, and sending and receiving inputs all the time. Players do not want to wait 5 seconds for the network to do nothing except synchronize clocks.
Easel uses an innovative in-band time synchronization algorithm which piggybacks on the existing game inputs messages themselves that need to be sent anyway. This makes it both reliable and efficient.
The hard work is done for you
Implementing rollback netcode correctly and performantly is fiendishly difficult, but we have done all the hard work for you.
Guaranteed determinism
For rollback netcode to work, the game must be deterministic, which means it must always produce the same output given the same input. There are many ways that non-determinism can creep into a game - floating point calculations, random number generation, inconsistent trigonometry functions, differences in order of execution, and so on. We have painstakingly made sure that anything written in the Easel programming language is always deterministic. You never need to worry about accidentally write non-deterministic code in Easel, and so you can never cause desynchronization bugs yourself.
Snapshotting executing code
Easel games are sometimes made up of thousands behaviors, all executing concurrently. Easel incrementally snapshots the execution state of all behaviors, and restores them correctly during rollback. We have never seen a rollback netcode implementation that can snapshot code mid-execution.
A programming language built for multiplayer
Easel was specifically designed for multiplayer games. We chose not to use embed Lua, JavaScript, or any other existing programming language because they did not go far enough to achieve:
- The precise level of deterministic execution required for rollback netcode
- The ability to snapshot and restore execution state of coroutines
- Our vision for how to make a domain-specific game programming language that is powerful enough to be interesting, yet simple enough to be accessible to everyone
No understanding required
Easel multiplayer implementation is full of technological advancements, but ultimately, you don't need to understand any of this to make multiplayer games with Easel. You can just code as if all your players are in one shared world, like a singleplayer game, and Easel takes care of the rest. See Multiplayer to learn how to make multiplayer games with Easel,