Rollback Netcode, Explained
Your players are coming from different corners of the world. A player presses the jump button, but there is a delay while that input packet crosses oceans to reach their opponent. This delay is called latency, and it can make games feel unresponsive. Rollback netcode is a technique that hides this latency, making a game feel responsive even when players are far apart.
Depending on your playerbase, it is common to play against people who are a 50-100 milliseconds away. Without any special techniques, this delay will make your game feel laggy.
What is rollback netcode?
Normally, a game would have to wait for all inputs to arrive before simulating a frame. The waiting time is what causes the latency. You press jump, but then there is a 50-millisecond delay while your computer waits for your opponent's inputs to arrive.
Rollback netcode allows your game to move forward without waiting. The game predicts what the missing inputs are likely to be, then simulates the next frame using those predicted inputs. If later the actual inputs arrive and they differ, the game rolls back to the point in time where the prediction was made, and then re-simulates the game using the real inputs, correcting any misprediction errors.
Rollback netcode lets the game appear to respond immediately to player inputs, even when there is significant latency between players. Press jump, and your character jumps immediately. It is commonly used in fast-paced player-vs-player multiplayer games where the milliseconds matter.
Rollback netcode is commonly used in fast-paced player-vs-player games such as Rocket League, Street Fighter, and Mortal Kombat. These are games where the milliseconds matter and there is a lot of player-to-player interactivity, so they are well-suited to rollback netcode.
Misprediction errors
Your opponent is walking towards you. Rollback netcode predicts the opponent will keep walking forward, and so it simulates the next frame with that prediction. But later, once their input arrives, we discover they turned around in retreat and so the prediction was wrong.
In this case, there will be a little blip when the game corrects itself, but it can be smoothed over using rubberbanding. Normally the correction is so small and quick that the correction is barely noticeable. Most importantly, the game feels smooth and responsive whatever happens.
Why does rollback netcode work?
Generally players only provide input a few times a second, while the game is simulated 60 ticks per second, and so most frames can be predicted accurately without needing to wait for all inputs to arrive. That is why rollback netcode is able to hide latency so well in practice.
Why is rollback netcode hard to implement?
Rollback netcode requires two things:
- The game must be able to snapshot and restore a previous game state, so that it can roll back to a previous point in time and re-simulate from there.
- The game must be deterministic. That is, given the same inputs, it must always produce the same outputs, on every computer, no matter what.
These requirements must be fulfilled 100% correctly across your entire game's codebase. If even one line of code is not deterministic/snapshottable, it will cause desynchronization bugs where different players see different game states, ruining the multiplayer experience.
Most game engines cannot support rollback netcode
When you are coding everything yourself, you can ensure that everything you write is deterministic and snapshottable. But the problem is that most games are built on top of other people's code - normally a game engine, a game framework, or a physics engine. These will not be 100% deterministic and snapshotable unless they were specifically designed to be that way.
That is why most game engines cannot support rollback netcode. When people use rollback netcode in a game engine that was not designed for it, they do it by reimplementing certain parts of the engine themselves, which can be a lot of work.
Famously, Nintendo's Super Smash Bros Ultimate, chose to avoid rollback netcode as "the side effects were too big". Super Smash Bros is a fighting game that should be very suitable for rollback netcode, but when your codebase is large and consists of both your own code and third-party code, it is very difficult to guarantee that everything is 100% deterministic and snapshottable all the way through.
A rollback programming language
There is an easier way!
Bake the rollback netcode into the programming language itself, and then anything you write in the language will automatically be rollback netcode-ready. This is the approach that Easel takes.
In Easel, multiplayer has been built into the fabric of the Easel programming language, which means that anything you write in Easel is automatically multiplayer-ready. All the hard work is done for you, so you can just code as if all your players are in one shared world, like a singleplayer game, and without any additional effort from you, Easel makes your game multiplayer, completely automatically. It's like magic!
Easel's state-of-the-art rollback netcode implementation
Easel has one of the most advanced rollback netcode implementations of any game engine. This section details the unique features of Easel's rollback netcode.
Automatic multiplayer
Easel's rollback netcode is baked into the programming language itself, and so it is completely automatic. It transparently applies rollback netcode to all your game code, without you needing to do anything.
Incremental snapshotting and rollback
Many rollback netcode implementations must snapshot the entire game state at every tick. Sometimes this takes longer than executing the game simulation itself. In Easel, only the changes are snapshotted and rolled back, not the entire game state, which improves performance dramatically. We call this incremental rollback netcode.
Incremental rollback netcode enables bigger, more interesting games with more players and more content.
Easel uses a copy-on-write mechanism to implement incremental rollback netcode. When a value is changed for the first time in a tick, its original value is snapshotted. Normally when you change a value, you are already reading it, meaning it is likely already in hot CPU cache, making snapshotting much faster. All of this is completely automatic and invisible to the game developer as it's baked into the fabric of the Easel programming language itself.
Incremental physics
Physics engines allow you to simulate gravity, bouncing, jumping, collisions, etc and are often a core part of games. Physics engines normally pose a big challenge for rollback netcode because they hold a lot of state, and are not normally designed to be deterministic.
Thankfully, Easel's custom-built physics engine is designed specifically for incremental rollback. Only the parts which change are snapshotted, making it possible to have large worlds without performance problems. Easel simply doesn't snapshot/rollback any of the static geometry in your game, e.g. as the ground, walls and other obstacles, enabling you to make bigger levels that your players can truly get themselves lost in.
Because rollback performance is not an issue for Easel's custom-built physics engine, Easel is able to support some important features missing from other rollback netcode implementations, such as continuous collision detection which is important for fast-moving objects like bullets. Think about how many games have bullets. Lots! Continuous collision detection is a must-have.
True peer-to-peer
Easel uses peer-to-peer networking because this means every packet can take the shortest, most direct route between players, minimizing latency.
A common design of peer-to-peer games is for one of the player's computers to act as the host. This gives the host an unfair zero-latency advantage, while doubling the latency for all other players as they must wait for a full roundtrip to the host and back.
In Easel, there is no host player. Inputs are sent to all other players equally, as quickly as possible, and every client forms its own view of the game whenever the inputs arrive. Latency is distributed fairly to all players with no one getting an unfair advantage, and no one suffering doubled latency.
True peer-to-peer with no host is only possible because of rollback netcode. The need for a host comes from the need for an arbitrator to decide the one true input sequence, which comes from the fact that in most games, clients cannot recover from simulating the wrong input sequence. Rollback netcode allows clients to recover from mispredictions, eliminating the need for a host. Every player can form their own independent picture of the game state, and it will eventually converge to the same state for all players.
Perfectly-located servers
In traditional multiplayer games, the servers are in a fixed location, and they are never quite perfectly placed to the current set of players that are online right now. Easel's peer-to-peer architecture is like there is a virtual server effectively located at the exact midpoint between all current players, giving them the best possible latency for the players right now. If the players change, the virtual server's location changes too.
Because the peer-to-peer virtual server can sometimes be "placed" in geographically impossible locations, like the middle of the ocean, Easel is able to achieve lower latency than traditional multiplayer architectures could ever hope to achieve.
Fair latency distribution
Sometimes not all latency can be eliminated by rollback netcode. In these cases, Easel assigns the latency so that each player experiences the latency they contribute to the game.
Imagine a game where 9 players are in New York, but there is just 1 single player in Paris. That one Paris player is contributing almost all the latency to the game, and so Easel will allocate them most of the latency.
With the virtual server being located at the midpoint between all 10 players, it will located as if it is 1/10th of the way from New York to Paris in the middle of the Atlantic Ocean. This results in the Paris player experience 9/10ths of the latency, which is fair because they are the one contributing it. You get the latency you give. This is the fairest way to handle latency in multiplayer games.
While in other rollback netcode implementations are only able to assign latency to all players equally, in Easel, latency can be asymmetric.
Flexible regions
Traditional multiplayer games have fixed servers in fixed locations, and so players must select a server region to play in. If they select North America, they cannot play with those in Europe, and vice versa. This subdivides the player base, making it harder for the game to reach a critical mass of players.
As Easel games are peer-to-peer, it can match players more flexibly. There is no region selector. Players simply join, and Easel uses geolocation to determine who they are close enough to play with, whether they be in North America, Europe, Asia or anywhere else in the world. The region doesn't matter, just the distance. This means Easel can help your game reach a critical mass of players more easily.
See Regions to learn more about how Easel matches players together.
Protected by Cloudflare
Peers connect to each other through their nearest Cloudflare servers, which protects their IP address, and also ensures that players behind NATs and firewalls can still connect to each other. Cloudflare has an international network of 400+ data centers, with most people on Earth being within 10ms of a Cloudflare data center, and so this gives excellent performance for most players while protecting their privacy.
Fan out by Cloudflare, not client
Normally in a peer-to-peer architecture, if a player must connect to 10 other peers, it must send 10 copies of the same message, which can cause congestion on the player's network connection and affect their latency. In Easel, the message is sent just once to Cloudflare, and then Cloudflare's server fans it out to the other peers. The Cloudflare servers are connected to each other through high-bandwidth links so they have no trouble handling the load.
Continuous time synchronization
When packets arrives late, Easel must decide whether the difference is due to either latency or clock differences, and so good time synchronization is essential for rollback netcode to work well.
A common way to do time synchronization is by explicitly sending packets to measure round-trip time. The packets need to return as fast as possible to get an accurate measurement, which means the network must be uncongested. This is not practical - in games we need to be downloading assets, and sending and receiving inputs all the time. Players do not want to wait 5 seconds for the network to do nothing except synchronize clocks.
Easel uses an innovative in-band time synchronization algorithm which piggybacks on the existing game inputs messages themselves that need to be sent anyway. This makes it both correct but also efficient.
Because Easel uses in-band time synchronization, the game can continually synchronize the clocks throughout the game, correcting for any drift without any additional overhead. This is great for games that run for hours at a time.
Server-authoritative
Even though messages are sent peer-to-peer, the Easel server still has ultimate authority over the input sequence. If there is any disagreement between peers, the server's version of the truth wins, meaning it is impossible to cheat.
Lag spike protection
One of the problems with peer-to-peer is one player can introduce lag for everyone else, and if there is a lag spike, it is difficult to confirm which side is at fault. As Easel uses a server-authoritative peer-to-peer model, the server is able to detect when one particular player is causing lag for everyone else, so it is able to mitigate lag spikes from individual players, ensuring a smooth experience for everyone.
Let's say one player's WiFi cuts out temporarily, delaying all their inputs for 10 seconds. Once the inputs arrive, do we make all other players roll back to 10 seconds ago? This would cause a big stutter as all players recompute 10 seconds worth of game simulation. Instead, Easel reschedules the late inputs to be processed at the current time. The lagging player will see a big blip as their client synchronizes with the 10 seconds they have missed, but everyone else will be unaffected.
Automatic rubberbanding
When a rollback occurs, the predicted game state may differ from the actual game state. Easel uses rubberbanding to smooth out prediction errors, making them less jarring to players. This happens automatically and most game developers won't even know it is happening!
The hard work is done for you
Implementing rollback netcode correctly and performantly is famously difficult, but we have done all the hard work for you.
Guaranteed determinism
For rollback netcode to work, the game must be deterministic, which means it must always produce the same output given the same input. There are many ways that non-determinism can creep into a game - floating point calculations, random number generation, inconsistent trigonometry functions, differences in order of execution, and so on. We have painstakingly made sure that anything written in the Easel programming language is always deterministic. You never need to worry about accidentally write non-deterministic code in Easel, and so you can never cause desynchronization bugs yourself.
Snapshotting executing code
Easel games are made up of thousands behaviors, all executing concurrently. In programming language terms, these are called coroutines, and they are a very popular because they let you express your game logic as a bunch of independent step-by-step sequences.
Coroutines are normally very difficult to snapshot and restore because they are not just data, they are executing code. In fact, we have never seen a rollback netcode implementation that can snapshot coroutines, except for Easel. Easel was designed from the ground up to be able to support this, which is why you can use this powerful programming construct in your multiplayer games.
A programming language built for multiplayer
Easel was specifically designed for multiplayer games. We chose not to use embed Lua, JavaScript, or any other existing programming language because they did not go far enough to achieve:
- The precise level of deterministic execution required for rollback netcode
- The ability to snapshot and restore execution state of coroutines
- Our vision for how to make a domain-specific game programming language that is powerful enough to be interesting, yet simple enough to be accessible to everyone
No understanding required
Easel multiplayer implementation is full of technological advancements, but ultimately, you don't need to understand any of this to make multiplayer games with Easel. You can just code as if all your players are in one shared world, like a singleplayer game, and Easel takes care of the rest. See Multiplayer to learn how to make multiplayer games with Easel,