State synchronization across connected virtual-worlds
Multiplayer immersive game development is one of the most interesting projects for indie creators and teams alike, presenting challenges covering a wide-spectrum from user experience to distributed computing.From Minecraft to WoW open ended MMOs have entertained and educated us and created communities for gamers and trolls alike and also pushed the boundaries of our collective creativity and imagination.
King’s landing recreated in Minecraft [Source]
In recent years, MMO game-dev has seen a transition from intricate worlds created via hundreds of thousands of lines-of-code to player-defined worlds which provide building blocks for creating persistant artifacts and experiences. As the cutting edge in game economics, storage and UX technologies hit mainstream applications, it’s not overtly optimistic to assume that we’ll see some truly amazing MMO gameplay and immersive virtual worlds in the time to come..
For some perspective, WoW at its peak had 12 million monthly active players. LoL which is the largest online Multiplayer game has over a 100 million monthly active players (Although it is not really a MMO and maintains rooms of 10 players hosted on a single shard). By contrast, Youtube has 1.5 Billion monthly active users and Facebook has 2.2 billion monthly active users. Last year during Oculus conference, Mark Zuckerberg gave a teaser of things to come…
Why you s̶h̶o̶u̶l̶d̶n̶’̶t̶ absolutely should be creating a MMO —
Client SDKs, game-networking and cloud-native database stacks provide standardized solutions for most common access-patterns. It has historically never been a better time for creative professionals and programmers to come together and create MMO prototypes for emerging clients. Crowd-funds and ICOs have often removed the barriers the for indie game developers and small teams to go from prototype to production. Here are some major technological and social factors that will propel immersive world applications into the future.
1. Liquidity for game tokens in the real world
Liquid game tokens can be created on the blockchain
It’s far too long been the case that the only multiplayer games outside of big production studios making money were the ones which provide services with Fiat such as Poker or Betting games..Game currencies, such as Linden Dollars (the currency of Second Life) while pioneering struggled to gain credibility and acceptance.. mostly because they were created out of thin air by game studios…many recently published white-papers and proof-of-concepts illustrate that game token and virtual world currencies are one of the most natural use-cases of the blockchain. For instance, if the currency earned through industry in the virtual world can then be used in the real world and vice-versa and we’ll see many more interesting artifacts and development in immersive massively multiplayer landscapes. imagine you being able to hold an auction for the house or the bridge you built in Minecraft in game tokens and being able to convert the game tokens immediately to Fiat.
2. Production ready VR SDKs
Good news everyone ..and not just through netsuits
Virtual Reality and Augmented reality have finally crossed the peak of inflated expectations and are almost past the trough of disillusionment of the the Gartner hype cycle as real world usages are being tested and factories are creating production lines. From Unity and Unreal Engine major game engines are providing VR app client SDKs natively and vendors are already churning out VR devices for niche use-cases.
3. Micro-services enabled seamless gameplay across clients
The most played MMOs will provide cross-client experiences
When the app middleware is built as micro-services, it allows you-the game developer to support gameplay for multiple game clients and progressively enhance game play for more advanced clients. Imagine an MMO where players can do the full spectrum of actions such as participate in battles via the oculus rift client but are also able to chat with other players or water their garden via the android client.
4. Cloud-native, auto-sharded distributed storage engines
Cloud native databases provide consistent low latency reads and writes across regions and zones
Network latency has been the issue with playing online games even at moderately high FPS. This is because any network traffic over the public internet will incur latency resulting in considerable lag. Using dedicated networking provided by cloud service providers, we can limit the interaction with public internet to the last few hops, drastically improving P99 latencies over a large number of reads and writes. Advances in distributed database technology supporting multiple access patterns and providing fast reads and writes means that a lot of game networking and load-balancing logic can move from the middleware, down to the data layer.
Game middleware architecture
1. Authoritative servers and dumb clients
To the user the client is the game, clients will render all the graphics and allow user actions, the game client can be as rich as Runescape or WoW or as simple as slitherio. The client should not however authoritatively execute any logic based on user action and must act on values provided by the server. The reason for this is that we must always assume that the client is an attacker when developing a game in multiplayer contexts. For instance if the client were allowed to dictate it’s position and propagate it to other clients, it may become trivially easy for a malicious player to hack their coordinates and set it to be on the enemy flag and propagate it’s position to other very confused clients. It’s also easy to see how an authoritative server architecture scales better than authoritative client architectures.
Instead for every user action, the client must encode the state into a data structure and pass it down to the server which then interprets it against the rules of the game to see if the action makes sense, it then sends the position back to client which then regards it as it’s final position.Additionally the server might take a batch of such inputs and run it against an analytics and inference engine to verify player actions. For instance did the player just hit 10 out his last 10 shots, are the mouse clicks on the screen differing from input coordinates?
2. Network latency and state propagation
Having authoritative server means that clients cannot cheat but it also means that the when you press the key W, the client sends the request and waits for the server response to tell it to move forward, when this happens over the internet it will visibly cause lag in the game experience. To get over this game clients use several smoothening techniques such as client side prediction, server reconciliation and interpolation. This article does not go into the details of client side compensation techniques, you can check out this excellent explanation here which also includes this demo.
On the server side, we can reduce network latency by limiting the number of hops made over the public internet and by using high speed network peering provided by IaaS providers as much as possible. YugaByte supports single click deployment and scale-up/down operations on primary nodes and read replicas in both AWS and GCP, while leveraging VPC peering to enable high speed networking and private traffic across regions and data centers.
3. Server latency and linear scale-out models
So how come we can have millions of people viewing the same webpage but only thousands on the same shard in an MMO?
In the article titled MMO Population Problems, the author explains how the network and server load multiplies exponentially on a MMO shard and the problems in handling state consensus in the application layer.
Scaling out from 25 to 50 nodes doubles read and write throughput Source
YugaByte DB solves the problem of scale by providing models that linearly scale-out instead of scaling up. For instance doubling the size of a 25 node cluster to 50 nodes causes the performance and throughput to double. You read the benchmark details here.
4. Automatic sharding and load balancing
When creating game databases, teams are often faced with tough questions about the physical and logical partition of the data servers like — how many databases should I have per node? How should I decide which nodes I need to scale faster than others. This creates a chicken and egg problem in which you need to know access patterns of game interaction before you have actually built the game. YugaByte simplifies this for game developers by providing auto-sharding data models from the outset that let’s you elastically scale out your cluster as your game grows.
The game servers could simply sit behind dumb load-balancers serving requests from nearest clients, which will in turn return the most recent consistent snapshot of the data maintained via MVCC. Load balancing is handled on the data layer through auto-sharding which automatically splits tables based on game models into multiple shards. This allows your team to not have to manage load-balancing in the application layer and your architecture would look more and more like if you were developing a single player game.
Sharding of data into tablets. Source
5. Read replicas for high-speed read access patterns
MMOs access patterns usually follow frequent reads (for all the actors in the grid) and infrequent writes (for player actions)In many access patterns where the actor is not writing but only reading, timeline consistent reads from async replicas may provide lower latencies. For instance if the player is not doing anything and only needs to read the data of all actors in their grid of reference, they can get timeline consistent copies from a Read-Replica. Read-Replicas may also be used to run cluster aware analytics jobs on live data to infer real time intelligence from player actions without straining the primary nodes.
Efficiency of modeling APIs
Games are computationally taxing applications and often test the limits of clients, networks and servers (on a scale), which is also the reason why efficiency matters most just just in the space it takes store the dirty state but also the code needed to serialize/deserialize it. Most MMO games follow access patterns dictating fast reads for the state of all objects in the player’s grid and fast write for the player’s information when the player does an action. Redis API provides simple constructs to store game state objects with minimal serialization/deserialization costs which can be used to build more complex structures. Often in browser games, the game state is passed down as a JSON document via HTTP.
1. Redis Modeling
Redis API provides developers with a set of very useful data structures such as sets, sorted sets, and hash maps that can be used to build more advanced structures that model most state transitions in MMO objects efficiently.
However traditional Redis implementations as a pure in-memory cache has its own limitations, the first is that the entire data-set needs to fit in-memory which might be a challenge for game servers with high user activity in a single shard. Another layer of complication is that implementing Redis + DB models requires further access pattern logic integrated into the application layer. YugaByte solves this problem by providing a persistant Redis with Block-caches serving hot reads while providing sub-ms latency even for workloads that don’t fit the cache. You can read more about YugaByte Redis implementation here.
2. CQL Modeling
Fast access patterns for large datasets or those requiring strongly consistent distributed transactions such as marketplace transactions can be modeled over the CQL API. Moreover access patterns requiring range scans or queries that intuitively models the column family structure can be modeled via CQL API over the DocDB storage engine. YugaByte also supports Graph API access patterns via Janusgraph and KairosDB APIs for time series metrics.
3. JSON Modeling
MMO game states especially for web-based games are also often intuitively modeled as the JSON documents, Support for JSON type with Postgres API compatibility is in the works.You can find details about the PR here.
Real-time Analytics over game state
Any commercially viable virtual world will be running real-time analytics jobs on game data to gather intelligence on game play improvements, targeted advertising, cheating detection etc. Some examples of questions that you’ll likely be asking are —
- Is the matchmaker service matching players according to their skill levels?
- Are some players cheating? If so how are they cheating?
- What can I infer about player’s likes/dislikes etc. from gameplay patterns?
- Are players clicking on in-game advertisements?
Real time analytics using the SKY stack allows the services to be designed on top of the data layers which lets you infer meaningful knowledge from player actions.
Simplifying the MMO data and operations stack
Multiplayer game networking and state propagation is hard and definitely not something your team wants to focus on. While staying within the realm of what is feasible you want your models to be defined by providing the best user experience and not necessarily for minimizing data packets. YugaByte DB can simplify a game engine server stack by combining the caching and persistance layer into one, pushing the load-balancing and sharding logic down from the application layer to the data layer and use the speed of Redis with persistance guarantees for in-game state and CQL APIs for Player data and web-service models.
About me —
I am a developer at YugaByte where we build data solutions for distributed, high-density applications and games. If you or your team are building immersive virtual-world experiences and exploring scale-out data-models, you can check out the docs here.
Meanwhile you can head over to the Github project and take a deep dive into the code base.