I'm doing some live R&D in github for my next article. "How to create a first person shooter that scales to millions of players". https://lnkd.in/dNDs3t-4 So far R&D into player input processing with XDP indicate are that it's definitely possible to hit the 1M player mark with each player sending 1mbps of data from client -> server for inputs to a "player server". Each player server needs a 100G NIC and conservatively can process ~50k players each, so you need 20 player servers to hit 1M players at a cost of ~40c US per-player per-month so far. Blocked right now on what appears to be a ring buffer bug in XDP/BPF, so I'm going to move forward with researching other aspects of hyper scaling FPS netcode: 1. Simulating local player state forward in BPF hash maps, storing it back in the map post sim. Sending local player state back to clients on request in XDP. Should be relatively easy, given the same bandwidth amount back down to clients for player state as at is up for inputs. ~1mbps per-player @ 1200 bytes of state per-player. Just need to make sure the player simulation and BPF hash map accesses can keep up at 100HZ with 50k players. 2. Building snapshots containing other player state with the world divided into some geospatial areas. Most likely 10x10 grid for simplicity, eg. 100 "world servers". Estimating 1000 other players visible on average, gives 100*1000 bytes of player state at 100HZ. Expect this will be around 100mbps per-player. Yes, this part is definitely going to require 1G internet. You could bump to 10,000 other players visible for 1gpbs per-player, however I have concerns around line rate and packet loss amplification at this rate, even if you do have 10G internet. 3. Thinking about player simulation update code interaction with the world primitives. eg. raycasts, damage other player, find players in volume etc... it seems possible, but radically different to how game server netcode is written now for FPS. 4. A concept of a "world database" sort of like "Redis" for game servers enabling large game worlds much bigger than can be simulated or processed on a single machine is something that I keep coming back to. Player simulations must interact with other players and the game world somehow. If it all turns into asynchronous calls, like golang goroutines blocking on IO and then waking up to do work I can see how this can all come together. One goroutine per-player on the player server, async calling to this world database. Interested to hear your thoughts on this. What do you think?
Josip Petanjek
7mo
Really liked your article on XDP, glad to see this developing further. As you point out, it's a very context/assumption dependent problem and solution set. On point 2; I assume you are sticking to Quake-stlye/full world state? Does this mean we are getting a definitive "Gaffer on multiserver" blog post 😄?
Senior Netcode Engineer at Unity Technologies
7moRegarding your BPF bug (you probably already read up on this) if you are using virtual machines then there are a few additional timing related issues you have to overcome: https://kinvolk.io/blog/2018/02/timing-issues-when-using-bpf-with-virtual-cpus/ Another interesting article about AF_XDP: https://networkbuilders.intel.com/docs/networkbuilders/af-xdp-sockets-high-performance-networking-for-cloud-native-networking-technology-guide.pdf With a note about high performance: "While it is possible to use AF_XDP on any driver, it only achieves high performance levels if the Linux driver has zero-copy support added and has been optimized for this. " Looking forward to more of your findings...a very interesting project indeed!