News for hackers. Stuff that matters.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by 013a on Friday February 09, 2018 @12:45PM (#16342210)
c4.8xlarges are the smallest c4 instance that guarantees 10G networking performance.

Those costs can easily be cut in half with reservations. Its likely there's a lower-bound of the number of reserved instances they use as a baseline performance guarantee, then they use on-demand to scale above that.

No one at their scale actually pays the list price.

Basic math: There are ~100 players in each game. At a 30 tickrate, that's 3000 RPS per game minimum. Each of those requests likely involves a number of 3D math calculations, including hit detection, collision detection, real-time cheat detection, etc. Those updates then need to batch back to the players at 30 tick. All of this needs to happen with as little eventual consistency as possible; a difference of milliseconds degrades the player's confidence that the server is correctly calculating what is happening in-game.

Point being; multiplayer game programming is an entirely different beast than normal web programming. The same rules don't apply. Its an n^2 problem where every additional user in one "lobby" actually increases resource utilization exponentially because you need to update every other user's game-state with the actions of that new player. Additionally, Battle Royale style games are the most demanding multiplayer games ever created. Games like WoW have way more players per realm, but servers only need to worry about the interactions of a select few in your surrounding area, and it doesn't have as stringent real-time requirements. Games like CoD only load in 5-20 players.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by rocqua on Friday February 09, 2018 @07:27PM (#16345021)
> Additionally, Battle Royale style games are the most demanding multiplayer games ever created.

I'd like to draw your attention to Planetside 2. A pretty recent FPS that was run a lot like WoW, with 3 'continents' and complete seamlessness on those continents. Firefights where 100 people were actively engaging each other were an almost constant experience back when I played it. I think there was a cap of 1000 players per continent. Those players could gather wherever they wanted. I'm pretty sure I was in some 300 player battles, and nothing but lag would prevent all 1000 people on a continent from going to the same base.

It has to be said that in the big battles (at around a 100) lag became an issue. I'm not sure whether this was clientside, serverside or both.

(There is also eve-online where recently there was a 'fight' with 5000 players in the same location, but that is a different beast from FPSes, and the main logic of that game is single-threaded)

Re:Postmortem of Service Outage at 3.4M Concurrent Users by JCharante on Friday February 09, 2018 @08:35PM (#16345315)
> It has to be said that in the big battles (at around a 100) lag became an issue.

I remember the crown frequently having 96+ per faction without any lag problems. Must have been clientside.

> (There is also eve-online where recently there was a 'fight' with 5000 players in the same location, but that is a different beast from FPSes, and the main logic of that game is single-threaded)

It was 6000, and it was terrible to be in. It took me 10 minutes to enter a station (normally takes a second) and I had to relog because I couldn't get out of the station. Relogging took another hour of black screens. It was pretty neat when I finally got out, but there wasn't much of a fight compared to previous encounters.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by wyager on Friday February 09, 2018 @11:45PM (#16345938)
Planetside 2 was horrendously single-threaded, so anything but recent Intel chips would leave you with really bad performance in large battles. I would often max out my AMD CPU (well, one core of it) at around 12 FPS without my graphics card breaking a sweat in 100-200 player battles.

PS2 was for the most part an amazing game. The graphics were truly beautiful and the gameplay was very fun. Unfortunately, their shitty performance basically killed the game by making it unplayable for many users during the trademark huge battles.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by rocqua on Saturday February 10, 2018 @06:37AM (#16346886)
I recall some fights in biolabs that were really laggy, same with some fights at crossroads watchtower. The issue was really when a lot of people had mutual line of sight and we got grenade spam (especially revive grenade spam).

I might be off on the numbers though.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by vvanders on Friday February 09, 2018 @02:07PM (#16342973)
FYI most game network servers tick the network at 10-12Hz(although the internal sim may run higher).

We generally didn't go higher than 10Hz since you'd start saturating lower end connections and any well written game will be good at reconciling state via dead-reckoning. You have to handle 100ms+ spikes anyway so it doesn't make sense to run the network super-fast.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by AlphaSite on Friday February 09, 2018 @05:02PM (#16344231)
Most competitive games go well above 10Hz ticks, OW does it, CSGO does it (I don't really know the tick-rates of other games but when it matters, so no comment there, but there is an important area for it).

OW just has a restricted rate option for low bandwidth connections (added when they went from 23 -> 61 Hz).

Re:Postmortem of Service Outage at 3.4M Concurrent Users by vvanders on Friday February 09, 2018 @05:17PM (#16344343)
So that might be the case for LAN games, generally internet tends to stay lower. Valve for instance recommends 13Hz[1] for Source(CSGO):

> // 60 for updaterate is LAN ONLY use 13 for internet

> // 20 is default but will cut the maxplayers you can handle in 1/2

> // for SRCDS Servers use 30 - you might be able to use 20

> // sv_maxupdaterate 60

> sv_maxupdaterate 13

Like I said, depending on how good your potential-vis code is you may be able to go higher but you risk saturating your client links and most game netcode doesn't handle congestion nicely. My info is a little out of date(2-3 years) but back then we had to keep data rates below 20kb/sec(and ideally 10kb/sec) if you wanted low jitter across the link.

[1] https://support.steampowered.com/kb_article.php?ref=5386-HMJ...

Re:Postmortem of Service Outage at 3.4M Concurrent Users by gsich on Friday February 09, 2018 @10:05PM (#16345624)
This information is very much outdated. (Windows XP is referenced in it)

CSGO matchmaking servers (run at 64 ticks) force you to use cl_updaterate 64.

Even in CS 1.6 days, most leagues and servers enforced updaterate 100 and rate 20000. Nearly every player could handle it. And those who couldn't, well, bad luck for them. Playing with 100ms+ is no fun for anybody.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by vvanders on Friday February 09, 2018 @10:27PM (#16345694)
Like I said in the root, talking generally 10-12Hz is really common because you need to handle dropped packets cleanly.

I don't know about the CS 1.6 days, I used to play a ton of early cs and was an admin on one of the most popular Frontline Force servers. We ran tick rate of 12 if I recall correctly.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by gsich on Friday February 09, 2018 @10:42PM (#16345739)
It is mainly because PUBGs network code sucks. You can see the effects here [0]

Unless you had dial-up users, a tickrate of 12 is very bad and not fun. I don't know if you had hardware restrictions, but tickrate is independent of updaterate that is sent to the clients. By enforcing a low tickrate you maintain a very old snapshot of the world on your server (83 ms), which makes clients interpolate more.

[0] https://www.youtube.com/watch?v=u0dWDFDUF8s

Re:Postmortem of Service Outage at 3.4M Concurrent Users by Tobba_ on Saturday February 10, 2018 @06:46AM (#16346916)
CS(:GO) would be completely unplayable at any update rate below 50. Especially with the necessary interpolation, you would end up with delays of over 100-200ms+ at an update rate around 10. Most weapons in the game could even fire more than one shot in that time.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by dvlsg on Friday February 09, 2018 @10:23PM (#16345680)
Strong disagree for the recommended tick rate for csgo. I played a decent amount of competitive cs, from 1.3 all the way to csgo. Lots of competitive server hosts will go up to 120 (ESEA comes to mind) , and the difference between 120 and 60 is certainly noticeable. Going down to 13 would be almost unplayable.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by learc83 on Friday February 09, 2018 @06:14PM (#16344652)
>So that might be the case for LAN games

Overwatch has a 63Hz tick rate over the internet.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by vvanders on Friday February 09, 2018 @06:26PM (#16344713)
And they handle congestion cleanly[1] which is one of the things I predicated you need if you didn't do 10-13Hz(along with smarter vis sets).

Every game is going to be unique and gameplay structure has a huge impact on your data rate. We had games back in '97 like Subspace[1] that handled hundreds of players in the days of dial-up and 200ms+ pings. Generally if your data rate per-link goes above 10kb/sec you'll start to see degradation of jitter across your userbase. There's smart ways to handle that but if you want to do a fixed-tick you run up against that limit really quickly.

[edit]

Other games will do smart things to keep skill high and tick rates low. CS(but not CS:Go that I know of) used to famously rewind time based on your latency so your view of the world was accurate, but lead to fun behaviors like rubberbanding as you got shot around a corner(due to movement speed slowing down at that past point in time and the game having to reconcile it with the current timestamp).

[1] http://www.eurogamer.net/articles/2016-08-12-looks-like-bliz...

[2] https://en.wikipedia.org/wiki/SubSpace_(video_game)

Re:Postmortem of Service Outage at 3.4M Concurrent Users by bcassedy on Friday February 09, 2018 @07:24PM (#16345005)
Man Subspace/Continuum was such a great game. I do recall issues with packet loss, but in general the game functioned really well even on my 33.6kbps modem with hundreds of players and pretty frenetic gameplay. It's not until you mentioned it that I realize how amazing that feat is.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by vvanders on Friday February 09, 2018 @07:42PM (#16345100)
Oh yeah, I love to hold up Subspace as a perfect example of why games is so cross-discipline driven. For networked games building design about predicting where someone will be means that frame-to-frame latency almost doesn't matter. The Newtonian physics of that game plays into it perfectly leading to a something that works over primitive modems. Even FPS games to a large degree are about predicting/leading the positions of the other players.

Compare that to say, fighting games which are very reactionary and very hard to run over any latent connection without some seriously complicated netcode involving time rewinding.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by Tobba_ on Saturday February 10, 2018 @08:32AM (#16347222)
CS:GO (and all other source games) do the same lag compensation as CS did (though it's horrifically buggy, but it was pretty messed up in 1.6 as well). People seriously underestimate how impossible it is to play fast-paced games without it - you wouldn't be able to hit a HS on a moving player in CS:GO even with ~30ms ping without leading your shot if that wasn't there.

It's also not actually why you rubberband - that's just caused by your own latency (moving while already dead). It does mean you can get hit after moving behind a wall though.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by gsich on Friday February 09, 2018 @09:59PM (#16345600)
Only PUBG and Fortnite have such shitty tickrates.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by troncjb on Friday February 09, 2018 @04:43PM (#16344094)
What do you mean by "game network servers tick the network"? Like data going server -> client ?
Re:Postmortem of Service Outage at 3.4M Concurrent Users by Thaxll on Friday February 09, 2018 @01:33PM (#16342664)
Since gameservers are mostly single threaded they probably run 35~~ of them on a single instance ( c4.8xlarges have 36 vcores ), so if you have a good matchmaking algorithm you can fit 3500~~ players per instance, to reach 3.5M you need between 1000 and 2000 instances.

As for your last comment it's wrong, a game like Battlefield is probably more demanding compare to a battle royale type of game, a FPS like Battlefield runs at 60hz and is using a lot of physics everywhere at fast pace, battle royal games don't have that much action going on.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by Flammy on Friday February 09, 2018 @01:49PM (#16342835)
Likely more than 35 games on a single instance, as ideally you wouldn't be starting all games at the same time on the same instance. Staggering the start of games on each instance would be ideal.

For example you could have dozens of late stage games that combined have less players than a single, just started, 100 person game.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by evanosaurus on Friday February 09, 2018 @02:08PM (#16342980)
We run 16 game servers per c4.8xl instance for Battle Royale. We run more than that (on a different instance type) for StW.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by Thaxll on Friday February 09, 2018 @02:00PM (#16342926)
You usually don't run two or more game server on single a thread / core, it kills performance, and with VMs with many physical processor you have other issues like NUMA so you have to be very careful.

I'm pretty sure they have a ratio of one gameserver per thread / core.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by evanosaurus on Friday February 09, 2018 @11:43PM (#16345935)
We don’t tick at 30Hz. Somewhat slower than that. Working on making that better, though.

And we do indeed take advantage of reserved instances. We’re also looking at other options along these lines.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by bpicolo on Friday February 09, 2018 @01:22PM (#16342568)
> and it doesn't have as stringent real-time requirements

Blizzard is absolutely stringent about their real time requirements for WoW. It's crucial for PvP, Raids, etc.

Fortnite doesn't need to broadcast nitty player info for players on the other side of the map to you, either, I would think. That should definitely be optimized away

Re:Postmortem of Service Outage at 3.4M Concurrent Users by kevingadd on Friday February 09, 2018 @07:41PM (#16345095)
Battle Royale games generally need to broadcast player state to all players within a large distance (iirc for PUBG, it's one km) because it's common to spot another player 500+ meters out and take some shots at them. If they're not moving much you might hit them.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by bpicolo on Saturday February 10, 2018 @03:08PM (#16349303)
Even if all 100 players are nearby, 100 isn't unusual in the context of MMOs. That's not to say it's trivial to do, it's just not a new feat unique to battle royale games.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by mbid on Friday February 09, 2018 @08:59PM (#16345407)
> Its an n^2 problem

> actually increases resource utilization exponentially

I think you'll have to choose there.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by welder on Friday February 09, 2018 @04:39PM (#16344063)
> No one at their scale actually pays the list price.

I've always wondered what AWS rates do large users receive from negotiations?

Re:Postmortem of Service Outage at 3.4M Concurrent Users by AlphaSite on Friday February 09, 2018 @05:03PM (#16344235)
MAG somehow managed to be a competitive FPS with 256 players back in 2008ish, its not an impossible battle.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by jsmeaton on Friday February 09, 2018 @05:07PM (#16344271)
I think MAG was using clever tricks. There were multiple “zones” per battle, and I think each of those zones was a separate instance with some coordination between.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by xchaotic on Thursday February 15, 2018 @11:50AM (#16385483)
to be fair you could apply the same tricks to any Battle Royale type game
Re:Postmortem of Service Outage at 3.4M Concurrent Users by bufferoverflow on Saturday February 10, 2018 @12:25PM (#16348309)
> actually increases resource utilization exponentially

Quadratically actually.

Exponential would mean each user doubles the amount of calculations, for instance.