News for hackers. Stuff that matters.
Re:Ask HN: What is the ops architecture like for AAA multipl... by dijit on Sunday September 08, 2019 @04:01AM (#20909190)
This question sounds like it's pointed directly at me.

However, I can only speak for one AAA gaming company, and my team operate a bit differently than most in the company.

My team operates the infrastructure for "Tom Clancy's The Division" video game series (1&2).

Most of the programming approach is spent on doing the cheapest (in terms of CPU) possible thing, everything is C++

Things like: matchmaking will happen ideally on a single machine with no shared state, everything will happen in memory, which is much faster and can be more reliable than any distributed state or failover.

(it's less reliable if you're in matchmaking and the server or service dies; But then everyone's client will reconnect to the newly scheduled matchmaking instance and populate in memory state.)

We use a lot of windows, nearly every machine that doesn't handle state is a windows server. This has pros and cons, from my ops perspective I try to treat windows like cattle, but windows doesn't like that. they have their own way of operating fleets of machines which include using SCCM and packaging things. There's nice GUI's, but we use saltstack and we removed AD, because it was a huge burden to: create a machine, link it to AD, reboot it, finally get a machine worth using.

From a dev perspective, Windows is good, IO Completion Ports is superior to the linux epoll in terms of interface and performance, so we can have machines that take 200,000+ connections.

How you decide which dedicated server you connect to is up to your client, it does a naive check as it's logging in where it will do a TLS handshake with a random gameserver in a region, for each region. (during the login phase we send your client a list of all currently active gameservers and an int to represent the region).

This works fine until there's packet loss on a particular path because your single ping might be fine but overall your experience could be better elsewhere; if you're not able to ping anything then we fall back on geoip.

That said, if you have friends on another datacenter, we try to put you on the same server. So that if you join groups or whatever then it's just a phase transition rather than a full server sync.

Everything is orchestrated with a "core" set of servers which handle player authentication and matchmaking centrally, then each of the gameserver datacenters (geographically distributed to be closer to players) is attached via ipsec VPN.

In Division 2 we spread out into GCP as well as having physical machines, so we developed a custom auto-scaler. The autoscaler checks the capacity of a region and how many players are currently in a region, keeps a record over 10 minutes and makes a prediction. If the prediction goes over the current capacity in 20 minutes or less, it will create a new instance (since making windows servers on GCP takes longer than linux servers)

If the prediction goes lower than the capacity of a server, it will send a decomission request to the machine, which takes up to 3hrs to complete (to give people time to leave the server naturally).

Idk, I've been doing this for 5 years now so I can talk at length about how we do it, but ultimately out biggest challenges are the fact that we can't use cool shit or new trends because latency matters a lot and we use windows everywhere.

--

As an aside; the overwhelming majority of other ubisoft games (excption: For Honor) use something very similar to what we released open source in collaboration with google to do matchmaking: https://agones.dev/site/

Re:Ask HN: What is the ops architecture like for AAA multipl... by Slartie on Sunday September 08, 2019 @04:24AM (#20909256)
What is the rationale behind using Windows on the servers? I'm wondering because the argument I usually hear is that someone high up in the ops team hierarchy declares "but we need everything in the AD, including all the servers, otherwise I can't sleep well". In your case however this apparently doesn't apply, as you do not register them in the AD.

Is it so the game devs don't need to write cross-platform server code?

Re:Ask HN: What is the ops architecture like for AAA multipl... by dijit on Sunday September 08, 2019 @04:40AM (#20909305)
Real reasons for:

* One platform to develop on; back-end coders tend to go back and forth between client and server programming.

* Faster iterations. (just hit F5 in visual studio)

* IOCP

Stupid reasons for:

* Old IT Director denied the use of virtualisation software. (mac address randomisation wasn't great and it caused a switch crash if two people had the same mac)

* Windows licenses are really cheap compared to developer time. (until we went to cloud, where Microsoft charges insane amounts for licensing)

Re:Ask HN: What is the ops architecture like for AAA multipl... by speedplane on Sunday September 08, 2019 @04:56AM (#20909355)
I no longer do windows development, but I miss it. It's strengths are greater than you portray.

The C# / .NET libraries are really beautiful and well thought out. I most often program in Python, and while the Python language is more concise and elegant than C#, the Python libraries are not nearly as well organized and consistent as C# / .NET. Java libraries are better than Python but worse than C#, and the Java language is worse than both C# and Python. As far as dev enivronments, Visual Studio is far better than Eclipse and roughly equal to Python's PyCharm. Because C# is compiled, it's far more performant than Python and a bit better than Java.

The downside of course, is that once you start down that road, you are pretty locked into the MS world.

If I was starting a new project, and I was a scrappy startup, I'd probably go with Python. But if I had resources (ie $$$) to support MS licenses and performance actually mattered, I'd go with MS. Side projects for experimentation may lead to node or rust. But sorry Java, can't really imagine a scenario where I'd start with you.

Re:Ask HN: What is the ops architecture like for AAA multipl... by brianpgordon on Sunday September 08, 2019 @10:21AM (#20910702)
I share your high regard for C# and the .NET libraries but not your disdain for Java. Java the programming language is pretty ugly but the JVM is robust and there are other languages which compile down to Java bytecode. In your "scrappy startup" scenario I would pick Scala or Clojure long before Python, mainly because I'm skeptical of the idea of using a dynamically typed programming language for large-scale backend development - for human scalability reasons, not performance ones. Personally I'd also have to pass on Windows even in the AAA bigcorp scenario, because I prefer the tooling and open ecosystem around Linux and epoll isn't that much worse than IOCP.
Re:Ask HN: What is the ops architecture like for AAA multipl... by dfinninger on Sunday September 08, 2019 @11:43AM (#20911190)
Python supports gradual typing now, and it's made the experience for me much more manageable. Lots of good IDE integration too. It's not a replacement for Java's or Go's (etc...) type system, but it really increases the amount of Python you can support given the human scalability constraints.
Re:Ask HN: What is the ops architecture like for AAA multipl... by scarface74 on Sunday September 08, 2019 @08:35AM (#20910167)
I was an exclusive Microsoft developer for a little over 20 years. It wasn’t until I started doing cloud deployments - like the original poster implied - that I started avoiding Windows as often as I could. Once you add Windows to the mix, everything gets worse - startup time, resource requirements, automation, snd costs.

Luckily, you can do C# without having to use Windows with .Net Core.

Re:Ask HN: What is the ops architecture like for AAA multipl... by brianpgordon on Sunday September 08, 2019 @10:28AM (#20910740)
Ooh, you may be the perfect person to ask a question that I've been wondering about. Is .NET Core on Linux viable yet for serious use? I've done some tinkering with it and gotten a project working on MacOS but I did the development in Visual Studio on Windows. Is it realistic to do your development in Rider, with the open source version of msbuild for CI, and have it perform reasonably on Linux in production, without ever paying Microsoft anything?
Re:Ask HN: What is the ops architecture like for AAA multipl... by throwaway8941 on Sunday September 08, 2019 @11:10AM (#20910993)
The kind of development we do probably wouldn't count as "serious" here on HN, but since the other comment didn't really answer your question, I'll pitch in. We've been developing using .NET Core/Linux/Rider for the past two years. It's been pretty great. Once you get to know the IDE really well, I'd say Rider beats VS in terms of functionality and usability.

.NET Core was developed with CLI usage in mind and common operations (like migrations/boilerplate generation) can be driven from CLI or Rider, you don't need VS for that.

You may have some problems with 3rd-party libraries. Not all of them have been ported to .NET Standard. I'd recommend checking your project's dependencies in advance. (We did have some problems with legacy COM-based cryptographic libraries and had to build a separate "microservice" hosted under Windows to offload this stuff to. COM stuff works fine under .NET Core, but only under Windows (obviously).)

Re:Ask HN: What is the ops architecture like for AAA multipl... by scarface74 on Sunday September 08, 2019 @01:17PM (#20911840)
If you are just running a few static Windows servers the slight cost benefit of running Linux for .Net Core is really not that great. If you are doing anything more dynamic where you are rapidly bringing “servers” up and down is where you see the real cost and performance benefits. Of course you can’t run Windows instances with either Fargate or lambda but even with regular ECS (Docker) or autoscaling EC2 instances where you can, Windows costs more, takes more resources, and is slower to launch.
Re:Ask HN: What is the ops architecture like for AAA multipl... by scarface74 on Sunday September 08, 2019 @10:57AM (#20910913)
My “Linux deployments” with C# have been Docker and Nginx with Fargate (AWS Serverless Docker) [1] and Lambda. I haven’t had to worry about performance, stability, or scalability. With either technology, you get scalability for free and with Fargate you don’t have to worry about cold starts.

I develop on Windows with Visual Studio and deploy to “Linux”. We use CodeBuild to compile and create the zip file (lambda) or Docker Container (Fargate).

CodeBuild basically is “Serverless builds”. It launches s prebuilt or custom Docker Container and you just run .net core “publish” command line and it gets all of your Nuget packages and then you run your standard packaging commands.

But Rider should be good. I’m a huge fan of R# and you can cross target Linux, Mac and Windows from either host using msbuild.

Re:Ask HN: What is the ops architecture like for AAA multipl... by hermitdev on Monday September 09, 2019 @01:11AM (#20915355)
Curious, how've your cross platform .Net Core experiences been? I've not done much with Linux, but when I was benchmarking writing to MS SQL with the same binary on Linux & Win10, .Net Core had about half the throughput to a remote SQL server instance than Win10. Not sure if others have seen this, but .Net Core seems to run significantly slower on linux vs Windows, in my experience.
Re:Ask HN: What is the ops architecture like for AAA multipl... by scarface74 on Monday September 09, 2019 @07:38AM (#20917177)
Re:Ask HN: What is the ops architecture like for AAA multipl... by james-mcelwain on Sunday September 08, 2019 @10:55AM (#20910904)
Is C# more performant than Java? It was my understanding that HotSpot's JIT performs significantly more optimizations, almost bordering on esoteric magic, compared to the CLR.

I can understand how value types would help a lot, but it is possible to optimize Java to avoid object headers on the heap at great pain, i.e. avoiding classes and doing index arithmetic on flat arrays of values. Value types coming to the JVM will obviously help here.

Curious to hear about other advantages the CLR has.

Re:Ask HN: What is the ops architecture like for AAA multipl... by Havoc on Sunday September 08, 2019 @04:44PM (#20913305)
As a complete noob yet somehow armed with a VS enterprise license...could you elaborate on

>* Faster iterations. (just hit F5 in visual studio)

Is this some sort of code sync to servers/cloud?

Re:Ask HN: What is the ops architecture like for AAA multipl... by the8472 on Sunday September 08, 2019 @07:06AM (#20909777)
Does io_uring change the equation a little?
Re:Ask HN: What is the ops architecture like for AAA multipl... by dijit on Sunday September 08, 2019 @10:10AM (#20910634)
Yes, but I can't talk about future developments.

sufficed to say: my job is getting easier in future (thanks mostly to Stadia)

Re:Ask HN: What is the ops architecture like for AAA multipl... by vvanders on Sunday September 08, 2019 @10:23AM (#20910714)
Kinda surprised to see some one say a positive thing about Stadia. I did a fair amount of network programming back when I was in the industry and I just don't see how it can work.

I see enough congestion on just the local 2.4Ghz spectrum out here it doesn't really matter how fast the data center is. Without some form of dead reckoning you don't have much room for error.

Re:Ask HN: What is the ops architecture like for AAA multipl... by dijit on Sunday September 08, 2019 @11:38AM (#20911152)
Oh, certainly. It's a serious undertaking for everyone, I'm skeptical to how it can work when we're often measuring screen latency and input latency in nanoseconds. (and obviously not getting sub-millisecond latencies on those significantly faster/less congested subsystems).

However, Stadia is Linux, so if you want to be on stadia, you need a linux version of the game, which means primitives need to be ported. Once primitives are ported for the client, they can be optimised for the server, then you have a linux gameserver. Which is good for me.

Re:Ask HN: What is the ops architecture like for AAA multipl... by godelmachine on Sunday September 08, 2019 @05:51AM (#20909532)
>>(until we went to cloud, where Microsoft charges insane amounts for licensing)

Did you move to Azure?

Re:Ask HN: What is the ops architecture like for AAA multipl... by dijit on Sunday September 08, 2019 @06:12AM (#20909588)
No, it's cheaper on Azure (and they give us some huge discounts because we're so MS heavy).

But they really gouge the other cloud providers (GCP/AWS) on licensing, more than a third of our infra costs are just windows licenses (when looking only at cloud).

Re:Ask HN: What is the ops architecture like for AAA multipl... by kamyarg on Sunday September 08, 2019 @11:04AM (#20910958)
> more than a third of our infra costs are just windows licenses

Wow, that is brutal.

Re:Ask HN: What is the ops architecture like for AAA multipl... by dom96 on Sunday September 08, 2019 @05:17AM (#20909437)
What I'd be curious to hear about is how your servers handle simulating the game world. How many players are simulated per world and what are some of the tricks that you use to ensure the simulation doesn't fall behind, or if it does how do you make sure the clients don't notice?

I'm currently writing a (much much simpler agar.io-like) multiplayer game and finding it quite challenging to make the experience smooth, I haven't implemented any lag compensation yet though so maybe that's why but I wonder if that is the only problem. Do you or anyone else have any resources that could help me?

Re:Ask HN: What is the ops architecture like for AAA multipl... by exlurker on Sunday September 08, 2019 @05:23AM (#20909448)
Re:Ask HN: What is the ops architecture like for AAA multipl... by dom96 on Sunday September 08, 2019 @01:06PM (#20911749)
yeah, this is where I got the knowledge about lag compensation from :)
Re:Ask HN: What is the ops architecture like for AAA multipl... by druerridge on Sunday September 08, 2019 @06:01PM (#20913720)
Speaking to language usage (just as one topic) for Online Services for Games. I now contract in the games services space (after half a dozen years at a AAA studio) and have gotten to see a bit of an overview of how folks approach it. For small and medium studios, I see a lot of Python/Django approach to the problem which is quick to develop in and has a lot of problems solved for you. I also think Python is a frequent choice because a lot of the engineers are pulled from gameplay engineering, and python is familiar to a lot of folks from scripting their tools or plugins (e.g. Maya). Node/express pokes its head out for some small studios as well for similar reasons to Django. Go is increasingly popular for both AAA and indie companies for pretty obvious reasons (online services is literally what it was built for). C++ and then Java/Scala are common choices for older and more established studios.

In my experience, all of these languages are fairly well equipped to the problem space - even the interpreted/garbage collected ones can handle these problems well as they are (in my experience) infrequently CPU bound, and frequently the scaling considerations come in around your usage of databases and caches (network or IO bound). When you have enough players to hit CPU boundaries, you also typically have enough money to buy some pretty beefy CPUs. The caveat to this is that concurrency gives huge wins, since a lot of the requests/data are not related to each other (beyond the 10-20 people in a game) and so languages or frameworks that don't solve concurrency and/or async IO well are at a disadvantage. This puts Python/Django and Node.js further down the totem for me. Of course, there are ways to resolve those problems even within those languages so they are far from ill-equipped.

When I get my choice, I choose something garbage collected in a VM, with mature web frameworks (basically Java & C#) for online services in games. Knowing an engineer's errant null pointer isn't going to tip over your whole process is pretty handy, and the CPU gains in C++ or Go (native code) aren't put to as good of use here as they are in gameplay engineering where you're trying to squeeze cycles out of client machines. Client machines don't really get more powerful the more players you have, but your servers can, haha.

Re:Ask HN: What is the ops architecture like for AAA multipl... by mninm on Sunday September 08, 2019 @06:47AM (#20909710)
What's your take on using AWS Flexmatch & GameLift for matchmaking and autoscaling? Do you see any trouble spots in their approach?
Re:Ask HN: What is the ops architecture like for AAA multipl... by bpicolo on Sunday September 08, 2019 @08:36AM (#20910178)
If they can do all of their matchmaking on a single box, doesn't seem like it's a big pain point.
Re:Ask HN: What is the ops architecture like for AAA multipl... by yomly on Sunday September 08, 2019 @05:06AM (#20909391)
What is AD? Sorry - hard to google...
Re:Ask HN: What is the ops architecture like for AAA multipl... by lubos on Sunday September 08, 2019 @05:09AM (#20909403)
Active Directory