News for hackers. Stuff that matters.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by matt_s on Friday February 09, 2018 @11:12AM (#16341319)
A lot of players of massive online games tend to get hand-wavey when there are problems and act like "dude just get more servers" is the answer.

This clearly shows how complex a system is needed that has to handle 3.4 million concurrent, connected users. I think the connected part compounds any scale problems you have since it is implied they are connected to each other.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by rightos on Friday February 09, 2018 @12:36PM (#16342134)
> act like "dude just get more servers" is the answer.

The biggest problem here is that it largely used to be a distributed system. You used to just be able run your own dedicated server on whatever provider you liked. The dev would just run a single server list. Now many game developers have decided they're the only ones who get to run servers - primarily because they can charge more for micro transactions and private servers this way as far as I can tell.

It really hurts games - PUBG is the best example I've seen - constant lag issues, complete lack of server side checks for things like shooting through the ground (because hey, that costs CPU and every additional cloud server they need means less profit), etc. It's basically made the game unplayable.

Game developers are unfortunately stuck between immersion in their games and the rage that leaves players with when technical issues occur. The more immersed your players are, the more rage they'll experience when your game crashes or lags at the wrong time.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by chipperyman573 on Friday February 09, 2018 @01:39PM (#16342729)
>primarily because they can charge more for micro transactions and private servers this way as far as I can tell.

It also hurts the game's image if many servers spam players with "Donate $20/mo for the MASTER rank and get exclusive weapons that deal twice the damage of normal weapons!"

I haven't played in a while but I remember playing Minecraft when I was still in school and servers were filled with this- many would let you donate hundreds for admin/OP/whatever, many would restrict core parts of the game to "donating" members (such as mining certain ores, going into certain areas, etc). Eventually it got so bad that Mojang actually had to step in and (vaguely) threaten to sue any server owner who does this - http://andrewtian.com/mojang-threatens-lawyers-against-pay-t...

Also, admins that can ban players for any reason, who are picked by the server owners (and not the devs), can result in backlash if someone influential is banned.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by pathseeker on Friday February 09, 2018 @01:45PM (#16342789)
>It also hurts the game's image if many servers spam players with "Donate $20/mo for the INVINCIBILITY rank!"

That's a problem with the client design. Don't allow the server to send banners or any map tiles and that problem goes away.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by Flammy on Friday February 09, 2018 @01:52PM (#16342861)
That isn't really a realistic solution if you also want to support the mod ecosystem (which IMO is much more valuable to a game's community than the potential downsides)
Re:Postmortem of Service Outage at 3.4M Concurrent Users by pathseeker on Friday February 09, 2018 @01:55PM (#16342892)
Ok, but the mod ecosystem is a separate discussion. And even if you do want the mod ecosystem, you can distribute mods as client add-ons so the mods can be evaluated independently of the host server.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by chipperyman573 on Friday February 09, 2018 @02:03PM (#16342943)
Which is exactly what Gmod did, and they have this problem.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by robotresearcher on Friday February 09, 2018 @02:05PM (#16342961)
Just make mods opt-in. Vanilla servers by default.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by zbentley on Friday February 09, 2018 @10:41PM (#16345737)
Dedicated game servers are only part of the problem. The matchmaking/community-connecting servers were also implicated in the outages discussed in TFA.

Community/matchmaking servers don't distribute ("run your own server") too well. If you only ever want to play with and against people you know, that's fine, but a lot of people don't want that: some want to play on randomized (or ranked, if the game supports it, across a large number of players; not just the ones in the potentially-altered rankings on a single person's server) teams. Others (more commonly) want to play against new and different teams.

Unless your graph of social connections willing to agree to use a given server is big enough and the right shape, hosting your own matchmaking/community services just results in isolated islands without a vibrant overarching community.

Put another way: the separate-server approach of e.g. TF2 works fine for some, but a lot (judging by play counts) of other folks prefer games with a centralized community.

To be clear, I think it's fine to have the option of a separate community server. But if a game developer already has the centralized infrastructure, is making tons of money off of it, and spending tremendous resources keeping it intact (as Epic apparently is), it's a pretty tough sell to say "oh and you should also spend the resources to make a hosted option for this".

"Take the complex thing designed for centralized, single-instance internal use with a team of support staff, put it in a box, and let users run it on their own hosts" is far from a simple proposition.

I'm unsure if any companies doing centralized community servers/community federation also have a large portion of their actual game servers not hosted by the company; I'd love some examples of that phenomenon. The examples of Minecraft and TF2 come to mind as doing some subparts of that, but those games only have limited matchmaking capabilities.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by ankjl on Friday February 09, 2018 @03:33PM (#16343603)
I think that part of this trend may be tied to games, such as PUBG, becoming popular while still "under development". I imagine once a game has a stable server build it is much more practical to distribute that server to all of the community operators and have them update on a fixed / infrequent schedule. Whereas, while changes are still being constantly made to the server application, this would probably result in users only being able to connect to a small fraction of available community servers while waiting for the other servers to update.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by TheCapn on Friday February 09, 2018 @01:24PM (#16342582)
>primarily because they can charge more for micro transactions and private servers this way as far as I can tell.

I believe its more about standardizing the user experience. Now that enough games have dedicated servers with a standard expectation of latency your game looks poor if your user experience is beholden to community servers with selfish moderation and varying stability.

The added benefit of centralized server hosting and control is exactly what you point out though.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by SiempreViernes on Friday February 09, 2018 @02:46PM (#16343269)
No: standardisation is easily enforced by just having two server lists, one for normal/central hosting and one for custom games where any host can register. At most you might want additionally give the user the ability to blacklist servers for themselves, and that's it.

The reason for banning community servers is probably mostly for combating piracy; by eliminating the ability to run on a LAN the game must always call home to be played.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by always_good on Friday February 09, 2018 @11:51AM (#16341712)
Agreed. Gamers also are among the most entitled users I've ever had to deal with.

I'll never accept donations again for a game I've made available for free from the amount of problems someone has caused me after they gave me $5.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by jjjensen90 on Friday February 09, 2018 @12:16PM (#16341956)
I never realized the level of feelings of entitlement and melodrama from gamers until I ran a passion project to make a private server for a dead game. We don't even accept donations and people still treat the dev team like dirt, and will often write nasty manifestos about how we are ruining their lives or we don't know what we are doing and if only we would listen to their every whim, etc...

It's one of those things, though, that reveals the human inclination to remember negative events more even among a sea of positive events... Frequently as a game developer you will get loads of positive and helpful feedback but the small minority that want to shit on you for no good reason stick in the front of your mind.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by woolvalley on Friday February 09, 2018 @03:15PM (#16343485)
I think it comes from a combination of a good set of the potential populace being children, the content being mostly emotional and not needing to be professionally employed to use the game. Those 3 things alone are great filters in getting good users and you are lacking that in games.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by dleslie on Friday February 09, 2018 @12:34PM (#16342125)
Out of curiousity, what's the project?
Re:Postmortem of Service Outage at 3.4M Concurrent Users by jjjensen90 on Friday February 09, 2018 @12:36PM (#16342143)
London 2038 https://london2038.com/ It's a revival of the MMOARPG Hellgate: London which released in 2007, closed in 2009, was revived as a pay-to-win Korean reincarnation in 2011 and shut down again in 2015. Our version matches the version of the game from 2009.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by dleslie on Friday February 09, 2018 @12:42PM (#16342182)
Hey, I've heard of this! Good work. :D

One thing I notice from your site is that you provide _many_ ways of giving feedback. The Discord widget, the 'Contact Us' menu item, the FAQ is peppered with communication links, there's a Forum, Wiki, et al. Basically, I'm seeing an incredibly low barrier to entry for anyone wanting to express themselves to the developers.

Have you considered paring that back? IE, ensuring all communication channels are moderated, and limiting it to only one or two services?

Re:Postmortem of Service Outage at 3.4M Concurrent Users by jjjensen90 on Friday February 09, 2018 @01:37PM (#16342717)
Great feedback, I have actually done a lot to pare it down since we launched open alpha last Halloween. Typically people know now to make forum posts for long-term discussions and to come chat in Discord for quick back and forth. Nobody has actually used the contact form on the site, now that you mention it.

My team of community manager volunteers is pretty active with moderation in-game and out-of-game so it's gotten pretty good. But still, I think I need to isolate myself and the other developers from the public, because it can be really demoralizing when you spend 30-40 hours of free time per week on a side project for free and you hear some terrible things.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by dleslie on Friday February 09, 2018 @11:46PM (#16345943)
Sounds like you've got a sound plan, and I whole-heartily endorse placing a barrier and filter between you and your fellow developers and the community at large. :)
Re:Postmortem of Service Outage at 3.4M Concurrent Users by dleslie on Friday February 09, 2018 @12:19PM (#16341985)
Gamers aren't simply entitled; they can be aggressive, threatening and altogether horrible. I've personally received threats on my own life from users. There was a nice medium piece on it recently. [0]

Anecdotally, I'm seeing more and more peers leave the industry as a result of being disgusted and worn down by the constant, unending abuse that game developers receive from users.

0: https://medium.com/@morganjaffit/the-cost-of-doing-business-...

Re:Postmortem of Service Outage at 3.4M Concurrent Users by Noos on Friday February 09, 2018 @03:54PM (#16343744)
I think modern game devs create entitlement through games as service models that rely on unhealthy pve or player versus player content either real or implicit, and quickly force them into factions to compete for limited dev time.

The games as service part prevents players from keeping proper emotional distance to games by not allowing them to have closure; to look at the game, say you've beaten it, and move on. Every game needs some kind of grind or persistence beyond the actual play, through online connectivity and often actual player competition. So you kind of prime players through negative play models very often, especially when in-game purchases exist.

What's worse instead of an end to a game, devs are forced into chasing the mythical idea of balance instead. This actually sets players up into warring factions because balance is often zero sum; my warrior gets buffed at the expense of your paladin. There's no existing balance state, ever; you can look at Overwatch to see it just ending up into random changes to heroes that please and incense players in equal measure.

These things combine to make players strung out over time; their nerves are jangled through constant grinding, death (screw roguelikes!), or competition, and then you end up with developer as God who made decide to alter everything to please a particular player faction, or even no one at all but themselves. You make your players invest a hundred hours in your game, and they aren't going to be disinterested enough to take changes in perspective; you don't give them the chance to be, they have to be logged on every day doing busywork either through ranking or grinding.

I think this explains why modern gamers can be so enraged. It's more than personality; a lot of the structure of many modern games can really create this. The games make players too invested in them too easily, over too long a time.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by trevyn on Friday February 09, 2018 @12:00PM (#16341822)
Did you quickly offer/send a refund? This is going to be my strategy for upset customers, and I’m curious about people’s experiences with this approach.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by always_good on Friday February 09, 2018 @12:08PM (#16341890)
No, but it's not like they are privately emailing me with their complaints. They drum up threads in the subreddit and rally others to complain as if they are being personally wronged.

I suppose I could offer a refund, but refunding a $5 donation is just another example of how a donation box is a waste of time.

In the future I would at least consider a Patreon system where donations renew every month since server and development costs certainly do. Way too many people think they're buying an annuity against your free time with a one time donation/purchase.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by lsmarigo on Friday February 09, 2018 @02:45PM (#16343265)
Couldn't agree more, I shut down my profit generating gaming project because of death threats, not worth.