News for hackers. Stuff that matters.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by zemo on Friday February 09, 2018 @01:02PM (#16342374)
I run game servers. Our intra-day swing gives us about 10x traffic during primetime hours compared to off hours. Autoscaling sheds a lot of overhead.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by evanosaurus on Friday February 09, 2018 @02:10PM (#16342990)
Hi. Guy who does the autoscaling code here at Epic. It's really fun to watch the graphs of how many server sessions we have in a region at any given time over the course of a day.
Re:Postmortem of Service Outage at 3.4M Concurrent Users by matt_s on Friday February 09, 2018 @03:08PM (#16343434)
I like that you're in here commenting. After reading through the postmortem, it reminds me of scaling issues we had at previous job. We had hundreds of thousands of clients that would get "hyper active" if they had issues connecting, retry loops FTW.

System goes down and it was hard resurrecting it since the traffic just kept pounding away. No autoscaling, no cloud. It woulda been handy to just fire up some more servers, let alone have things auto-scale via CPU %.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by pathseeker on Friday February 09, 2018 @01:53PM (#16342877)
Have you run the calculations to determine where that cutoff is? If you have a daily primetime of 10x traffic that's pretty consistently the same amount, then it's still cheaper (if we're talking the scale of hundreds of instances) to rack up the servers in a datacenter (COLO or your own) with the capacity for the standard 10x load.

Auto-scaling instances on EC2 only makes sense if you don't have the in-house skills to troubleshoot/manage hardware servers or if the traffic is unpredictable and the bursts are infrequent (e.g. once or twice a week or just busy for a short time period of the year).

Re:Postmortem of Service Outage at 3.4M Concurrent Users by spookthesunset on Friday February 09, 2018 @05:36PM (#16344454)
> in-house skills to troubleshoot/manage hardware servers

Which costs time, money and opportunity cost. Something that all the "rack and stack" people seem to forget when they compare owning bare-metal servers against managed cloud providers.

The amount of time these "bare-metal" shops waste building and maintaining infrastructure management tools, scaling tools, reporting and monitoring tools is mind boggling. None of this stuff has anything to do with the core business the company actually is in. Opportunity costs here are immense....

Re:Postmortem of Service Outage at 3.4M Concurrent Users by Thaxll on Friday February 09, 2018 @02:07PM (#16342974)
You don't use physical servers in a DC to pay the 10x peak time, it's a big waste of money, if you want to go this way you pay the minimum numbers of servers at any time and you use cloud for the rest. But again I think noways it doesn't make any sense to buy servers in DC.

You pay for hardware that you need to mange and have to be sure that you will be using it after the game launch. ( most games player base decrease by 2x just after release. )

Re:Postmortem of Service Outage at 3.4M Concurrent Users by twic on Friday February 09, 2018 @04:57PM (#16344197)
> You don't use physical servers in a DC to pay the 10x peak time, it's a big waste of money

No, it's cheaper once you factor in the real performance of physical vs virtual machines.

Someone really needs to bring some actual numbers to this conversation.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by Thaxll on Saturday February 10, 2018 @09:31AM (#16347407)
Numbers were done it's cheaper to run in the cloud nowdays, also VM ( EC2 on AWS ) have the same perf that baremetal. Have a read: https://aws.amazon.com/ec2/instance-types/c5/
Re:Postmortem of Service Outage at 3.4M Concurrent Users by twic on Saturday February 10, 2018 @09:51AM (#16347505)
> VM ( EC2 on AWS ) have the same perf that baremetal

Extraordinary claims require extraordinary evidence.

Re:Postmortem of Service Outage at 3.4M Concurrent Users by cthalupa on Sunday February 11, 2018 @03:21PM (#16354567)
http://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtuali...

Brendan Gregg has written here and elsewhere about the performance of VMs on EC2. Worth reading if you're interested in figuring out how close it is (or isn't).

Re:Postmortem of Service Outage at 3.4M Concurrent Users by ApolloFortyNine on Saturday February 10, 2018 @02:20PM (#16348988)
AWS makes $20 billion a year in revenue.

Maybe it's possible that you're wrong?

Re:Postmortem of Service Outage at 3.4M Concurrent Users by twic on Saturday February 10, 2018 @07:48PM (#16350506)
That is a complete non sequitur. AWS has traditionally never been the cheapest way to rent machine power, but people don't use it because it's cheap, they use it because it's easy to get started with a wide range of facilities without dedicated sysadmins.