Re:Compute Engine Predictive Autoscaling
by dijit on Saturday July 03, 2021 @10:03AM
(#27721680)
Back when I was working at Massive Entertainment (part of Ubisoft) we actually implemented a predictive autoscaler on top of SaltStacks REST API -- in C++ of course for The Division 2[0]
This would have been quite handy back then, it seems to operate on almost exactly the same principles.
The hardest parts were the fact that it was a 10-20 minute endevour to create a new gameserver instance, and predictions get more difficult the further into the future they are. We didn't use any fancy AI stuff, just linear regression with smoothing across the history. We preferred to overshoot, but the predictions had a tendency to spike sometimes with relatively normal trends. You don't want to start creating a machine with a 20 minute lead time and abort it. It's a wasted hour of compute time. (and an hour is the minimum windows licensing cost).
Re:Compute Engine Predictive Autoscaling
by ksec on Saturday July 03, 2021 @11:47AM
(#27722397)
Sorry if this is a dumb question.
>a new gameserver instance.........Windows licensing was 50% of our costs on GCP.
Are there any reason why the gameServer part is on Windows? And if this is a norm in Games industry. I always assumed the Server part would be a Linux stack.
but tl;dr: When you tell developers that they can't run VMs and new hardware takes 6-12 months and a lot of bureaucracy to acquire, you make it work on your machine.
And, of course, Gamedevs understand win32 API very well.
And, making high performance networked AsyncIO is a bit nicer on Windows (IOCP vs EPOLL).
Re:Compute Engine Predictive Autoscaling
by jiggawatts on Saturday July 03, 2021 @06:24AM
(#27720608)
What does it say about AWS branding that even after googling the term and reading the top three hits, I can't tell if "AWS infinidash" is a joke or a real product?
Re:Compute Engine Predictive Autoscaling
by davidjytang on Saturday July 03, 2021 @02:49AM
(#27719853)
My experience with GCE's autoscaling has been that the up-scaling is much more aggressive than anticipated. I see with this predictive autoscaling, up-scaling would be even more aggressive.
Re:Compute Engine Predictive Autoscaling
by otterley on Saturday July 03, 2021 @03:22PM
(#27724057)
At least with AWS, auto scaling is biased towards aggressive scale-outs and conservative scale-ins, under the general principle that customers would prefer to err on the side of providing service and meeting their SLAs over conserving every last dollar of spend.
It makes sense that GCE would adopt the same attitude.
Re:Compute Engine Predictive Autoscaling
by lofties on Saturday July 03, 2021 @10:26AM
(#27721822)
And the downscaling in my experience takes too long. Sometimes it feels it's done this way on purpose to rake in a few extra hours of compute pennies. It starts adding up... imagine if you have a million customers and you can suddenly charge them an extra few dollars per month.
Re:Compute Engine Predictive Autoscaling
by klenwell on Saturday July 03, 2021 @10:55AM
(#27721989)
I have an app that has been running on the old Python 2.7 App Engine platform for a few years. It gets very light traffic and usually costs $1 or $2 a month, mainly for datastore read ops.
Suddenly this month there was an additional $11.65 charge for "frontend instances". I've searched the dashboard, documentation, and my configuration files trying to figure out what this means. I still have no idea where it came from or how to stop it, short of turning off the app.
Re:Compute Engine Predictive Autoscaling
by ahmedtd on Saturday July 03, 2021 @05:04PM
(#27724905)
If you search for "frontend" on the App Engine Standard Pricing Page [1], it seems like your usage might be exceeding the free tier for frontend instances. Did the load on your app suddenly increase?
This would have been quite handy back then, it seems to operate on almost exactly the same principles.
The hardest parts were the fact that it was a 10-20 minute endevour to create a new gameserver instance, and predictions get more difficult the further into the future they are. We didn't use any fancy AI stuff, just linear regression with smoothing across the history. We preferred to overshoot, but the predictions had a tendency to spike sometimes with relatively normal trends. You don't want to start creating a machine with a 20 minute lead time and abort it. It's a wasted hour of compute time. (and an hour is the minimum windows licensing cost).
Windows licensing was 50% of our costs on GCP.
[0]: https://cloud.google.com/customers/massive-entertainment
>a new gameserver instance.........Windows licensing was 50% of our costs on GCP.
Are there any reason why the gameServer part is on Windows? And if this is a norm in Games industry. I always assumed the Server part would be a Linux stack.
I answered this a bit more in depth here: https://www.gcppodcast.com/post/episode-265-sharkmob-games-w...
but tl;dr: When you tell developers that they can't run VMs and new hardware takes 6-12 months and a lot of bureaucracy to acquire, you make it work on your machine.
And, of course, Gamedevs understand win32 API very well.
And, making high performance networked AsyncIO is a bit nicer on Windows (IOCP vs EPOLL).
https://twitter.com/jna_sh/status/1410178986978775040?s=20
Did produce some quality comedy material!
=> https://twitter.com/TartanLlama/status/1410959645238308866?s...
https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-au...
It makes sense that GCE would adopt the same attitude.
Suddenly this month there was an additional $11.65 charge for "frontend instances". I've searched the dashboard, documentation, and my configuration files trying to figure out what this means. I still have no idea where it came from or how to stop it, short of turning off the app.
[1] https://cloud.google.com/appengine/pricing
I think I figured out how to stop it from doing this: https://old.reddit.com/r/AppEngine/comments/od11sg
Thanks for the link.