News for hackers. Stuff that matters.
Re:We use Kubernetes and spot instances to reduce EC2 billin... by juliansimioni on Monday February 24, 2020 @10:46PM (#22410763)
On the one hand, building fault tolerant infrastructure that can, as a side effect, work painlessly on spot instances is great.

On the other hand, you can purchase reserved instances and get ~60% cost savings with zero engineering work. Its worth thinking long and hard about whether the cost of engineering time is worth that next 20%.

There's also a lot of useful ground in between "critical state, must never be lost (like a database)" and "can handle being terminated with 2 minutes notice". A service that can be re-created if necessary but takes 10 minutes to start up is really scary if run on spot instances, but can still be pretty useful.

Re:We use Kubernetes and spot instances to reduce EC2 billin... by jrockway on Tuesday February 25, 2020 @12:23AM (#22411114)
Back when I used AWS, I went the reserved instances route. The pricing is pretty okay, and you are guaranteed to have machines at busy periods of the day.

The problem with spot autoscaling is that if everyone does it, it stops working. Everyone in us-east-1 gets most of their traffic from 9am EST to 5pm EST, because everyone hosts close to their users, their users are human, and humans are diurnal. Most "batch" workloads that people have also follow the same cycle; they're working on something in the office and want results now, not tomorrow morning. So they run their batch jobs during the day. If you can figure out how to get your traffic spikes from 9pm until 5am, spot instances are going to be great -- millions of CPUs are sitting idle waiting for your novel workload. But if your customers are working 9-5 jobs like you, and you care about latency enough to host close to them... you're competing for instances with every other computer user in the region.

Re:We use Kubernetes and spot instances to reduce EC2 billin... by lukasLansky on Tuesday February 25, 2020 @02:46AM (#22411533)
Most of batch workloads are not that latency sensitive. You can send them to the other side of the world as everyone is sleeping there right now.
Re:We use Kubernetes and spot instances to reduce EC2 billin... by satanspastaroll on Tuesday February 25, 2020 @03:30AM (#22411675)
A thing to keep in mind is AWS's major outobound traffic costs, at $90/TB. So taxiing the data oversea might not be affordable either
Re:We use Kubernetes and spot instances to reduce EC2 billin... by oblio on Tuesday February 25, 2020 @05:54AM (#22412218)
Even for inter-AWS region transfers?
Re:We use Kubernetes and spot instances to reduce EC2 billin... by mi100hael on Tuesday February 25, 2020 @12:04PM (#22414852)
It depends on how the data is being transferred. In general, the answer is yes because you'd be using public IPs.

If you're transferring between EC2 instances in peered VPCs meaning you can still use private IPs, it's more like $20/TB.

https://datapath.io/resources/blog/what-are-aws-data-transfe...

Re:We use Kubernetes and spot instances to reduce EC2 billin... by jrockway on Tuesday February 25, 2020 @04:47AM (#22411975)
That's not been my experience, actually. The first CI service I used was hosted entirely in Europe, and we needed to ssh in to debug something, the keystroke latency was maddening. We eventually unsubscribed and just bought a big EC2 instance in us-east with Jenkins running on it. It cost approximately 100x more, but our productivity was high and frustration low. Well worth.

I personally think it will breed huge organizational problems if things like CI are slow. "I'll get a cup of coffee while this runs" and then you come back and forget what you were going to release. Soon it becomes "let's get another change into this build before we release" and then it's "well, it's been six months since we've released anything, what do we do." You have to start fast and stay fast if you want to keep developers productive. So saving a couple bucks on computers that are half a world away can end up being a huge expense if you're not careful.

As other comments mention, you also have to be careful about transfer costs. In the CI case, getting your source code into the CI server is cheap, but getting the containers out is going to cost you, especially if you don't make an effort to optimize them. For batch data processing jobs, the same applies; getting the result out is cheap, but getting the data in is going to be a lot of transfer. (If you were using Small Data, you could just run the job on your laptop, after all.)

The speed of computers half a world away is not great either. I remember updating some Samsung drivers once, which were served out of a Korean AWS region instead of CloudFront... and the downloads were glacially slow. Their website is the same way. I couldn't believe how a multinational corporation could push bits at me so slowly. When you're reading their documentation all day, or tweaking drivers, you notice it, and you start to think "next time I'm going to buy Intel". (Compare Samsung's SSD website with McMaster-Carr's website. What site do you hope to interact with again in the future?)

Anyway, you get a bill for compute resources, and you don't get a bill for unhappy employees context-switching all day, so I see why people want to craft clever schemes to save pennies on their compute costs. But be careful. Not every cost is charged directly to your credit card.

Re:We use Kubernetes and spot instances to reduce EC2 billin... by heartbeats on Tuesday February 25, 2020 @08:42AM (#22413082)
What about using something like mosh for latency?
Re:We use Kubernetes and spot instances to reduce EC2 billin... by NikolaeVarius on Monday February 24, 2020 @11:03PM (#22410820)
Note, AWS savings plans make this even easier

https://aws.amazon.com/savingsplans/

Re:We use Kubernetes and spot instances to reduce EC2 billin... by ckdarby on Monday February 24, 2020 @11:07PM (#22410830)
^ Came here to say the above. Majority of the time the Saving Plan is actually what you want.

Unless you have legacy app you know must be around for 3 years and have zero efforts to try to refactor.

Re:We use Kubernetes and spot instances to reduce EC2 billin... by 40acres on Monday February 24, 2020 @11:29PM (#22410912)
Any idea of when this product was introduced?
Re:We use Kubernetes and spot instances to reduce EC2 billin... by NikolaeVarius on Monday February 24, 2020 @11:32PM (#22410928)
Re:We use Kubernetes and spot instances to reduce EC2 billin... by 40acres on Monday February 24, 2020 @11:50PM (#22410985)
So is this blog post redundant?
Re:We use Kubernetes and spot instances to reduce EC2 billin... by QuinnyPig on Tuesday February 25, 2020 @12:42AM (#22411176)
No. Spot gives deeper discounts than savings plans do.
Re:We use Kubernetes and spot instances to reduce EC2 billin... by chillydawg on Tuesday February 25, 2020 @04:33AM (#22411904)
And you can stack them.
Re:We use Kubernetes and spot instances to reduce EC2 billin... by halbritt on Monday February 24, 2020 @11:05PM (#22410824)
I did something like this in GKE with preemptible instances which are guaranteed to go away at least once every 24 hours. I had separate node groups for stateless and stateful work loads. My clusters were roughly a 50/50 mix of each. Worked out pretty well and yielded some decent cost savings.
Re:We use Kubernetes and spot instances to reduce EC2 billin... by tuananh on Monday February 24, 2020 @11:52PM (#22411001)
preemptible instances are worse than spot. we observed there were spot instances that last a year for us.
Re:We use Kubernetes and spot instances to reduce EC2 billin... by halbritt on Tuesday February 25, 2020 @12:00AM (#22411024)
That presumes that an instance type that last longer is "better".
Re:We use Kubernetes and spot instances to reduce EC2 billin... by ody4242 on Tuesday February 25, 2020 @03:01AM (#22411577)
Yes, preemptible instances are expected to be expendable, I don't see this as a bad thing.
Re:We use Kubernetes and spot instances to reduce EC2 billin... by tuananh on Tuesday February 25, 2020 @03:43AM (#22411718)
i believe what he meant is by being shutdown more frequent, we will be forced to build a more resilent system. otherwise, we would be more complacent.
Re:We use Kubernetes and spot instances to reduce EC2 billin... by rawoke083600 on Tuesday February 25, 2020 @12:12AM (#22411072)
Yea were wondering the same thing... How much does this cost in engineering hours ? Its like asking someone who smokes... they always forget/lie about at least 50% of the cigarettes... And even if the engineering hours was super minimal compared to reserved instance... will this be true for other teams ?
Re:We use Kubernetes and spot instances to reduce EC2 billin... by tuananh on Tuesday February 25, 2020 @12:40AM (#22411171)
i hear all these engineering hours cost but only because engineering cost in US is so high. we're in Southeast Asia where senior engineer only cost $1000 a month back then in 2016.
Re:We use Kubernetes and spot instances to reduce EC2 billin... by satanspastaroll on Tuesday February 25, 2020 @03:31AM (#22411682)
Isn't cost of living also much lower in Southeast Asia?
Re:We use Kubernetes and spot instances to reduce EC2 billin... by mjlee on Tuesday February 25, 2020 @05:19AM (#22412102)
Sure, but AWS infrastructure costs pretty much the same no matter where you live.
Re:We use Kubernetes and spot instances to reduce EC2 billin... by tuananh on Tuesday February 25, 2020 @04:05AM (#22411790)
i believe he was talking how it cost more in engineering cost versus the saving we can get from infrastructure by migrating.

eg: if the infra cost down by 10,000 USD per month, they may say it's not worth it because they are paying more for 1 developer in US in a month.

Re:We use Kubernetes and spot instances to reduce EC2 billin... by ec109685 on Tuesday February 25, 2020 @12:48AM (#22411197)
Reserved instances and savings plans are predicated on predicting usage for 1 to 3 years in advance.

With spot instances, you don’t need to do this planning, assuming you can fall back to reserved quickly enough.

Re:We use Kubernetes and spot instances to reduce EC2 billin... by johnmarcus on Tuesday February 25, 2020 @12:06AM (#22411049)
this is why i love google cloud over aws these days - i can get that cost saving with doing none of the work nor making any of the commitment. It's usually that commitment business hate, imho.
Re:We use Kubernetes and spot instances to reduce EC2 billin... by tuananh on Monday February 24, 2020 @11:51PM (#22410989)
why not both :)

we did mix reserved instances and spot instances for our production workload. worst case scenario will be reserved + on-demand