News for hackers. Stuff that matters.
Re:Ask HN: What is your Kubernetes nightmare? by garciasn on Monday June 27, 2022 @09:12AM (#31894104)
We use GKE.

1. I don't know about any of this; we don't seem to have problems.

2. This sounds like an architecture issue, not a k8s issue.

3. Our entire GKE infrastructure costs less than $50 a month.

4. You're right here; it doesn't work 'well', but it works 'well enough' for our use cases.

5. I'm sure you're talking about some event that was far more complex than the few times we've had to drain our pool, but we did what we needed to do without downtime in production. While annoyingly esoteric, I thought it worked pretty fucking well compared to our alternatives.

Re:Ask HN: What is your Kubernetes nightmare? by ColdHeat on Monday June 27, 2022 @09:56AM (#31894648)
With regards to 3, isn’t the management plane alone $72 for GKE without considering the cost for the nodes? How are your costs so low?
Re:Ask HN: What is your Kubernetes nightmare? by seized on Monday June 27, 2022 @10:15AM (#31894873)
Re:Ask HN: What is your Kubernetes nightmare? by garciasn on Monday June 27, 2022 @10:55AM (#31895431)
As noted by another HNer, we are using one cluster and up to 10 nodes as the top end, with only one running 90% of the time and only up to 3 covering the other 9%. We set 10 to the upper limit to ensure we're not going to have borked runs in case some crazy random ML model isn't going to take everything out and force killed jobs. The vast majority of workloads running on the cluster are HIGHLY variable with most running on Monday morning/afternoon and Month or Quarter starts + 3 days.

We are running many hundreds of jobs on those peak days with only a handful running on any other day. While many bring up examples where 24/7 infrastructure from a single box is more than plenty, we find that we can run micro VMs in this configuration and not have to worry about resource contention as our jobs run.

Pre-GKE, we were managing the timing manually, which was fine until we started to scale, but we found this to be a far better situation. Particularly because we simply don't have to think about it.

YMMV.

Re:Ask HN: What is your Kubernetes nightmare? by otabdeveloper4 on Tuesday June 28, 2022 @10:51AM (#31908453)
> 3. Our entire GKE infrastructure costs less than $50 a month.

At this scale, you don't need Kubernetes, invest in a pocket calculator instead.

Re:Ask HN: What is your Kubernetes nightmare? by throwaway894345 on Monday June 27, 2022 @11:43AM (#31896151)
Agreed. We've had only one autoscaling issue and it was on Google's end (datacenter ran out of nodes of a particular type and thus failed to scale up). Our GKE infrastructure costs a lot more, but we do a lot of heavy compute.
Re:Ask HN: What is your Kubernetes nightmare? by garciasn on Monday June 27, 2022 @01:43PM (#31898016)
Most of our 'heavy compute' happens in BigQuery so we pay for it there, instead, but we make sure the rest of our analytics infrastructure needs are engineered to be as light as possible to keep costs low.

We are an ESOP so, as employee owners, it behooves us to be as cost-conscious as possible.

Re:Ask HN: What is your Kubernetes nightmare? by JamesSwift on Monday June 27, 2022 @10:47AM (#31895299)
> Our entire GKE infrastructure costs less than $50 a month.

Uhhh what? I mean even my personal DO based cluster runs about $40 a month. I'm skeptical a production cluster is at $50.

Re:Ask HN: What is your Kubernetes nightmare? by garciasn on Monday June 27, 2022 @11:03AM (#31895547)

    Service                   Compute Engine Kubernetes Engine
    Cost.                     $191.78        $62.34 
    Discounts                ($13.96)       ($62.34)
    Promotions and others     $0.00          $0.00 
    Subtotal                  $177.83        $0.00
The GCE instance cost includes some of our 24/7 VMs which are the lion's share of that line item, not the micro VMs we use for the cluster.