News for hackers. Stuff that matters.
Re:We're Leaving Kubernetes by bhouston on Monday November 04, 2024 @10:25AM (#42042366)
I also recently left Kubernetes. It was a huge waste of time and money. I've replaced it with just a series of services on Google Cloud Run and then using Google's Cloud Run Tasks services for longer running tasks.

The infrastructure now incredibly understandable and simple and cost effective.

Kubernetes cost us >$million in both DevOps time and actually Google Cloud costs unnecessarily, and even worse it cost us time to market. Stay off of Kubernetes as long as you can in your company, unless you are basically forced onto it. You should view it as an unnecessary evil that comes with massive downsides in terms of complexity and cost.

Re:We're Leaving Kubernetes by kbolino on Monday November 04, 2024 @11:02AM (#42042781)
As far as I can tell, there actually is no AWS equivalent to GCP Cloud Run. The closest equivalents I know of are ECS on Fargate, which is more like managed Kubernetes except without Kubernetes compatibility or modern features, or AppRunner, which is closer in concept but also sorely lacking in comparable features.
Re:We're Leaving Kubernetes by Imustaskforhelp on Monday November 04, 2024 @11:35AM (#42043180)
wow very very interesting. I think we can discuss about it on hours.

1.) What would you think of things like hetzner / linode / digitalocean (if stable work exists)

2.) What do you think of https://sst.dev/ or https://encore.dev/ ? (They support rather easier migration)

3.) Could you please indicate the split of that 1 million $ in devops time and google cloud costs unnecessarily & were there some outliers (like oh our intern didn't add this specific variable and this misconfigured cloud and wasted 10k on gcloud oops! or was it , that bandwidth causes this much more in gcloud (I don't think latter to be the case though))

Looking forward to chatting with you!

Re:We're Leaving Kubernetes by elcomet on Monday November 04, 2024 @10:26AM (#42042380)
Aren't you afraid of being now stuck with GCP?
Re:We're Leaving Kubernetes by bhouston on Monday November 04, 2024 @10:28AM (#42042411)
It is just a bunch of docker containers. Some run in tasks and some run as auto-scaling services. Would probably take a week to switch to AWS as there are equivalent managed services there.

But this is really a spurious concern. I myself used to care about it years ago. But in practice, rarely do people switch between cloud providers because the incremental benefits are minor, they are nearly equivalent, there is nothing much to be gained by moving from one to the other unless politics are involved (e.g. someone high up wants a specific provider.)

Re:We're Leaving Kubernetes by spwa4 on Monday November 04, 2024 @10:45AM (#42042594)
How does the orchestration work? How do you share storage? How do the docker containers know how to find each other? How does security work?

I feel like Kubernetes' downfall, for me, is the number of "enterprise" features it (got convinced into) supporting and enterprise features doing what they do best: turning the simplest of operations into a disaster.

Re:We're Leaving Kubernetes by bhouston on Monday November 04, 2024 @10:59AM (#42042738)
> How does the orchestration work?

Github Actions CI. Take this and make a few more dependencies and a matrix strategy and you are good to go: https://github.com/bhouston/template-typescript-monorepo/blo... For dev environments, you can add post-fixes to the services based on branches.

> How do you share storage?

I use managed DBs and Cloud Storage for shared storage. I think that provisioning your own SSDs/HDs to the cloud is indicative of an anti-pattern in your architecture.

> How do the docker containers know how to find each other?

I try to avoid too much communication between services directly, rather try to go through pub-sub or similar. But you can set up each service with a domain name and access them that way. With https://web3dsurvey.com, I have an api on https://api.web3dsurvey.com and then a review environment (connected to the main branch) with https://preview.web3dsurvey.com / https://api.preview.web3dsurvey.com.

> How does security work?

You can configure Cloud Run services to be internal only and not to accept outside connections. Otherwise one can just use JWT or whatever is normal on your routes in your web server.

Re:We're Leaving Kubernetes by dthg on Monday November 04, 2024 @08:52PM (#42047952)
> But you can set up each service with a domain name and access them that way. Are you using Cloud Run domain mappings for this or something else?

I have been converging on a similar stack, but trying to avoid using a load balancer in an effort to keep fixed costs low.

Re:We're Leaving Kubernetes by bhouston on Monday November 04, 2024 @10:23PM (#42048314)
Yup domain mappings for now. There is some label support in Cloud Run but I haven’t explored it yet. You can also get the automatic domain name for a service via the cloud run tools.

Yeah I definitely want to also avoid a load balancer or gateway or end points as well for cost purposes.

Re:We're Leaving Kubernetes by sofixa on Monday November 04, 2024 @10:43AM (#42042565)
One of Cloud Run's main advantages is that it's literally just telling it how to run containers. You could run those same containers in OpenFaaS, Lambda, etc relatively easily.
Re:We're Leaving Kubernetes by rglover on Monday November 04, 2024 @10:30AM (#42042436)
What stack are you deploying?
Re:We're Leaving Kubernetes by bhouston on Monday November 04, 2024 @10:33AM (#42042459)
Stuff like this, just at larger scale:

https://github.com/bhouston/template-typescript-monorepo

This is my living template of best practices.

Re:We're Leaving Kubernetes by rglover on Monday November 04, 2024 @11:30AM (#42043113)
I'd investigate getting a build out to Node.js (looks like you already have this) and then just doing a simple SCP of the build to a VPS. From there, just use a systemd script to handle startup/restart on errors. For logging, something like the Winston package does the trick.

If you want some guidance, shoot me an email (in profile). You can run most stuff for peanuts.

Re:We're Leaving Kubernetes by bhouston on Monday November 04, 2024 @11:41AM (#42043272)
> I'd investigate getting a build out to Node.js (looks like you already have this) and then just doing a simple SCP of the build to a VPS. From there, just use a systemd script to handle startup/restart on errors. For logging, something like the Winston package does the trick. If you want some guidance, shoot me an email (in profile). You can run most stuff for peanuts.

I appreciate the offer! But it is not as robust and it is more expensive and misses a lot of benefits.

Back in the 1990s I did FTP my website to a VPS after I graduated from Geocities.

Google Cloud charges based on CPU used. Thus my servers have no traffic, they cost less than a $1/month. If they have traffic, they are still cost effective. https://web3dsurvey.com has about 500,000 hits per month and it costs me $4/month to run both the Remix web server and the Fastify API server. Details here: https://x.com/benhouston3d/status/1840811854911668641

Also it will autoscale under load. Thus when one of my posts was briefly the top story on Hacker News last month, Google Cloud Run added more instances to my server to handle the load (because I do not run my personal site behind a CDN, it cost too much, I prefer to pay $1/month for hosting.)

Also deploying Docker containers that build on Github Actions CI in a few minutes is a great automated experience.

I do also use Google services like Cloud Storage, Firestore, BigQuery etc. And it is easier to just run it on GCP infrastructure for speed.

I also have to version various tools that get installed in the docker like Blender, Chromium, etc. This is the perfect use case for Docker.

I feel this is pretty close to optimal. Fast, cheap, scalable, automated and robust.

Re:We're Leaving Kubernetes by Imustaskforhelp on Tuesday November 05, 2024 @12:48AM (#42048909)
to be really honest , why don't you use cloudflare for blog post hosting / their storage mechanism if you really want comments hosting.

Why are you actually using google cloud for blog post hosting.

Also you said a million $ kubernetes

wait a second ,have you converted those million $ to 4$ per month

what tom foolery is this

Re:We're Leaving Kubernetes by bhouston on Tuesday November 05, 2024 @09:15AM (#42051612)
I am demonstrating my point using simple personal projects that I can easily explain.

But I am also the Founder/CTO of Threekit.com.

I hope that makes sense now.

Re:We're Leaving Kubernetes by Imustaskforhelp on Monday November 04, 2024 @11:38AM (#42043235)
there was some recent HN post which showed that they didn't even use docker but rather there was some other mechanism and it was so so simple , I really enjoyed that article
Re:We're Leaving Kubernetes by Imustaskforhelp on Monday November 04, 2024 @11:38AM (#42043226)
yeh I have same thoughts , also if possible , bun can also reduce memory usage in very very basic scenarios https://www.youtube.com/watch?v=yJmyYosyDDM

Or just https://github.com/mightymoud/sidekick or coolify or dokku or dockify , like there are million of such things , oh just remembered kamala deploy from DHH and docker swarm IIRC (though people have seemed to forget docker swarm !)

I like this idea very much !

Re:We're Leaving Kubernetes by candiddevmike on Monday November 04, 2024 @10:36AM (#42042504)
You know that Cloud Run is effectively a Kubernetes PaaS, right?
Re:We're Leaving Kubernetes by richards on Monday November 04, 2024 @10:38AM (#42042525)
Google employee here. Not the case. Cloud Run doesn't run on Kubernetes. It supports the Knative interface which is an OSS project for Kubernetes-based serverless. But Cloud Run is a fully managed service that sits directly atop Borg (https://cloud.google.com/run/docs/securing/security).
Re:We're Leaving Kubernetes by dboreham on Monday November 04, 2024 @11:13PM (#42048545)
Parent said "effectively", which it appears you confirm.
Re:We're Leaving Kubernetes by bhouston on Tuesday November 05, 2024 @09:38AM (#42051755)
But it skips out on the Kubernetes part, which is the important part. :)
Re:We're Leaving Kubernetes by chanux on Monday November 04, 2024 @10:43AM (#42042566)
I guess the point is that for the OP, Kubernetes is now someone else's problem.
Re:We're Leaving Kubernetes by bhouston on Monday November 04, 2024 @10:40AM (#42042545)
> You know that Cloud Run is a Kubernetes PaaS, right?

Yup. Isn't it Knative Serving or a home grown Google alternative to it? https://knative.dev/docs/serving/

The key is I am not managing Kubernetes and I am not paying for it - it is a fool's errand, and incredibly rarely needed. Who cares what is underneath the simple Cloud Run developer UX? What matters for me is cost, simplicity, speed and understandability. You get that with Cloud Run, and you don't with Kubernetes.

Re:We're Leaving Kubernetes by davewritescode on Tuesday November 05, 2024 @08:01AM (#42051161)
I haven't looked at Cloud Run pricing but running Kubernetes in the cloud is pretty cheap these days and my experience with solutions like Cloud Run in the past is that they end up becoming expensive.

Kubernetes can be as complex or as expensive as you'd like but it's also fairly possible to run a pretty bulletproof simple Kube cluster.

Re:We're Leaving Kubernetes by bhouston on Tuesday November 05, 2024 @09:24AM (#42051673)
Maybe we misconfigured Kubernetes?

Here are my concerns:

With Kubernetes is that you need to pay for a few node just to keep it up, and then you need to pay for your nodes, no matter how much you use them.

Remember that Cloud Run charges based usage, so if a service sits unused for a while, which often happens in a heterogeneous microservices environment, you don't pay for it.

Also autoscaling is slow (Cloud Run autoscales really quickly, about as fast as your docker can be loaded and started, which for me is 1-3 seconds, where as I found Kubernetes auto-scales on the order of minutes) unless you over-provision, which is costly. This lets one scale to zero even without much of a hit.

I also ran into massive issues trying to get GPUs to work in Kubernetes - it was a driver nightmare that has wasted weeks of time collectively over the years. Whereas they are auto-provisioned properly on Cloud Run if you request them.

Lastly job systems on Kubernetes are a nightmare of configuration. The built-in scheduler cannot handle a lot of jobs but Argo also has its own issues if you actually try to use it. We've wasted weeks of effort on this. Cloud Run Tasks just skips this and is ultra fast too and handles scaling up to do a lot of jobs in such a simple fashion.

Honestly, managing Kubernetes is just overall a pain that has little benefit.

It is really hard to figure out what the benefits of Kubernetes is from my point of view. It has been a massive source of pain and costs and lost developer time.