1.) What would you think of things like hetzner / linode / digitalocean (if stable work exists)
2.) What do you think of https://sst.dev/ or https://encore.dev/ ? (They support rather easier migration)
3.) Could you please indicate the split of that 1 million $ in devops time and google cloud costs unnecessarily & were there some outliers (like oh our intern didn't add this specific variable and this misconfigured cloud and wasted 10k on gcloud oops! or was it , that bandwidth causes this much more in gcloud (I don't think latter to be the case though))
Looking forward to chatting with you!
But this is really a spurious concern. I myself used to care about it years ago. But in practice, rarely do people switch between cloud providers because the incremental benefits are minor, they are nearly equivalent, there is nothing much to be gained by moving from one to the other unless politics are involved (e.g. someone high up wants a specific provider.)
I feel like Kubernetes' downfall, for me, is the number of "enterprise" features it (got convinced into) supporting and enterprise features doing what they do best: turning the simplest of operations into a disaster.
Github Actions CI. Take this and make a few more dependencies and a matrix strategy and you are good to go: https://github.com/bhouston/template-typescript-monorepo/blo... For dev environments, you can add post-fixes to the services based on branches.
> How do you share storage?
I use managed DBs and Cloud Storage for shared storage. I think that provisioning your own SSDs/HDs to the cloud is indicative of an anti-pattern in your architecture.
> How do the docker containers know how to find each other?
I try to avoid too much communication between services directly, rather try to go through pub-sub or similar. But you can set up each service with a domain name and access them that way. With https://web3dsurvey.com, I have an api on https://api.web3dsurvey.com and then a review environment (connected to the main branch) with https://preview.web3dsurvey.com / https://api.preview.web3dsurvey.com.
> How does security work?
You can configure Cloud Run services to be internal only and not to accept outside connections. Otherwise one can just use JWT or whatever is normal on your routes in your web server.
I have been converging on a similar stack, but trying to avoid using a load balancer in an effort to keep fixed costs low.
Yeah I definitely want to also avoid a load balancer or gateway or end points as well for cost purposes.
https://github.com/bhouston/template-typescript-monorepo
This is my living template of best practices.
If you want some guidance, shoot me an email (in profile). You can run most stuff for peanuts.
I appreciate the offer! But it is not as robust and it is more expensive and misses a lot of benefits.
Back in the 1990s I did FTP my website to a VPS after I graduated from Geocities.
Google Cloud charges based on CPU used. Thus my servers have no traffic, they cost less than a $1/month. If they have traffic, they are still cost effective. https://web3dsurvey.com has about 500,000 hits per month and it costs me $4/month to run both the Remix web server and the Fastify API server. Details here: https://x.com/benhouston3d/status/1840811854911668641
Also it will autoscale under load. Thus when one of my posts was briefly the top story on Hacker News last month, Google Cloud Run added more instances to my server to handle the load (because I do not run my personal site behind a CDN, it cost too much, I prefer to pay $1/month for hosting.)
Also deploying Docker containers that build on Github Actions CI in a few minutes is a great automated experience.
I do also use Google services like Cloud Storage, Firestore, BigQuery etc. And it is easier to just run it on GCP infrastructure for speed.
I also have to version various tools that get installed in the docker like Blender, Chromium, etc. This is the perfect use case for Docker.
I feel this is pretty close to optimal. Fast, cheap, scalable, automated and robust.
Why are you actually using google cloud for blog post hosting.
Also you said a million $ kubernetes
wait a second ,have you converted those million $ to 4$ per month
what tom foolery is this
But I am also the Founder/CTO of Threekit.com.
I hope that makes sense now.
Or just https://github.com/mightymoud/sidekick or coolify or dokku or dockify , like there are million of such things , oh just remembered kamala deploy from DHH and docker swarm IIRC (though people have seemed to forget docker swarm !)
I like this idea very much !
Yup. Isn't it Knative Serving or a home grown Google alternative to it? https://knative.dev/docs/serving/
The key is I am not managing Kubernetes and I am not paying for it - it is a fool's errand, and incredibly rarely needed. Who cares what is underneath the simple Cloud Run developer UX? What matters for me is cost, simplicity, speed and understandability. You get that with Cloud Run, and you don't with Kubernetes.
Kubernetes can be as complex or as expensive as you'd like but it's also fairly possible to run a pretty bulletproof simple Kube cluster.
Here are my concerns:
With Kubernetes is that you need to pay for a few node just to keep it up, and then you need to pay for your nodes, no matter how much you use them.
Remember that Cloud Run charges based usage, so if a service sits unused for a while, which often happens in a heterogeneous microservices environment, you don't pay for it.
Also autoscaling is slow (Cloud Run autoscales really quickly, about as fast as your docker can be loaded and started, which for me is 1-3 seconds, where as I found Kubernetes auto-scales on the order of minutes) unless you over-provision, which is costly. This lets one scale to zero even without much of a hit.
I also ran into massive issues trying to get GPUs to work in Kubernetes - it was a driver nightmare that has wasted weeks of time collectively over the years. Whereas they are auto-provisioned properly on Cloud Run if you request them.
Lastly job systems on Kubernetes are a nightmare of configuration. The built-in scheduler cannot handle a lot of jobs but Argo also has its own issues if you actually try to use it. We've wasted weeks of effort on this. Cloud Run Tasks just skips this and is ultra fast too and handles scaling up to do a lot of jobs in such a simple fashion.
Honestly, managing Kubernetes is just overall a pain that has little benefit.
It is really hard to figure out what the benefits of Kubernetes is from my point of view. It has been a massive source of pain and costs and lost developer time.
The infrastructure now incredibly understandable and simple and cost effective.
Kubernetes cost us >$million in both DevOps time and actually Google Cloud costs unnecessarily, and even worse it cost us time to market. Stay off of Kubernetes as long as you can in your company, unless you are basically forced onto it. You should view it as an unnecessary evil that comes with massive downsides in terms of complexity and cost.