The "adhoc" part is the problem. K8S is standardized and offers high-availability, failover, logging, monitoring, load balancing, networking, service discovery, deployments, stateful services, storage volumes, batch jobs, etc. And it can run self-contained on a single machine or scale out to 1000 nodes.

Why piece all of that functionality together yourself into some fragile framework instead of using a industry standard?

"Why piece all of that functionality together yourself into some fragile framework instead of using a industry standard?"

Quite recently developed "industry standard". Many tools mentioned have been used for tens of years, they work robustly, are well documented and there is lots of people who can use them. I personally would use the word "industry standard" a little bit differently.

You still have to put them all together into some custom solution just for your setup which adds overhead and fragility. New employees will have to learn that instead of using K8S APIs. Deploying new components can’t take advantage of the wide and fast growing ecosystem.

There’s really nothing like the full suite that kubernetes provides.

> New employees will have to learn that instead of using K8S APIs

You know tech has made it when people offer argumentum ad Java.

'Sure, X might not be the best solution, but you can always buy someone to do it.'

It's also very empowering for people whose job isn't to run things, but to build the things that are to run. I've worked on a dozen-or-so of bespoke "industry standard" setups, and each and every one had a number of weird quirks, involved learning some new "industry standard" components and either made it very hard and dangerous for non-ops/devops/infra people to run their things themselves, or had homegrown tooling that pretty much replicated what the k8s API can do, just only a small subset and badly. Some YAML and kubectl are well within what a typical data scientist can be expected to understand, more so if it means they can run their things on a dev cluster themselves, and in a pinch, that data scientist can debug prod issues of their things, because it all works the same way. We have a very useful bot that was built and deployed by someone decidedly not ops while waiting for jobs to finish – a simple K8S deploymend YAML is like 50 LOC, with 40 being pretty much standard, et voilà, running bot, without having to build lots and lots of automation in-house or having to take up ops time to deploy this for them or having to grok advanced sysadmin-ing first. Used with appropriate caution and safeguards, it's super powerful.

> Quite recently developed "industry standard". Many tools mentioned have been used for tens of years, they work robustly, are well documented and there is lots of people who can use them.

k8s is based off of Borg, which has existed for far longer.

https://research.google/pubs/pub43438/

"What Google does" is not an industry standard.

That's not what I'm saying. Your point was about the maturity of k8s which makes sense if it sprung from nowhere, but k8s encapsulates a lot of "lessons learned" of a very stable and mature product (even if proprietary).

Really it's just "inspired" by borg. Using both, they're really not very similar.

It can be worth it to piece things together yourself. A complex tool can also be fragile if you don't take the time to learn and understand every facet of it.

If you only need certain parts of what k8s offers, building those parts yourself can offer you more stability, control, and insight into what your application is doing.

As with anything else it varies case-by-case.

Building it yourself is also a case of "resume-driven development".

Not that there aren't good reasons or good outcomes from having done a deep dive, but just putting it there.

I think using something like k8s prematurely is more an example of "resume driven development" than the other way around.

Building it yourself doesn't mean building all of it. For example, it's quite easy to get zero downtime deployments with a tiny bit of systemd configuration and the SO_REUSEPORT socket option. That seems easier for a team to understand than "here is kubernetes and everything that comes with it"

That is, however, a very tiny slice of what might be needed.

You need to deliver the application to the machines somehow. You might need to configure the network to reach the application, etc. All quite common tasks.

(And honestly, systemd can bite you just as much if not worse than k8s definitions, because the API is much less cohesive and the defaults like to cut off your hands)

Building one yourself is a good exercise for understanding something, I agree.

The problem with that logic is everyone on your team will need to do it, so you're going to be stuck picking a standard. Should it be yours, should it be mine, or should we both just learn something with a large community behind it?

Nothing is perfect of course, but k8s makes a really good target for CI/CD, which is something you want when you're developing as part of a team. If you're not quite a team yet and you don't know how to bootstrap k8s and CI/CD, then you need to figure out when those types of things are important.

Probably lots of people could stick with a monolith and a VM for longer than they did, but automated testing will save you a fair bit of time if you're not figuring out how to do it at the same time.

k8s is nice, yes. But, by its age, no more standardized now, than MySQL was 15 years ago.

k8s also adresses a very small portion of the market. If you have to scale, yes, you might need k8s. Chances are, you don't. Really.

I stopped counting the (supposedly big) customers that burnt themselves into k8s when they logistically need not to, and only brought organisational issues on them.

"some haproxy" - what is the haproxy configuration? I tried doing this myself and then realized setting up a cheap k8s cluster on DO was way easier. Now, yes, there have been occasional problems, but not since I just let DO handle the whole thing. I can do zero-downtime deploys, cronjobs (why is spawning a pod so wasteful? it spawns it, runs the job, and then kills it?), all with a single "kubectl apply" command that takes a split second to run.

If you're in AWS, you can use their ELB (Elastic Load Balancer) service instead of setting up your own haproxy. I worked on a team that used it for years without any issues orchestrating zero-downtime deployments. It was extremely easy and didn't require any real configuration.

Having done it myself in the past?

Setting up and taking care of k3s is much easier.

yeah, we started out with ansible aswell. but some stuff is just way harder. especially no downtime. haproxy is fragile and sending files via haproxy over ssh aswell. docker is also a standarized package+repository format. before we used some kind of hacky cdn solution, etc. it was stuff glued together, written by myself and I was the only guy who understood it and ever will be.