So instead of knowing about K8s services, ingests and deployments/pods I have to learn 15 tools.

Ingests are not much more complicated than an nginx config, services are literally 5 lines each pod, and the deployments are roughly as complicated as a 15 line docker file.

If you're familiar with Linux (which should be considered required-reading if you're learning about containers), most of this stuff is handled perfectly fine by the operating system. Sure, you could write it all in K8 and just let the layers of abstraction pile up. Or, most people will be suited perfectly fine by the software that already runs in their box.

I work in a small company, we don't have a sysadmin, so mostly we want to use managed services. Let's say we want a simple load balanced setup with 2 nodes. Our options are:

- Run our own load balancing machine and manage it (as said, we don't want this)

- Use AWS/GCP/Azure, setup Load Balancer (and rest of the project) manually or with Terraform/CloudFormation/whatever scripts

- Use AWS/GCP/Azure and Kubernetes, define Load Balancer in YAML, let K8S and the platform handle all the boring stuff

This is the simplest setup and already I will always go for Kubernetes, as it's the fastest and simplest, as well as the most easily maintainable. I can also easily slap on new services, upgrade stuff, etc. Being able to define the whole architecture in a declarative way, without actually having to manually do the changes, is a huge time-saver. Especially in our case, where we have more projects than developers - switching context from one project to another is much easier. Not to mention that I can just start a development environment with all the needed services using the same (or very similar) manifests, creating a near-prod environment.

I think the argument there is that it's only simple because the complexity of k8s has been taken away. I don't think anybody has claimed deploying to a k8s cluster is overly complex; running it well, handling upgrades, those are huge time sinks that need the requisite expertise.

Much like Multics was "simple" for the users, but not for the sysadmins.

That's the point though right? A good (couple of) sysadmins can run a k8s cluster that can be leveraged by dozens (even hundreds) of dev teams. Instead of every team having to re-invent the wheel you get a common platform and set of deployment patterns that can fit most any use case. Of course if you don't have multiple different teams (or every team is running their own k8s cluster) then that is definitely a problem. But just because a handful of teams make an ill-advised investment in k8s when they could do easily with something much simpler doesn't mean that k8s is "too complex." Too complex for that use case sure, but for the vast majority of k8s deployments I would wager that it does add a lot of value and subsume a lot of the inherent complexity of running distributed, fault-tolerant, multi-tenant applications.

Taking the complexity of k8s away was just gonna happen. As someone who built everything from scratch at a previous company, I chose eks at a start-up because it meant that the one-man-systemsguy didn't have to worry about building and hosting every single cog wheel that is required for package repos, OS deployment, configuration management, consul+vault (minimum), and too many other things that k8s does for you. Also, you can send someone on a CKA course and they know how your shit works. Try doing that with the hodge-podge system you built.

Training is a great point, and I think that's why major clouds are going to be stickiest (in terms of using them vs migrating to new things).

The central problem of most companies has been finding / affording people who can maintain their stuff.

If Amazon / MS / Google can make it simple enough that skilled people can be quickly cross trained, and then have enough architecture knowledge to be productive, that's a huge win over "require everyone to spend 6 months muddling through and learning our stack we built ourselves and partially documented."

Set up servers at linode and use the linode node balancer?

> Being able to define the whole architecture in a declarative way

With k8s (and other 'cloud' stuff) you seem to need to know a whole mess of a lot of the tool's stuff up front, vs a "progressive enhancement" way of doing one thing, getting it working, doing something else, getting it working, etc.

You run a small company, I'd argue that you aren't "the average user". For you, Kubernetes sounds like it integrates pretty well into your environment and covers your blind spots: that's good! That being said, I'm not going to use Kubernetes or even teach other people how to use it. It's certainly not a one-size-fits-all tool, which worries me since it's (incorrectly) marketed as the "sysadmin panacea".

I have been professionally working in the infrastructure space for a decade and in an amateur fashion running Linux servers and services for another decade before that and I am pretty certain that I would screw this up in a threat-to-production way at least once or twice along the way and possibly hit a failure-to-launch on the product itself. I would then have to wrestle with the cognitive load of All That Stuff and by the way? The failure case, from a security perspective, of a moment's inattention has unbounded consequences. (The failure case from a scaling perspective is less so! But still bad.)

And I mean, I don't even like k8s. I typically go for the AWS suite of stuff when building out systems infrastructure. But this assertion is bonkers.

Why? You still need to manage all that for your server even if you are running kubernetes on top of it.

I can’t imagine anyone with root access to a kubernetes server is any less dangerous that root on a simple webserver.

No, I don't, because I can yawn dramatically and I can go to any cloud provider and get a k8s cluster with generally consistent and at worst a moral-equivalent set of standard building-block cloud tools already set up. It won't cost me much, it will work mostly-predictably out of the box, and there's support right there for when it fails. Like, that's what k8s is there for. I use AWS pretty exclusively so this doesn't appeal to me, but what does is doing the moral equivalent and having ECS just...there. (Or even better, Fargate, if I can't solve the bin packing problem by myself.)

I haven't "managed a server" outside of my house for a few years now, and I quite like it. I theoretically have had root to ECS clusters, but I've never logged into them. Why would I? Amazon is going to be better at it than I am. Not only do I have more important things to be doing, but I'll do a worse job of it than they will. And to be clear: I consider myself pretty kinda really good at this stuff. But not good enough to make it a competitive advantage unless it's what I want to sell, and I sure as heck don't.

And the article's point, that whatever comes next will probably be better and might even be The Real Thing--I think that is wise.

> Why would I? Amazon is going to be better at it than I am.

Until it's not. Then suddenly you're trying to decipher cryptic cloud provider error messages in a service that made a false promise to you that it's abstraction was so air-tight that you'd never have to learn the underlying technology at all.

Then suddenly, you do need to know the underlying implementation, and quickly.

Yup! I used to feel exactly as you do, and I make it my business to understand what is below the abstraction besides because some old habits die hard (and because I just like this stuff, tbh). But I started working at places with the kind of conservation and pre-testing that make that much less critical. Those organizations also that pay a great deal of money for the kind of support to make knowledge a habit of curiosity and personal fulfillment rather than save-the-worlding.

I haven't needed to do something like that in production, as opposed to pre-production deployment suss-out, since (and I went and checked my enough to be sure) 2017. Though, to be fair, I've been working in devrel since last August, so call it four years of rooting around in the trenches, not five. ;)

> most of this stuff is handled perfectly fine by the operating system

No, you have to write or adopt tools for each of these things. They don't just magically happen.

Then you have to maintain, secure, integrate.

k8s solves a broad class of problems in an elegant way. Since other people have adopted it, it gets patched and improved. And you can easily hire for the skillset.

Okay, so let's add a couple of things.

How do you do failover?

Sharing servers to save on costs?

Orchestrate CI/CD pipelines, preferably on the fly?

Infrastructure as Code?

Eventually you a point where the abstraction wins. Most people will say "but AWS...", but the reality is quicker, easier to use, and runs via multiple providers, so I think it's going to keep doing well personally.

Not the OP here.

We aren't really comparing apples and oranges in all cases that have been talked about in the larger thread. Some of the comparisons seem to be between "self hosted LAMP stack" vs. "kubernetes as a service on AWS". These are vastly different things. We should compare "self hosted LAMP stack" vs. "hosted in cloud LAMP stack" for example or "self-hosted kubernetes" vs. "self-administered kubernetes on EC2" vs "kubernetes as a service on AWS". All of these will have vastly different characteristics, pros and cons depending on your company and teams' realities.

Failover is something that a load balancer does automatically for you. Your services just need to provide a health check. Now where you actually run those nodes is a different thing. These might be slow to procure servers hosted at your provider. Or these might be manually set up EC2 instances or terraformed EC2 instances. Dunno what everyone uses as load balancers nowadays but a previous place for example had F5s and we had our own vsphere farm.

Sharing servers: I don't think this is a good idea at all except if you mean internally and if you do that then there's good and bad ways (see above on vsphere farm. If one project caused another to starve performance wise because of what was running on the same physical machines it was easy to resolve. If this was virtual servers at a traditional hoster, good luck. AWS is probably somewhere in between with EC2 and especially their storage.

Dedicated CI/CD pipelines: This is an awesome one to have and can cost an arm and a leg. I enjoy this very much at my current place w/ EC2 CI agents that scale with the number of devs currently working and dedicated "complete copy of Prod" dev environments (basically a kubernetes namespace for each dev/QA person/e2e test run to play with as they like).

Infra as code: Does not require kubernetes at all but can be implemented with kubernetes. If you already used docker to run stuff anyway for example and you can "abstract away" the kubernetes complexities to your SRE team and/or AWS, go ahead and use kubernetes. But be aware that if nobody at your place actually know kubernetes because you just relied on the hosted version of it, you're at the whim of their support people when something blows up in Production. You may not be big enough to have your own SRE team to take care of this but then you might also just not really benefit enough from kubernetes complexity and a simpler arrangement could have been easier for the people you do have to actually understand.

I think you've missed the point I was making.

Essentially if you work back from the desired state of having IaC, CI/CD, test environments per MR, you likely see something like k8s as a framework that helps you achieve that.

Of course, if you start from "I just need a LAMP stack" you might have a very different conclusion. But when you reach the same endgame ( actually I need an environment for every MR ), you've probably incrementially built something more complex and bespoke.

This will explain why there are dozens of us who are quite happy with the product. The only real question is, do you already know it and do you find it much harder to ship a deployment to a managed k8s cluster vs systemd unit files?

If not, it might be an abstraction worth having. If you don't already know how though, then you might have better things to be doing with your time.

This really depends on how many boxes you have.

> I have to learn 15 tools.

kubectl, kustomize, helm, istio, graphana, various flavors of ingress controllers, overlay networks, service meshes, storage controllers, etcd, etc.

from tfa: https://landscape.cncf.io/

you still have to learn 15 tools, just now they are hidden behind the scenes, and you still have to understand the underlying systems to reason about your containers.

this isn't for or against k8s - i'm a right tool for the job guy - but as a tool kubernetes doesn't solve problems, it encapsulates them and shifts them around.

Plus all the cloud tools are immature with terrible error handling and logging.

So after an enjoyable time crafting a 30 level deep json file you get a failed helm deployment with a error message like "timed out waiting for the condition".

15 mature well documented tools are a lot easier than 15 kludged ill thought out Kubernetes definitions.

Any serious Kubernetes environment is not 5 lines per pod, its the hell of rbac and pod security policies and all sorts of overly cryptic cruft.

“ For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.”

Or… you could not.

https://news.ycombinator.com/item?id=9224

The difference, is that Dropbox is user-facing software, while Kubernetes is software engineer-facing. Dropbox has to be usable by tech-illterate people. Tech-illiterate people have no idea what a Kubernetes is.

There is value in creating a vertically integrated solution in a space, similar to what Dropbox did, so if you find yourself building many of the pieces of Kubernetes internally, it's worth considering if adopting Kubernetes wouldn't be a more efficient use of resources.

That comment has aged brilliantly.

Thanks for that!

how is quoting this here relevant? nobody's saying k8s isn't successful or going to be successful—the argument is whether its complexity and layers of abstraction are worthwhile. dropbox is a tool, k8s is infrastructure. the only similarity between this infamous post and the argument here is that existing tools can be used to achieve the same effect as a product. the response here is "that'll never catch on" (because obviously it has), rather it's "as far as infrastructure for your company goes, maybe the additional complexity isn't worth the turnkey solution"

"You don't need Kubernetes, for a Linux user you can already build a custom solution quite trivially by setting up a custom package repo then build and distribute your application using apt, then configuring SysVinit to monitor your services, whilst using Ansible to configure iptables rules in combination with a simple load balancer you can manage yourself, then use syslog to monitor logs across all your machines whilst hand-waving away secrets management as a problem with 'serverless'"

Yes, you could. Some people do. Others don't, because even if you need a small portion of the features a turnkey solution is likely a better choice in the long run than hand-rolling your own mix of 15+ different technologies to achieve the same thing.

Confounded why sshfs wasn't chosen.

So you have a version of Kubernetes that is as easy to use as Dropbox? Where do I sign up for the beta?

I'm personally glad that Kubernetes has saved me from needing to manage all of this. I'm much more productive as an applications engineer now that I don't have to stare at a mountain of bespoke Ansible/Chef scripts operating on a Rube Goldberg machine of managed services.

Instead, you can now admin a Rube Goldberg machine of Helm charts, which run a pile Docker containers which are each their own microcosm of outdated packages and security vulnerabilities.

> Rube Goldberg machine of Helm charts

I love k8s but I do want to say that I hate the 'standard' way that people write general purpose Helm charts. They all try to be super configurable and template everything, but most make assumptions that undermine that idea, and I end up having to dig through them to make changes anyway.

I have found much more success by writing my own helm charts for everything I deploy, and putting in exactly the amount of templating that makes sense for me. Much more simple that way. Doing things this way has avoided a Rube Goldberg scenario.

your argument seems to be "its ok if youre rube goldberg"

just wait till you have a predecessor

That's the opposite of my argument. I'm saying that the predominant style is Rube Goldberg, but Helm charts don't have to be written that way. Instead of writing an unreadable mess that is 90% template, just template the 5% that you need, and the whole thing is very readable.

thats what you are hearing. what everyone else is hearing: "_my_ code is self documenting, so obviously its more legible!"

This x10. Each such setup is a unique snowflake of brittle Ansible/Bash scripts and unit files. Anything slightly different from the initial use case will break.

Not to mention operations. K8s give you for free things that are a pain to setup otherwise. Want to autoscale your VMs based on load? Trivial in most cloud managed k8s.

> Remember the days of assuming that your box was secure without having to do 10123 layers of abstraction?

Yep, I remember when I deployed insecure apps to prod and copied secrets into running instances, too.

Remember how the ops team kept installing Tomcat with the default credentials?

This was the funniest point in that comment to me.

Read the intended way, it's borderline wrong.

Read as "remember when people assumed security without knowing" is basically most of computing the further back in time you go.

Have you ever tried to package things with .dep or .rpm? It's a f** nightmare.

A place to store secrets and provide those secrets to your services.

"A problem that is unique to kubernetes and serverless. Remember the days of assuming that your box was secure without having to do 10123 layers of abstraction?"

I remember 10 years ago things were not secur, you know when people baked their credentials in svn for example.

lol. as someone who has packaged stuff I can tell you that this K8S is orders of magnitudes more complicated. Also, once you figure out how to package stuff, you can do it in a repeatable manner - vs K8s which you basically have to babysit (upgrade/deprecations/node health/etc) forever and pay attention to all developments in the space.

   FROM python:3.8
   RUN apt install libsomething
   ENV RELEASE=production
   COPY . .
   RUN poetry install
   CMD ["poetry", "run", "server"]

What would be the rpm/deb equivalent of those 6 lines? Would it work on MacOS?

let’s unpack this for a while.

what is python:3.8? is this reproducible?

what is apt? where is the install coming from?

What about poetry?

Yeah it’s cool for shits and giggles but when this thing breaks you’re going to be in a world of pain

for the rpm equivalent:

python setup.py bdist_rpm

wat?

Oh my. I'm not sure that I'd use Python to make a point about easy interop with distro package managers. It quickly descends into a nightmarish hellscape if you have more than a few dependencies or, different versions of python, or god forbid: C extensions.

bdist_rpm isn’t equivalent to the Dockerfile above. It can be made reproducible with a few changes (locking the upstream image to a hash, locking the apt package version), but that’s likely overkill. Because when it breaks you’re not in for a “world of pain” at all, you just have a failing CI for an hour.

I take it from the lack of an answer to the question that the equivalent non-docker packaging would be much more complex.

.deb packages are literally just a compressed archive with a folder structure that mostly mimics your folder structure on the hard drive. You've got some pre- and post-hooks where you can write some shellscript to do fancy stuff, and a signing process to ensure authenticity. Autostart is a SysV init script or systemd xml file away. How is that a f* nightmare?

Checkinstall makes packaging pretty easy for anything you aren't trying to distribute through the official distro channels.

https://help.ubuntu.com/community/CheckInstall

I can setup a Kubernetes cluster, a container registry, a Helm repository, a Helm file and a Dockerfile before you are finished setting up the infrastructure for an Apt repository.

Exactly, an autoscaling cluster of multiple nodes with everything installed in a declarative way with load balancers and service discovery, all ready in about 10 minutes. Wins hands down.

My experience is the opposite - an APT repo is just files on disk behind any webserver, a few of them signed.

Setting up all the infra for publishing APT packages (one place to start: https://jenkins-debian-glue.org ) is far easier than trying to understand all the rest of the things you mention.

I mean, Kubernetes is just some Go binaries; you can have it up and running in literal seconds by installing a Kubernetes distribution like k3s. This is actually what I do personally on a dedicated server; it’s so easy I don’t even bother automating it further. Helm is just another Go binary, you can install it on your machine with cURL and it can connect to your cluster and do what it needs from there. The Docker registry can be run inside your cluster, so you can install it with Helm, and it will benefit from all of the Infra as Code that you get from Kubernetes. And finally, the Helm repo is “just files” but it is less complex than Apt.

I’ve been through the rigmarole for various Linux package managers over the years and I’m sure you could automate a great deal of it, but even if it were as easy as running a bash script (and it’s not,) setting up Kubernetes covers like half this list whereas setting up an Apt repo covers one item in it.

Yeah I don't understand where all this fictional .deb and APT "complexity" is coming from. Everything uses standard abstractions that are decades old at this point..... oh no, you have to make some directories! You have to put a manifest file in the right place! Oh my god, now you have to run a command!

Now make it not-brittle and prone to falling over, without using hosted k8s. ;)

... but then you could pay a fraction for bare metal cloud hosting instead of paying out the nose for managed K8S at Google or AWS.

Its complexity and fragility are features. It's working as intended.

no. you cannot.

This is supposed to be an argument against Kubernetes?

Nope, just an argument against the "you must write all of this yourself" line. :)

There was some project where one wrote all of that (essentially what Kubernetes does) in like 8k lines of bash script. Brilliant, yes. But there is not way I want any anything similar in my life.

I am not the biggest fan of the complexity Kubernetes is, but it solves a problems there is no way I want to solve individually and on my own.

I think the point of the blog post in the OP is that it should be a bunch of bash scripts with very few interdependencies, because most of the requirements in the grandparent comment are independent of each other, and tying them all together in a tool like kubernetes is unwieldy.

Some of these are decent points, but a couple are misleading.

The security one is the big one. Things were just not as secure (and did not need to be as secure) “back then”. K8s has a lot of complexity, and security should definitely be simpler so it’s harder to misconfigure, but not doing anything is not viable.

Saying “Package Managers” is fine until you realise they solve only part of the problem. The mainstream ones are good tools to update package (and dependencies) from version X to Y. When you’re running a distributed system, it’s often not that simple if you want to be reliable. Coordinating a slow global update of your application from version X to Y (safely) is pretty tricky and I’m not aware of good self-contained solutions to this.

You're making their point for them.