I don't know my dude, all 3 major clouds offer "canned" k8s services that you can set up in a ridiculously short amount of time with Terraform and your CI platform of choice.

I agree with some other comments in this thread about a general fervor in the Enterprise space to "modernize" needlessly. This conversation usually lands on the company copying what everyone else is doing or what Gartner tells them to do. Cue "DevOps".

100 percent agree with your comments on something simpler. I can't tell you how many times I've debated with our Analytics teams to just use Docker Compose/Swarm.

> all 3 major clouds offer "canned" k8s services that you can set up in a ridiculously short amount of time with Terraform and your CI platform of choice

I don't agree. I spun up a Kubernetes cluster in Azure, which was indeed easy. But then I had to figure out how to write the correct deployment scripts to deploy my docker containers to it, and how to configure all the security stuff. After more than a week of trying to figure it out, I decided to ditch the whole solution and go for Azure Container Instances instead. It was too much for me to learn about all the concepts of Kubernetes, how you configure them, how to make it work for solutions that are not as simple as the example on the website, and how to navigate through the various different methods of deploying stuff.

Maybe I'm just too dumb. But I wasn't going to invest a month of my time into doing something that should be simple enough for an average developer to accomplish.

You're not dumb, you're just new to it, and it's fundamentally hard stuff anyways, and if you can find a higher-level abstraction that lets you get work done faster, then all the better. However, the question is comparing Kubernetes to traditional VM-based infrastructure (especially with pet nodes) whereas you're comparing Kubernetes to a higher-level abstraction.

For what it's worth, deploying in Kubernetes is pretty easy once you figure it out (and often finding the information is the hardest part). All you need to do is update the Deployment resource's "image" parameter. You can do that with `kubectl patch` like so:

    kubectl patch deployment foo -p '{"spec":{"containers":[{"name":"main","image":"new-image"}]}}'

Kubernetes will handle bringing up a new replicaset, validating health checks, draining the old replicaset, etc.

Please, do not manage deployments with imperative kubectl commands, I beg of you.

The question is: what tool to use? I'm a solo developer running a very small kube cluster for my hobby project. I very much wanted to have a declarative version controlled state of my cluster. Every time I try googling solutions I get flooded with some enterprise Saas offerings that do nothing I want.

I managed my stateful sets/services for a while with terraform, but my experience was absolutely terrible and I have stopped that eventually. I now use "kubectl patch" and "kubectl apply" with handwritten yaml, but the workflow feels very clunky.

Intuitively it seems obvious to me that there must be a tool helping with that, but for some reason I was absolutely not able to find anything that would be even a little bit helpful. I am considering writing a couple python scripts that will automate it.

kubectl apply on a directory full of yaml works fairly well for small stuff. Check it into git and it's version controlled.

If you need something more flexible than that, try making your own helm chart. Helm will give you some text templating, pre and post hooks, some basic functions, and some versioning and rollback functionality.

You can start simple by just pasting in your existing k8s yaml, and then just pull out the pieces you need into variables in your values file. If you need to change an image version, then you just update the variable and `helm upgrade mychart ./mychart`

kubectl apply on a directory doesn’t work because deleting a resource manifest won’t delete the corresponding resources.

It does work, as long as you’re not deleting anything. :) That might be good enough for a “very small kube cluster for my hobby project”

I only use ‘kubectl apply’ for small stuff where I only have a couple resources. Anything more complicated and a tool like helm is much more useful.

'kubectl delete -f dir/' will delete all resources in the directory.

Right, but the only resource you want to delete is no longer in that directory, so you’ve now deleted every resource you except the one you actually wanted to delete. :)

Ah, I slightly misunderstood. Still, you should always have your manifests under version control so this shouldn't ever be a problem :)

I'm a fan of emrichen, but simple text templating (Jinja) works almost as well. (j2cli means you can just provide a yaml file with per-stage configs.)

Fluxcd, or argo CD if you want a nice UI

I probably wouldn't do this, but what problem does this cause?

The declarative approach is a more sustainable way to run Kubernetes. If you define some desired state in manifests and apply them to a cluster, they can be applied again to new clusters or the same one and Kubernetes will attempt to maintain the desired state.

This state can be version controlled, written in stone, whatever you prefer and it can always be attained.

When administrators start issuing imperative commands to a cluster, state starts being changed and there is no record[0] of the state Kubernetes is being asked to maintain.

[0] Not entirely true, the state can always be retrieved from the cluster so long as it hasn't failed.

I find this to be a common misconception, stemming from a misunderstanding of what "declarative" means (especially common when people are discussing tools like Terraform).

Firstly as you point out, there is a record of the state Kubernetes is being asked to maintain: it's in the API server as the spec of each resource.

Secondly, using `kubectl` "patch" in the manner described is not making changes to the cluster state directly, it's making changes to the specification of what should be maintained, and the various controllers effect the state changes.

Fundamentally, the argument seems to come down to "you don't have a record of what you once asked the API server to do", and that's fair enough - you don't. But that has nothing to do with imperative or declarative models.

I'm not advocating actually doing this on a day-to-day basis, but the arguments against it are not ones of imperative vs declarative.

> Fundamentally, the argument seems to come down to "you don't have a record of what you once asked the API server to do", and that's fair enough - you don't. But that has nothing to do with imperative or declarative models.

You do actually have this record. First of all, because k8s has an audit log, and secondly because deployments maintain a revision history (so you can always rollback--kubectl even supports this via `kubectl rollback undo`).

You're correct, but, when approaching this from the human perspective, it's less about the technology and the reality and more about "do I have a record of the last state I asked the cluster to be in before I fucked it up?". Ultimately that's what matters. Auditability is great, but in practice, when the shit hits the fan, can I get the application that brings in our company's money, and our customers rely on to breath, back to the state it was after 5 years and 73 updates?

Updating the manifests, pushing them to version control and having CD deploy them encourages humans to "do the right thing".

It would be quite easy though to just tweak that one environment variable though while I patch in the new image version to update that one service; until the entire cluster dies and I can't retrieve the last definition and need an equivalent cluster up within the hour.

This is really more about the practice of writing down the state you want the cluster to be in (the spec) and showing it to the cluster, than just ordering it to do one thing without context.

In this sense, Declarative Vs Imperative is just a proxy term for "do I have a record of the state I asked the cluster to keep?"

Given that Kubernetes' docs[0] discuss using imperative commands, I think it's a fairly reasonable way to describe it.

[0] https://kubernetes.io/docs/tasks/manage-kubernetes-objects/i...

Why is applying a full manifest “declarative” but applying a patch is “imperative”? That’s clearly an error.

Applying a full manifest is "this is your state now".

Applying a patch is "make these changes to your existing state".

That dependency on existing state is a difference, and it seems to map reasonably well to what declarative/imperative seem to usually be used to mean in this context.

1. A strategic merge patch says "this is your state now" (the only difference is the scope of the state in question, with a full manifest including a bunch of extraneous stuff)

2. "make these changes to your existing state" is still declarative

> Applying a full manifest is "this is your state now".

It isn’t though. It’s “please make the state look like this eventually”. You do not patch state, just spec. Controllers effect the changes.

You're both correct, but you're talking about different states. The parent's "state" is the state of the etcd database and your state is the actual state of the resources that the controllers are managing.

That said, the parent is wrong that a full manifest is declaring the (etcd) state in a way that a (strategic merge) patch isn't--both are declaring etcd state, but a strategic merge patch is doing so in finer-grained increments. A strategic merge patch can declare zero state or many full manifests, while applying full manifests can only work in increments of complete resource manifests. But both are telling Kubernetes "this is your (etcd) state now".

Then open a PR.

This is a strange way to concede an argument, but I’ll take it.

People are incorrectly assuming that using kubectl implies invoking it from an administrator's laptop. Of course, you can and should invoke kubectl from your CD pipeline. The CD pipeline maintains its own record of runs, and Kubernetes deployments have a revision history.

Moreover, people in this thread also don't know what "declarative" means. The patch is declarative, and "declarative" doesn't provide the claimed benefits. For example, as with the patch command, I can create, apply, and delete a Kubernetes manifest file (indeed, I can apply directly from stdin!) and there is no additional record of the change beyond that which would exist if I had just applied the patch.

A patch is declarative in that it's an idempotent command which requires you to fully define the patch. It's still something that is being manually defined rather than contained in a file to be applied. Of course, if that file is just being locally held and modified on someone's laptop instead of committed to version control, it's a moot point.

I will grant you that `kubectl delete -f` breaks my argument, since it's imperatively modifying objects given a declarative manifest. Re: changes via stdin, I mean sure; you can also pipe a deployment manifest into `ex` or `sed` to make changes on the fly and then echo it back to `kubectl apply -` but I wouldn't recommend it.

Yeah, my point is that whether you're changing a manifest and doing an apply versus a patch isn't relevant, the relevant bit is whether or not you have a version history and in both cases you do via your CD pipeline and via k8s deployment revision history. You can also commit to git, but I don't think there's much value in committing every image version change to git.

Also, working in raw manifests isn't a panacea; if you delete a resource's manifest, Kubernetes won't delete its corresponding resource (maybe this is what you were referring to with your bit about `kubectl delete -f`). You need something that diffs the manifests against the current state of the world and applies the requisite patches (something like Terraform) but this isn't related to how you change image versions.

I agree with your points, but as I mentioned in another comment, if we take "imperative" and "declarative" as proxy terms for "I do or do not have a reference to the state I requested of the cluster OUTSIDE of the cluster"; my point is that updating the state, in full, by modifying the manifests and committing them to be deployed by your CD pipeline or otherwise is a better approach to ensuring you or someone else can rebuild your "empire" were it to unceremoniously burn to the ground.

I think you’re assuming that just because we’re updating the image via kubectl (which itself is invoked via CD pipeline) that the infrastructure isn’t codified and persisted, which isn’t the case. You can (and should) still have your infrastructure manifests saved in your infra repo/directory and version controlled; that’s orthogonal to how you update the current image version.

The same problems any imperative management within declarative config causes -- drift. If the tool you're using supports declarative configuration, all changes should be made exclusively via the declarative interface to prevent that drift. In this example, the new image should be added to the original manifest itself, not via a CLI update.

It depends on how you manage your changes. A lot of people don't have their infra-as-code manage the deployment's image field--rather, that's updated by the application's CD pipeline. There's no drift to worry about.

So upon deploy the CD pipeline calls kubectl with the proper deployment image and that's ok?

Yes.

Drift, as the other child comment mentions, but also loss of version control. I do not want to have to trawl through someone's shell history to figure out what they changed, nor do I want to have to redirect `kubectl get foo -o yaml` output into diff.

If everything is in code, and you have a reasonable branching strategy, it's much easier to control change, to rollback bad merges, to run pre-hooks like security checks and configuration validation tools, etc.

It's not drift, because the infra-as-code doesn't manage the image field (the application's CD pipeline does). You don't trawl through someone's shell history', you look at the CD pipeline history. Rollbacks are easy--you just deploy the prior version via your CD tool.

I think you're assuming that invoking kubectl means invoking it directly from a user's command line, but kubectl can also be called in a CD script.

If you're using ArgoCD or something then sure, but bear in mind the original statement you made was directed at someone who is new to K8s, and given a command that can be executed from their shell, they would likely assume that's what you meant.

It doesn’t have to be Argo, it can be Jenkins. Whether or not you use a CD pipeline is orthogonal to whether or not you use k8s. The best practice is to use a pipeline whether you’re targeting k8s or bare VMs or a higher level PaaS abstraction.

I'm not new to Kubernetes and I've been using containers since Solaris zones were introduced.

You know kubectl has a built in diff subcommand?

I do now! Thank you.

This is a declarative kubectl command.

Kubernetes' docs[0] disagree.

[0] https://kubernetes.io/docs/tasks/manage-kubernetes-objects/i...

They’re not being precise with their language. Applying a full manifest is no more “declarative” than applying a patch.

Spot on!

As an outsider, that command looks really easy to mess up.

* shell interactions with quotes * double quotes * interpolating into image namespace with no autocomplete * easy to forget an argument * do you get autocomplete against the deployment name?

Comparison: C# declaration of a complex type - it's less complex than the `kubectl` command above, but IDEs offer way more support to get it right.

* var x = new List<Dictionary<CustomType,int?>>()

This will light up warnings if you get anything wrong;

You get: * go to definition of `CustomType` * autocomplete on classnames * highlighting on type mismatches * warnings about unused vars. * if you initialize the var, the IDE will try hard to not let you do it wrong

So structurally,

1) in the code realm, for doing minor work we have really strong guarantees,

2) in deployment realm, even though it's likely going to hit customers harder, the guarantees are weaker.

I think this is behind the feeling that the k8s system is not ready yet.

About quoting rules: We desperately needs static analysis or IDEs (Emacs plugins, whatever) that "explain" exactly what quotes do in a particular context. When I returned to C++ recently, I was blown away by Clang-Tidy (JetBrains CLion integration). Literally: Clang-Tidy seems to "know what you really want" and give intelligent suggestions. For someone whom is a very average C++ programmer, it instantly leveled me up! I could see the same for someone writing /bin/sh, /bin/ksh, /bin/bash shell commands. If the IDE could give some hints, it would be incredibly helpful to many.

The quotes here have nothing to do with k8s, it's just how you encode JSON in shell.

apt-get install shellcheck

or

https://www.shellcheck.net/

You put it in a bash script that gets called from CI and move on with life. No one is typing this into their terminal every time they want to do a deploy.

> I think this is behind the feeling that the k8s system is not ready yet.

Whether you patch the deployment image from bash or from C# doesn't indict the k8s ecosystem.

correct! Here's why: Kubernetes is an operating system (a workload/IO manager) and its lineage is 16 years old. So it's like Unix around 1979, or Windows/DOS around ~1996. Imagine it's 1996 and you can pick between Windows NT (representing 16 years of MS development) or System 360 (~40 years out from IBMs first transistorized designs).

What I'm trying to say is that as an operating system, Kubernetes is now a young adult, and historically speaking, operating systems at this level of maturity have been adopted and ridden for decades, with much success. But, ya know, if you chose OS/360 in 1996 you would have a point.

1. You’ll use `kubectl set image deployment/gitlab gitlab=gitlab/gitlab-ce:14.9.2-ce.0` in production instead

2. kubernetes will abort the change if it doesn’t syntax check or the types don’t match.

Hah, I didn't even know about `set image`.

Don’t do that, use

    kubectl set image deployment/foo main=main:new-image

instead

It also accepts "*" if you don't know or care about the container name(s)

    kubectl set image deployment/foo "*=the/new/image"

and then, my other favorite (albeit horribly named, IMHO) "rollout status" to watch it progress:

    kubectl rollout status deployment/foo

TIL. Thanks for the advice.

It took me 4-5 hours to read and understand first few chapters of Kubernetes in action, I annotated a lot text as notes on the Oreily site, but didn't have to go back to it. I didn't have to do any security and account management stuff, it was being done by someone else. After 3 years of working with plain Kubernetes and OpenShift, I still didn't have to go back to the book. Basic concepts in Kubernetes are easy to understand, if you're working as a developer to deploy your apps in it and not in DevOps managing it.

https://www.oreilly.com/library/view/kubernetes-in-action/97...

You wrote: <<if you can find a higher-level abstraction>>

This a stretch, but to me Kubernetes is like the C programming language for infra. If you look at the entire software stack today, drill down (all the turtles), and you will eventually find C (everything goes back to libc or a C/C++ kernel). I assume any commercial (or non-!) "higher-level abstraction" for infra is already (or will be soon) built on top of Kubernetes. I am OK with it.

I write this post as someone who is uber-techincal, but I know nothing about actually using Kubernetes. I can do vanilla "hand-coded/snowflake" infra just fine in my constrained, private cloud environments, but nothing that scales like Kubernetes.

> If you look at the entire software stack today, drill down (all the turtles), and you will eventually find C (everything goes back to libc or a C/C++ kernel).

I might be nitpicking, but I'm not sure that's necessarily true. You could in theory write a compiler for a new language in C (or even assembly!), and once you have a working language, re-write the compiler in that new language. Now that there is no C code involved in the stack anymore, would that still count as a C-turtle?

Haskell for example, has some "bits" written in C, but a lot is written in Haskell or assembly[1]. So if you look at the WHOLE stack you'll find C _somewhere_ (probably most places), but I don't think _everything_ boils down to C.

Granted, a LOT of stuff is written on top of C.

[1] https://stackoverflow.com/questions/23912375/what-language-i...

Great point and not a nitpick at all! I hope the next generation of languages is written on top Python, Ruby, C++ (ultra modern), Java, or DotNet/C#. I wish more languages would figure out if they can host in JVM/CLR, which would provide some crazy interactions!

After seeing so much great work done with JavaScript in the form of "transpilers", I think a lot can be done in that area. I feel Zig is a crazy good idea: A brand new language that produces binaries that are 100% compatible with C linkers. If all goes well, in a few years, why would anyone use C over Zig? It seems like the future.

Lots of people think C++ is bat sh-t crazy complex (me too, but I still like the mental gymnastics!). What if there were different C++ dialects supported by transpilers that intentionally restricted features? I think kernel and embedded programmers would the primary audience.

Kubernetes may eventually underpin lots of higher level infrastructure, but most stuff currently isn't running atop Kubernetes. None of the higher level container-as-a-service offerings by the major cloud providers run on Kubernetes, for example. Nor does Heroku. And moreover a lot of people are still working with lower-level abstractions (VMs and auto-scaling groups) or no abstractions at all (pet servers).

I hear you about feeling dumb. I think some early decisions in the k8s ecosystem led to a lot of wasted time and effort, and this frustration.

YAML: significant whitespace is always unwelcome but YAML also introduces unexpected problems, like how it deals with booleans. For example, say you have this array:

    foo:
    - x
    - y
    - z

You might think this is an array of strings, but you'd be wrong.

It's also difficult to read through a YAML config and understand the parent of each key, especially in code reviews or just on GH.

I believe k8s life would have been easier with JSON configs, where it's impossible to confuse e.g. booleans for strings and where it's easier to understand the object's hierarchy.

Helm's use of gotpl: this choice exacerbates the problems with YAML. Now you're treating a structured language with a text template library. You have to spend energy thinking about indentation levels and how the values will be interpreted by both the templater and k8s.

I think helm would be less frustrating if they chose some templating library that made objects first class citizens. Where you can inject values at specific locations with one-liners or simple blocks of code (e.g. `ingress.spec.rules[0].append(yadda yadda)`)

I'm sure there was debate about these choices early on and I don't have any unique ideas here, so I don't want to be too critical. These are just a couple of pain points I've personally experienced.

Technically YAML is a superset of JSON - all valid JSON is valid YAML. So you could write all your configs in JSON and they'd work just fine.

This discussion from a couple of weeks ago suggests it isn't that simple: not all valid JSON is always valid YAML if your definition of superset requires that JSON would be parsed the same with a YAML parser as with a JSON parser.

If you're going to use JSON for config, it's better to use an actual JSON parser.

https://john-millikin.com/json-is-not-a-yaml-subset https://news.ycombinator.com/item?id=31406473

Thanks! That's really interesting.

You could mitigate some of the issues and get JSONs "strictness as a feature" by passing the document through e.g. `jq . $file` as a CI step, but I don't think that'd resolve the 1e2 issue. TBH I didn't know you could write numbers in JSON like that, so I imagine it'd be an issue that doesn't come up often. But it's disappointing that it wouldn't just work.

> Technically YAML is a superset of JSON

That's false. http://p3rl.org/JSON::XS#JSON-and-YAML

I'd like to amend my statement:

Technically YAML is (supposed to be) a superset of JSON - (almost) all valid JSON is valid YAML. So you could write all your configs in JSON and they'd (probably) work just fine (assuming you keep things relatively simple).

Is that yaml not an array of strings?

["x","y","z"]

I don't happen to have a YAML parser at hand, but I believe it's...

["x", True, "z"]

It's not, y is treated as boolean true.

Is that a k8s thing? Because when I load it in this page[1] it comes out as a string...

[1] https://yaml-online-parser.appspot.com/?yaml=foo%3A+%0A-+x%0...

In YAML 1.2 they fixed it (and also the Norway-problem of "no" meaning false) but many places are still using an older version.

for what its worth, and if im not mostaken, there is some support for JSON, ie. kubectl get deployment xyz -o json

JSON has a different set of problems, e.g. not having multi-line string support.

Similarly, we switched from self-managed k8s on EC2 to Fargate. Took about 2 months part-time with some consultants to get thing squared away.

Once we deployed, we ran into all sorts of SRE-issues. Turns out AWS sets all those “sane limits” that our own folks never did. Still hunting ghosts from the rollout 6 months ago.

Makes for good resume fodder, and makes me laugh at the prestigious titles and positions the folks who built this system went on to receive at big name firms.

Guess it is someone else’s management problem now. :shrug:

Azure is generally the worst (their new container offering looks better though)

If you try digital ocean k8s offering it's fairly straightforward.

Google cloud was the first to offer a decent k8s as a service, if I recall correctly. We didn't have DevOps back in 2015 and we were on GCloud.

Personally I don't pick k8s just because it's heavy to run and I don't want to waste machines (plain docker is good enough for a large part of what people actually need). Sometimes in a project when I can't figure something out with just docker, I just bite the bullet and install k0s.

And you probably didn't even get to TLS and or authenticated communication between containers, CI/CD, canary deployments, observability, monitoring etc.

What we need is a Next.js for Kubernetes. Something that delivers a full stack solution on top of base Kubernetes.

The core system is great, but we need to replace these DevOps with a framework or platform.

> What we need is a Next.js for Kubernetes. Something that delivers a full stack solution on top of base Kubernetes.

Doesn't Rancher fit this description? It's pretty resource-heavy though.

Rancher seems to now be a "multi-cloud container management platform".

- Deploys kubernetes clusters - includes Fleet for CI/CD - install apps with helm - ISTIO (awesome)

It feels like it is positioned for orgs with large needs. I'm looking for k8s for small nimble orgs.

I will look at Rancher more, thanks for reminding me of it!

Google's Anthos is also hard to describe now as they cover similar "everything" product features.

GCP now has Autopilot which lets you pay just for the cpu you use (no cluster management at all).

Anthos includes ISTIO which may at some point work on Autopilot. This would mean not having to fiddle with GKE Ingress (which I found unpleasant) and instead use the new standard Gateway.

I believe that eventually GKE Autopilot will offer running individual pods on GPU/TPU, pay as you go.

But when is this all as easy as using Next.js ?

I feel you. Took me 2 years of dabbling with k8s every once in a while on the side to finally "get" it. and if I stop looking for a few months, suddenly it already gained another set of features and deprecating some other.

Definitely a pain point I've seen in my past work as well. Even though spinning up a basic cluster has gotten easier with canned services like EKS, deploying and developing on the cluster is a major challenging for most developers without a higher level framework.

My cofounders and I are working on a solution to this at https://www.jetpack.io/. If you're interested in early access, we'd love your feedback!

Container instances are pretty good really, the app service for containers is pretty good too. I’ve been playing with k8s because that’s what everyone thinks they want and need to be able to speak to it, and use it when the time comes, but I’ve yet to run into a case where I really thought it was necessary, for the platform I work on (millions of users, big number of transactions per second).

Do you use ACI at that scale? It's interesting how Microsoft also promotes AKS rather than ACI. For Azure ML they even explicitly state that ACI should not be used for production purposes, which I find quite curious.

There are tools that let you convert a running docker/podman container to a kubernetes deployment manifest which you can then deploy with one line command to your cluster. Kubernetes is the new data center and I don’t expect anyone with <10 years of enterprise experience to master it. Certainly one can use it and deploy things to it if you work at a place where there are mature pipelines, but deploying your own cluster and all associated services plus networking plus security is the domain of experienced/senior engineers. How quickly one achieves that “status” is up to the individual. It took me 15 years before I could build an enterprise network from scratch. It took me two years to understand how to do the same with K8s.

We have a team to manage our Azure k8s. Blue/green clusters, switching traffic between clusters to be able to upgrade k8s, etc

Definitely a lot of work.

Literally a one-liner for AWS EKS:

eksctl create cluster --name mycluster --region us-west1 --with-oidc --fargate --external-dns-access --asg-access --full-ecr-access --alb-ingress-access

That is a quick and short line.

Now the fun starts:"Kubernetes Failure Stories"

https://k8s.af/

I’ve made a comment below, but long story short we’ve moved to Kubernetes running on Fargate and we don’t have downtime anymore.

Sure, one can break anything, but our anecdotal experience is we’re now focused on actually delivering code rather than fretting about node failures.

https://news.ycombinator.com/item?id=31581372

You can't run all types of workloads on Fargate. At least not yet.

That’s fair, for example I couldn’t manage to run clustered Redis. Something with EFS file system that Fargate nodes use.

You mean like AWS' EKS? That spares you kubeadm stuff in setup. Arguably complicated upgrade, because you're less familiar with what you have/need/rely on (their own upgrade docs point you to upstream changelogs/release notes etc.). You're still left with a kubernetes cluster to deploy stuff to, decide how stuff scales, etc., which is a lot of what is generally meant by 'DevOps' anyway?

The lower level infrastructure/platform/kubeadm type stuff isn't really 'Dev' related at all.

The reason why they provide it, is because everyone expects it. The reason everyone expects it, is because it was "cool" and an ecosystem grew around it. I use AWS ECS and it's really really good and easy to understand.

> 3 major clouds offer "canned" k8s services

...until you need to debug something somewhere in the enormous stack.

> Most people using Kubernetes should be using something simpler, but that's a separate problem.

This has been the biggest drain on my career. Everyone wants to be an "engineer" ready to handle every problem like its the next facebook. Like bruh, this service is going to get like 100 req/hour max and only when the sun is up - just **ing throw it on cloud run. We can tell the only thing you want to build is your resume.

It's a good point: tools made by giants for giants, such as Kubernetes, Bazel, etc, may not make sense for a smaller operation.

But what would you suggest in lieu of Kubernetes? What would save work for a shop which is not yet a giant but has already overgrown the capabilities of 2-3 manually managed boxes / instances?

I can think of several options. Management by Ansible / Saltstack / Chef can easily become a rabbit hole comparable to maintaining K8s's YAML. In your experience, does Nomad save SRE work at smaller scales? Does Terraform? CloudFormation?

If you only limit yourself to a subset of kubernetes's features, it's actually really great for small operations.

I run a small cluster for my side projects consisting of 3 nodes. 1 node is dedicated to run database containers and control plane node (it's also a worker node but strictly for database statefulsets only), and the other 2 as worker nodes, but one of them also run an NFS server that mounted as persistent volumes so every worker nodes can have access to it.

Self-healing? Internet-scale? I have on interest for them on my small cluster. I just want my apps to not go down while I'm updating them without writing complicated blue-green deployment scripts, ability to move around pods to other nodes when one got overloaded, and ability to add or remove nodes when needed without starting again from scratch. I basically treat it like docker-compose on steroid. So far it works really well.

A cloud service that can run containers directly, e.g. Amazon ACS, Google Cloud Run, etc.

> e.g. Amazon ACS

It's "Amazon ECS", and it works pretty well for standard fare CRUD web services, but more complex use cases quickly end up pulling in more and more AWS services (for example, if you need to run a cron job, you can use EventBridge to trigger an ECS task or just do it all with Lambda). This isn't dramatically worse--it's mostly just different. Kubernetes APIs are more consistent than in AWS, and Kubernetes tends to be more flexible and less limited than AWS. It's also much easier to extend Kubernetes or find a pre-existing extension than in AWS. But mostly it's not going to make or break your project one way or the other.

If you're just running CRUD web services, this is fine, but if you need to spin up a background job or do more complex orchestration then it can quickly become advantageous to

We run Nomad with about 7 to 9 client instances in prod. Quite happy with it.

Anybody who watched "Kubernetes: The Documentary" knows the answer: https://youtu.be/BE77h7dmoQU

Kubernetes only exists, because Google lost the Cloud Wars, and this was their Hail Mary pass.

And I might cynically offer it was "invented" to solve the problem of ex-googlers not having any useful immediately transferable skills as the Google internal tech stack had nothing in common with industry.

In 2005 Google search got into a sticky spot, where they were completely unable to deploy new versions of the search engine from the master branch of the code for a while, because there were performance regressions that nobody could find. Deployment and related things like "deploy a copy of the production configuration to these spare 2000 machines" were manual slogs. I was the noob on the team doing such important yet unfulfilling tasks as "backport this Python performance testing script to Python 1.x because our only working test environment doesn't have the same version of Python as production does". This was before borg aka kubernetes, and let me tell you, a whole bunch of stuff was dysfunctional and broken.

All this is not to say that Kubernetes is the right solution to your problem, or to almost anyone's problem. It was just an improvement to Google's infrastructure, for the sort of problems that Google had. For some people it makes sense... for you it might not.

Borg is not K8S, K8S is not borg.

Borg is far more advanced in its scaling abilities.

That's not really been my experience. The number of people who knew how to deploy software at Google was much smaller than the number of people writing software there. I was certainly the only person on my team who ever dealt with the nitty gritty of running stuff in production. That seems about like the industry standard; I'd say the average software engineer in industry, inside or outside Google, considers their work done when their code is merged.

At my first job outside of Google we used something called Convox, which was very similar to running things in production at Google. You triggered a package build from your workstation (!) and then adjusted production to pick that up. Very similar to mpm packages and GCL files. (The difference is that Google had some machinery to say "while the build might have been triggered from a workstation, all this code has actually been checked in and reviewed". Convox did not have that part, so yeah, you could just edit some files and push them to production without checking them in, which wasn't great. But when you are a 4 person development team, not the end of the world by any means.)

But the internal tech stack doesn't have much in common with Kubernetes, either.

That's actually my point.

If Kubernetes does indeed provide the best solution to provide scalability and availability, one can argue that this would result in a decreased demand for dev ops engineers, as they "would just have to use Kubernetes" and it would decrease their workload.

In reality this does not seem to be the case, that's why I asked.

> If Kubernetes does indeed provide the best solution to provide scalability and availability, one can argue that this would result in a decreased demand for dev ops engineers, as they "would just have to use Kubernetes".

I'd say it would result in either:

- the same scalability and availability with fewer DevOps engineers - better scalability and availability with a similar number or more DevOps engineers

In my experience, it's almost always the second case that happens. For example, a service would be moved from a few (virtualized or physical) servers that can only be scaled manually, to a k8s cluster with either autoscaling or at least scaling by changing a configuration file.

Right. Most companies aren't content to settle for doing the same thing they were doing but with fewer engineers when they realize they could be using those other engineers to automate even more things (via Kubernetes operators).

It could have been designed better to be easier to set up and use. Especially for ad-hoc or more casual use. If it is easy to use casually, people will use it more and they will learn how to use it faster. It is also easier to see why you would invest in learning to create more complex configurations.

Pretty much every piece of software I've written in the past decade that tends to have configs in production can also work with little or no config. Including clustered software that just exploits things that are widely available like Zeroconf to get a decent low effort cluster up and running. No, you probably won't be able to (or want to) use those features in production, but that's beside the point. The point is to lower the thresholds. And then keep aiming at lowering them wherever you can because asking other people to care about your software is seriously uncool. Other people will never care as much as you do.

It is normal for programmers to become defensive about software. Be it their own or software they really like. But it is far more productive to assume that when users think something is awkward, perhaps it is because it is awkward. And perhaps it could have been done better.

Nobody actually gives a crap what someone thinks Kubernetes was built for -- and what kind of rubbish experience they think is deeply justified by the goals.

It is either needlessly awkward to use or it isn't. And guess what: most people think it is awkward. And I seriously doubt it needs to be this awkward.

It still comes back to right tool for the job. There's a perception in the market that k8s is the right tool for running all compute. As someone recently told me "I've been interviewing cloud people for 2 weeks and all I can find is people that want to run k8s all day."

k8s is not the right tool for every job. Most companies are not at the scale where they need to worry about the problems that it's trying to solve. But it's a cargo cult - they see the blog posts about how moving to k8s solves a bunch of problems that come up as you scale and decide they need to be solving all of those problems also even though there are simpler solutions at their current scale.

There's a bunch of other platforms out there that are way more opinionated and less "awkward" but they don't have the buzz that k8s has.

Actually, I don't see it as being that much about scale, but more about moving complexity out of applications and about solving robustness challenges. Kubernetes offers a lot of things that are useful even at smaller scales. Which is why I'm not sure I think "but it's for scale" is a valid excuse.

Kubernetes ought at least to be the right tool for a wider range of compute task. And I think it could have been the right tool for a wider range of compute tasks.

When my system is running happily on a single VM with no real robustness issues then I don't really need to solve those robustness issues, and certainly not by bringing in a complex distributed platform to run it on. Or when my lack of robustness isn't costing me enough to make it worth it to spend the engineering cycles to adopt that platform. One way or another I have to be at a scale where it actually makes sense to make this investment vs just doing the simple/easy thing that's good enough for where I'm at.

There is a lot of ground between "runs on a single VM" to "runs on thousands of instances". In fact I would think most companies that deliver some online service fit into that category.

For instance, I don't regard one of our products that runs in three availability zones and has 3-4 instances per AZ as being "large scale". It is still a small system. And it doesn't run in multiple AZs for performance reasons but because we really need high availability.

We embedded discovery, failover and automatic cluster management in the server software itself. But it isn't really how we'd like to do it. But it is still less of a hassle than running K8S. (It also means that we can do that if you license our software to run it on-prem on pretty much most runtime environments, and that has its value, but again, this isn't functionality you want or should have to do yourself)

Agreed it's mostly undifferentiated heavy lifting still, and agreed it /should/ be easier. It previously took my team something like a year to get our infra all autoscaling - something I've found other teams aren't as willing to invest in if they're just running on a handful of instances.

At ~12 instances probably still in "pets aren't so bad" territory.

Most companies I've seen end up building some kind of framework around Kubernetes to make the developer experience tolerable. The threshold for getting started and getting a basic deployment up and running is way too high.

Of course, there's a huge cost to building your own framework as well... And it's easy to get wrong.

I started https://www.jetpack.io/ recently to try and build a better solution on top of Kubernetes. If you're interested in checking it out and giving us early feedback, you can signup on the website, or email us at `demo at jetpack.io`

Thanks, I will have a look!

My favorite part of this is that now K8S by itself isn't enough. People are laying "service mesh" and other things on top of it. And, yeah, I get the sales pitch for it, but at some point, it's too much.

The amount of benefits you receive is less than if you worked for the same benefits sans kubernetes. So it definitely saves a ton of work, but your latter point still stands that most of the time the benefits aren't always as... beneficial to some stages of development/scale so it's actually just adding work for the sake of it.

It does save work, consider what you would have to do to provide those benefits without a tool. You would have to monitor usage and spin up new instances manually for scalability. You would have to manually update instances when updates came out. You would have to manually handle failover and stand by instances for availability. Manually configure load balancers. etc.

Kubernetes is a great tool if you have scaling management problems.

That's an interesting second-order effect right? Before Kubernetes et al, it was so much work or so complex that few companies implemented those things. Kubernetes saves so much work, that now everyone is doing it and it feels like more work.

"Things" here mean different things for different people. For me, it's secrets/certs, better rolling deploys and infra as code.

Yeah, it is a premature optimization issue. Everyone is starting to implement scaling solutions before they have scaling problems.

The thread here full of people incorrecting each other about how to use Kubernetes seems pretty solid evidence for your thesis "it adds work".

> Kubernetes doesn't save work, it adds work. It saves, but for certain heavy scenarios like High Availability, autoscaling etc. I hope, one day we can see Kubernetes simplified, that will fit 95% of SOHO users.

It also by design locks you in one of the cloud providers in a quite deceptive way. There is no singular interface for networking, so you cannot really port your infrastructure from cloud to bare metal for instance. Bare metal also was heavily underdeveloped, so setting it up was fragile and fiddly and quite limited.

This is exactly what the "automation destroys jobs" argument gets wrong. When aggregate output becomes cheaper, aggregate demand increases.

What is this "something simpler"?

I have to manage like half a dozen docker images. K8s seems like a massive overkill, but managing by hand is rather error-prone.

Just use AWS ECS Fargate - probably much easier to manage/setup than your own k8s cluster or even ECS k8s route. Azure/GCP may provide something similar. I think any Google product is always needlessly complex that only solves their own use case and probably many other companies need not blindly adopt those unless absolutely necessary.

> It was designed for scalability and availability more than anything.

yet its really not great at the first, and I'm strongly suspicious about the latter claim. Its very simple to make boneheaded decisions about networking that makes thing fragile.

In terms of scale, you have a limit of 5k nodes. given how fucking chatty it is, it costs a lot to run loads of nodes.

Agreed. Kube makes the scaling out easier not the bootstrapping. An important difference.

I mean, obviously it was designed to reduce human work. A tool that adds work wouldn't be very useful, would it?

> A tool that adds work wouldn't be very useful, would it?

Sure could, if it provides benefits that outweight the additional work. Insurance companies will happily pay a couple more IT-specialists if it reduces the amount of cases they have to cover by an arbitrary percentage. Tools very much can serve other purposes than reducing the friction of human attention.

If you can containerize your infra and thereby mitigate threat vectors that may put your entire enterprise at risk if exploited, that's a good business call to make, even if the cost of your IT-department gets inflated by some margin.

A tool or technology that adds work is extremely useful.

More work means

- more employees needed

- more direct reports for managers

- more jobs for x technology

- more people learning x technology because of those jobs

- more StackOverflow questions for x

- more companies using x technology because it's "easy to hire for"

- more money to be made in teaching x

- more learning material for x

Seen before :-)

https://www-users.cs.york.ac.uk/susan/joke/cpp.htm

Warning to bashers...it's a fake interview...

I think the selling point here is getting k8s' benefits using other tools would result in more work than k8s

It reduces work, but imposes a high setup and operating cost. Once you get to the right scale then all the economics look correct. Large teams and organizations and absorb that cost more easily than many who are attempting to use k8s. Of course, being open source, anyone can try k8s, but the mileage is going to vary.

The key question is what are you comparing to k8s in terms of complexity?

Does it add work compared to setting up a VM with Docker and Watchtower? - for sure..

But does it add work compared to setting up something that gives you all the same benefits of k8s without using k8s? - imho definitely not.

> all the same benefits without using Kubernetes

That's the catch, if running your app on a manually setup VM is equivalent to running your app on Kubernetes then you don't understand what Kuberentes is or provide.

Today, we see someone who believes that just because a tool's goal is to reduce complexity, that's what it accomplished

> Have you ever tried setting up a Kubernetes cluster and deploying apps in it? Kubernetes doesn't save work

This is wrong, deploying on Kubernetes is easy and quick for most apps, you have one docker image one deployment spec and that's it.

https://kubernetes.io/docs/concepts/workloads/controllers/de...

And a service configuration, ingress, other networking, persistent volumes, a mechanism for updating deployed applications, management of the nodes (even with a managed service like EKS or other cloud solutions), logging, roles, security, etc. If you're a developer and all you have to worry about is one deployment spec, thank your devops team for making your life easier. Kubernetes is great for making the dev team's life easier, but someone did a lot of work to make it that easy for you.

Couldn't agree with this more. At my last company there was a fair bit of time put into making the deployment process for our microservices as simple as added a new YAML file to the deployment repo. That file pulled a custom chart, and as a dev you just needed to configure a few variables to get everything up an running. But if you were deploying something that couldn't use one of the pre-configured charts it was a bit more work, especially if you've never done it be hand before.

Probably 98% of the devs were blissfully unaware of that complexity that the charts abstracted, and it let them focus on the services they were writing. I wasn't one of them, and always made sure to thank the devops team for simplifying the day to day deployments whenever I had to deal with writing a custom one.

You can now do something similar with Bunnyshell.com

It Handles the devops jobs for dev teams.

Full disclosure, I work for Bunnyshell.

In the absence of any description of what “something similar” or “handles the devops jobs” actually means, this comes across as spam, not informative.

Sorry about that, I should have been more informative.

Bunnyshell makes it easy to create and manage environments. (EaaS - environments as a service)

You connect your k8s cluster(s) and git accounts/repos, it reads the docker-compose files and creates deployments on the cluster.

You don’t need to know or write Kubernetes manifests, those are created for you.

You also get auto updates and ephemeral/preview environments (when a PR is created against the branch of your env, Bunnyshell deploys a new env with the proposed changes).

You are not restricted to creating resources only on the cluster, you can use Terraform for any resource that is external to the cluster ( like S3 buckets, RDS instances, anything Terraform can handle).

Hope this helps,

THIS. In my homelab, I spent roughly a day cutting services over from Docker-Compose to Kubernetes. That day included writing the Helm templates for everything, bootstrapping three bare metal nodes with a hypervisor (Proxmox), clustering said hypervisor for HA on its own isolated network, making images and then installing K3OS onto six VMs across the three nodes (3+3 control plane/worker), installing and configuring persistent distributed storage (Longhorn) with backups to a ZFS pool as an NFS target (Debian VM configured via Packer + Ansible), configuring MetalLB, and smoke testing everything.

A day's work for one person to accomplish that isn't bad, IMO, but what that doesn't capture is the literal weeks I spent poring over documentation, trying things, running tests, learning what didn't work (Rook + Ceph is a nightmare), and so on. I went so far the day before the cutover as to recreate my homelab in Digital Ocean and run through the entire installation process.

Having services that magically work is hard. Having a golden path so you can create a new one with a few clicks is even harder.

This is still easier with Kubernetes than with other tools. Installing e.g. fluentd and Kibana is again a configuration / chart which you apply to the cluster. For monitoring and visualization again you have an operator / chart that you can apply to the cluster. Yes there is a lot of complexity and learning involved but overall I was still pretty amazed at how quickly I was able to get a cluster up to speed with the available tools.

Still need to maintain the cluster though, and boy does it require some fun maintenance. Kubernetes is more than just a distributed docker image scheduler, you also need to install and maintain basic dependencies like the pod networking implementation (a whole virtual IP space!!), DNS, loadbalancing, persistent volumes, monitoring, etc etc. Maybe your cloud provider sets it all up for you, but you're not going to escape having to debug and fix issues with that stuff, whether they be show stopping bugs or performance-impacting inconveniences like higher latency.

  > the pod networking implementation (a whole virtual IP space!!)

That part, at least, can be made simple: https://john-millikin.com/stateless-kubernetes-overlay-netwo...

  > DNS, loadbalancing, persistent volumes, monitoring, etc etc

None of that is part of Kubernetes, and you'll need it (or not) regardless of how you choose to handle process scheduling.

There's a sort of common idea that a "Kubernetes cluster" is an entire self-contained PaaS, and that (for example) monitoring in Kubernetes is somehow fundamentally different from what came before. It's easy to fall into the trap of creating an internal clone of Heroku, but Kubernetes itself doesn't require you to do so and it can be a lot faster to just run Nagios (etc).

> None of that is part of Kubernetes

Well, load balancing is part of Kubernetes (except that it does not provide a sane implementation), and tbh having to debug strange failures caused by seemingly-innocent Service configuration is my least-favorite part of Kubernetes.

I agree with you on other points.

DNS too

You'd weep if you see an actual deployment yaml. What documentation provides is something minimal, and that's very long for something minimal.

Also, self-healing can create interesting problems which are fun to trace, debug and understand.

"Before you begin, make sure your Kubernetes cluster is up and running."

"And if it's not, please roll-back your systems and install it from scratch."

I'm expecting you are getting downvoted because of your assumptive and dismissive statement. The type of deployment you are suggesting is overkill to apply k8s as there is minimal to no orchestration.

Also because they are just outright missing the point of the post they were replying to. Yes, deploying thing on kubernetes is pretty simple. Deploying kubernetes itself is definitely not very simple.

People are confused, you should most likely not opperate k8s yourself if you don't know or does not have a team for it, nowdays managed k8s are easy to use and operate, with 3 clicks you get a cluster up and running with sane defaults.

Not to sound trite but I feel like your continued pressing of this issue evokes the entire "Do you even lift, bro?" meme. There are PaaS that can get you up in 3 clicks, but now you are in bed with k8s which is critical for the infrastructure. I think it's quite necessary to definitely understand the inside-and-out of such a vital component for my infra.

Either you’re so good with k8s and are clueless about how rare that expertise is, or more likely, you’re so clueless of how complex and fickle k8s can be if configured wrong but you’re not even aware that you don’t know.

I’ve seen actually decent engineers (maybe they’re not decent?) bring down prod because they accidentally Kubectl deploy’ed from their command line.