This. My first thought when I saw the indentation hack was "it can't be serious, production-ready software".

My take on this is as follows.

If you have a simple use case, write your K8s manifests directly.

If you have a complex use case, Helm is often more pain than it's worth. Use alternatives, for example Jsonnet[0] with kubecfg[1]. Or emit manifests from your language of choice. Just don't use Helm.

[0]: https://jsonnet.org/ [1]: https://github.com/kubecfg/kubecfg

Terraform's design is that of a half-baked configuration management program. It's not even declarative programming, because declarative programming requires the program actually try to achieve the desired end state. Terraform won't do that. It will try to create or destroy something based on what it thinks the world looks like, rather than what the world actually looks like. If anything is inconsistent, it simply dies, rather than try to fix or work around the inconsistency. Which is the opposite of what configuration management is supposed to do. It's supposed to fix the system state, not run away crying. In many ways, Puppet, Chef or even (ugh) Ansible would be better than Terraform at handling infrastructure state.

For example, if you tell Terraform you want an S3 bucket, and it tries to create one, but it already exists, Terraform dies. In that case they want you to manually use an 'import' command to import the existing S3 bucket (which btw is a different syntax for every single resource). Why doesn't it automatically import resources? Hashicorp's "philosophy" is that automation is bad and why would we want a computer to make a human's life easier when we can make the human serve the computer? This basic but critical feature is so important that Google created an entire project dedicated to it, called Terraformer.

If you write some Terraform HCL, you may be able to create your resources with it. But it's also very likely Terraform will not be able to change or destroy those same resources. Every provider has its own rules about what you can do with a resource, and none of those providers are required to apply those rules throughout the resource's lifecycle. So you deploy something to prod, and then need to change it, only to find out that change is impossible, half way through a terraform apply in production. Leaving you with broken production infrastructure. Many provider resources also have rules that require you to add a Terraform lifecycle policy, or it will be impossible to create, destroy, rename, etc a resource. Rather than make those lifecycle policies the default for a given resource, Terraform requires you to know in advance that that resource requires that lifecycle policy and to add it to every instance of that resource. But it will cheerfully create resources without the policy, with a big gaping hole in the floor waiting for your infra to fall through when you next make a change.

This is a very brief list of the insane stupidity that is Terraform. A blindfolded configuration management tool that can't handle changes and doesn't like automation, validation, or sane defaults.

terraform is fully declarative; if I want a thing of the XYZ kind, I ask terraform «give me a thing of the XYZ kind and I don't care about how you will get it for me». Declarative programming operates with intentions without focusing on details of how the intention will actually be expressed; «give me XYZ» is an example of such an intention.

Declarative programming and the «desired end state» are orthogonal, and the desired end state may or may not even be the goal of a declarative program. For example, expert systems are all declarative, and they don't deal with «worlds» and can be «entered into» from «anywhere». And, since their «worlds» keep on constantly changing by introducing or retracting facts, at best it is only possible to think about a snapshot of such a world at a given moment in time.

> Terraform won't do that. It will try to create or destroy something based on what it thinks the world looks like, rather than what the world actually looks like.

Can you unpack that? terraform keeps the current state of «world» as a dependency graph in a state file and compares it with intentions expressed in .tf file(s) that under the hood translate into one or more new or changed graph nodes. If there are differences in two dependency graphs, it modifies the current in-memory dependency graph, applies changes to the infrastructure and incrementally updates the state file to reflect the dependency graph update progression in the infrastructure.

terraform is not a sentient being, therefore it can't solve philosophical problems nor can it reason about anything that goes beyond its state file. It is, essentially, a «make» for the infrastructure management with each provider supplying built-in build rules.

> For example, if you tell Terraform you want an S3 bucket, and it tries to create one, but it already exists, Terraform dies. In that case they want you to manually use an 'import' command to import the existing S3 bucket (which btw is a different syntax for every single resource). Why doesn't it automatically import resources?

Why? Because it is a feature.

If a S3 bucket already exists, it may mean of the following:

  1. It is already managed by another terraform project;
  2. It has another owner who might have created it using a different tool that manages the S3 bucket lifecycle differently;
  3. A coding mistake, e.g. a .tf file has been copied and pasted into;
  4. Something else.

Same about helm. I didn't write a single helm yet. But I like packages that I can helm install and I probably would avoid software without first-class helm. Helm is good for users. I can install it, upgrade it, configure it with values. Awesome.

Especially since k8s doesn't even use "text" for the format, it's just happens that JSON is the serialization, use datastructures and serialize them dammit :/

Helm just has too much auto-magic and remove's one of k8s best features, the git-blame / git-diff for Infrastructure.

    helm diff upgrade <release> <repo> --namespace <namespace>

If your team is big enough you can just write your own configs, but that takes a lot of time and often quite a bit of knowledge about the relevant service.

Rollbacks and resource cleanup is not to be underestimated either if you are not getting that from other tools like Argo.

Note: You can still use the Git-centric approach by generating configuration via `helm template`.

I've been using kustomize to deploy some things with ArgoCD and it's so much easier. Now, I'm trying to never use helm for 3rd party.

However, for your internal things, Helm is hard to replace. It's easy to start a chart that is capable of deploying most of your internal services, but maintaining it is a nightmare.

Actually, using helm, as in `helm` the binary, directly sounds bonkers to me and I wouldn't wish this upon anyone.

Absolutely would not write helm charts from scratch for normal deployments, and if I got these apps in a better format than helm, I’d probably drop it immediately.

Amen. I got turned off from k8s following a tutorial that used Helm. I ran it and watched mysteries and line noise scroll past and walked away for a year. I thought "no, I will never put this squirming mass of tapeworms in front of anyone."

Then I took up with k3s and got underway.

Yeah, I think helm will be the death of Kubernetes. Some other workload management tool will come out, it will have a computer-science-inspired templating system (think Lisp macros, not C #defines) that is also easy to use, and the increased maintainability will be a breath of fresh air to people in helm hell and they'll switch over, even if the workload management fundamentals are worse.

It is a shame that ksonnet was abandoned. jsonnet is a very pleasant programming language for problems of similar scope. I think that people have to see some adoption or they give up; so Helm stays motivated to continue existing despite the fact that the design is fundamentally flawed, while alternatives give up when nobody uses them. If you're looking for a lesson in "worse is better", look no farther from helm. Easy things are hard, but everything is possible. That turns out to be important.

I also liked kustomize a lot, but it definitely sacrificed features for security and a clean overall design posture. I don't really know what it's missing, but it must be missing something. (I use kustomize for everything, but it seems rare in the industry. And I don't use it for complicated things like "make a namespace and app deployment for every PR"; I think to manage things like that it's missing features people want. I say just run your development branch on your workstation and commit it when it works. Running the app you write shouldn't be rocket science, and so a computer program that configures it shouldn't be necessary. The industry definitely disagrees with me there, though.)

One of the biggest problems I have with helm is that it's not easy to make local modifications to charts. That needs to be a first class feature, and something everyone knows how to use. As a vendor who ships a helm chart, I feel like almost all of my work these days is hearing from users "our k8s security team requires that every manifest set field X", and then I have to go add that for them and release it. If kustomize were the default, they'd just add that themselves instead of asking. But hey, opportunity to provide good customer service, I guess. (Luckily, most of the requests are compatible with each other. At one point we ran as root because that's the default, a customer required running as a user, and now everyone can run a locked down version; nobody NEEDED to run as root. So these requests generally improve the product, but it's a little tedious to make them ask.)

https://kubectl.docs.kubernetes.io/references/kustomize/kust...

With rendering, you use something that is aware of yaml, and you feed it data and it outputs valid yaml.

I use my preferred language to emit manifests and built tooling around it. It does not template, and instead generates the manifest by transforming a data structure (hash) into json. I can then use whatever language feature or library I need to generate that data structure. This is much easier to work with than trying to edit template files.

I don't need to save the output of these because when I use the tooling, it generates and feeds that directly into kubectl. There's also a diff tool that works most of the time that lets me see what I am about to change before changing it.

In fact, I ended up adding a wrapper for Helm so that I can have all the various --set and values, source chart, chart repo, chart version pinning all versioned in git, and use the same tooling to apply those with the flags to install things idempotently turned on by default. It sets that up and calls helm instead of kubectl.

That tooling I wrote is open-source. You never heard of it because (1) I don't have the time and energy to document it or promote it, and (2) it only makes sense for teams that use that particular language. Helm is language-agnostic.

EDIT: and reading this thread, someone mentioned Kustomize. If that had been around in '16, I might not have written my own tool. It looks like it also treat YAML manifests as something to be transformed rather than templates to be rendered.

For better or for worse Helm is the defacto standard for deploying into k8s. Kustomizing toy apps or simple things may work but I have yet to see a large production stack use anything but Helm.

Definitely check out kustomize, it's not a toy and it can easily handle one main manifest with specializations for each unique deployment environment. It's a very nice model of override and patching config instead of some insane monster template and yaml generation like helm.

Large clusters, about 80 service teams and many hundreds of apps.

We (the platform team) managed roughly 30-40 services or controllers, including cert-manager, our own CRD for app deployments, ElasticSearch, Victoria Metrics, Grafana and others.

It was (still is, only I'm sadly not there anymore!) a high performing team and organisation with a lot going on.

We reviewed the decision to not use Helm many times and every time it was pretty much unanimous that we had a better solution than the opaque and awkward templating.

Taken straight from their docs.

I'm not using Makefile either.

Helm is a kind of least-common denominator software that's better than doing nothing, but the template paradigm leaves a lot to be desired. It's main advantage is being language-agnostic.

The entire existence of helm is superfluous. The features it provides are already part of Kubernetes. It was created (by /) for people who understand the package manager metaphor but don't understand how Kubernetes fundamentally works.

The metaphor is wrong! You are not installing applications on some new kind of OS. Using helm is like injecting untracked application code into a running application.

At best helm is just adding unnecessary complexity by re-framing existing features as features that helm adds.

In reality helm's obfuscation leads to an impenetrable mess of black boxes, that explodes the cost and complexity of managing a k8s cluster.

First off if you are packaging and/or publishing apps using helm charts, stop it!

There is purpose to the standardization of the Kubernetes configuration language. Just publish the damn configuration with a bit of documentation.... You know just like every other open source library! You're building the configuration for the helm chart anyway, so just publish that. It's a lot less work then creating stupid helm charts that serve no purpose but to obfuscate.

Here is your new no helm instructions: We've stopped using helm to deploy our app. To use our recommended deployment, clone this repo of yaml configs. Copy these files into your kubernetes config repo, change any options you want (see inline comments). Apply with `kubectl apply -f .`, or let your continuous deployment system deploy it on commit.

What have you lost?

(Or, hell, helm but with no templating whatsoever).

kustomize? raw manifests? "sed s/MY_PER_DEPLOYMENT_VALUE/whatever/" < manifest.yaml | kubectl apply -f -"? "jsonnet whatever.jsonnet | kubectl apply -f -"?

But yeah, a lot of people think Helm is mandatory and ask for it by name, and if you don't provide a chart they won't use your thing.

kubectl apply -f $MANIFESTS

> you're still going to be using Helm (if not indirectly via Argo) and you will inevitably need to template things that need to be dynamically configured at render-time.

Use Kustomize for dynamic vars and keep it at a minimum. Templating is the root of all evil.

Helm mostly adds unnecessary complexity and obscurity. Sure it's faster to deploy supporting services with it, but how often do you actually need to do that anyway ? The time you're initially gaining by using Helm might generate an order of magnitude more time in maintenance later on because you've created a situation where the underlying mechanics are both hidden and unknown from you.

How do you configure it? Like you're installing new version, do you go over manifests and edit those by hand over and over every update? Do you maintain some sed scripts?

helm is awesome because it separates configuration.

But for Kubernetes manifests specifically, the right tool for the job is Kustomize[1] (which in that case does what Helm does for you, keeping dynamic variables separate). It ships with kubectl and I'm a big believer in using default tools when possible.

> Like you're installing new version, do you go over manifests and edit those by hand over and over every update?

I check the patch notes, diff the configuration files to see if anything new popped up, do the required changes if necessary, jump the version number and deploy.

It sounds laborious but it's really not that much work most of the time, and more importantly it forces you to have a good sense for what and how everything works. Plus it allows you to have a completely transparent, readable environment. Both are important for keeping things running in a production environment. Otherwise you might find yourself debugging incomprehensible systems you've never paid attention to in the middle of the night with 2 hours left before traffic starts coming in.

[0]: https://github.com/mikefarah/yq

[1]: https://kustomize.io

Something simple like GitLab probably would be impossible to understand for a simple person.

That blog post doesn't alleviate any issues one might have using the traditional Go Templates + Helm.

Gives you a Typescript API for generating resources. The difference is that these templates are semantic (via the type system) not just syntactic as in text templates. Relatedly, they also factor and compose better because functions and imports.

The one thing it misses is a way to easily delete resources.