Probably even better is to ship a controller and a CRD for the config.
Doing it that means you ship a schema for the parameters of the config, and that you have code that can handle complexities of upgrades/migrations that tools like kustomize and helm struggle or fail at altogether.
But how do you package the controller + CRD? The two leading choices are `kubectl apply -f` on a url or Helm and as soon as you need any customization to the controller itself you end up needing a tool like helm.
I love how in Helm I can just look at the templates and figure out what values I need to change to get what I want, and I love each environment only needing a single values file to see all the customizations for it.
People complain about it being a template language, but that is exactly what you need!
Isn't the "weird syntax" just either Yaml files or just JSON Patches, which is a pretty easy standard?
>having to look at a bunch of different files to see what is going on
I consider that a feature, not a bug. prod/larger-memory-request.yaml makes it much easier for me to see what goes into deploying the prod environment instead of for example the test environment.
A template is much easier to read. I had zero experience with go templating, but was able to figure out what it all meant just by looking at the templates... they still looked like kubernetes resources
As for looking at a bunch of different files, if you like having a "larger-memory-request" file, you can still do that with helm... you can use as many values files as you want, just include them in precedence order. You can have your "larger-memory-request" values file.
Keep your customizations flat and compile them to yaml+grep to find out what’s getting overridden and where.
Helm's problem is that `values.yml` is basically the API and every helm chart provides its own (often incomplete) interface with poorly documented defaults. Some of those can spread over 3k+ lines and it's utterly overwhelming to figure out what to do with that.
templates are only good if your templates can remain simple and do not need to expose most of the output fields. my experience developing a chart for wide distribution has been very much that your templates will not remain simple (and will turn into an incomprehensible mess, since you'll need them to handle tasks templates are fundamentally poorly suited for) and that there is always someone, somewhere, that needs some particular resource field exposed in values.yaml. the
> As a result, the number of possibilities for configuration is often unreasonably large and complicated, mimicking the actual resources they want to create, but without any schema validation!
bit from the op is incredibly true. values.yaml grows, over time, to have every field in the objects it generates, just organized differently, without validation, and with extra complicated relationships with other settings
kustomize allowing you to provide a base set of resources that users can apply their own patches to avoids that config surface bloat problem entirely
The challenge has historically been that controllers are a lot harder to write, but I think that story has improved over the years
and all the customizations just end up being yaml merges from a configmap string or CRD if you're lucky
Maybe it's just us, but our operations team puts pretty hard restrictions on how we're allowed to talk to the K8s API directly. We can turn a regular Deployment around as fast as we can write it, but if we needed a controller and CRD update it'd take us like three days minimum. (Which, I even sort of understand because I see the absolute garbage code in some of the operators the other teams are asking them to deploy...)
If a product team sees itself needing to deploy an operator, it's likely the platform is subpar and should be improved, or the product team is overengineering something and could do with rethinking their approach.
As in most cases, a conversation with your platform/ops/devops/sre/infra team should help clarify things.
Internal applications don't have this problem, so you can easily keep your chart interface simple and scoped to the different ways you need to deploy your own stack.
With Kustomize, you just publish the base manifests and users can override whatever they want. Not that Kustomize doesn't have its own set of problems.
https://github.com/bjw-s/helm-charts/tree/main/charts/librar...
Full disclosure: our startup is trying to build a tool where you don't have to pick, so trying to test the hypothesis
Most of the problems associated with Helm go away if you stop using Kubernetes.
So yeah we do this. Our product runs in 3 public clouds (working on 5), single VM, etc. and our customers install it themselves. We're helm plus Replicated. AMA.
Also curious what issues you've seen replicated prevent.
Replicated saved work by handling a configuration gui for the end user, licensing/ entitlements, support bundle collection, private image proxy, things like that we didn't want to deal with.
What I found is that when doing such review, you realize 99 of the template variables are not relevant for you and the one place you need to template is missing a value. Just extracting the rendered manifests and modify them by hand from there becomes more maintainable. Like you say, there is a very limited set of common denominators.
For smaller orgs, just running a single container and increasing the Node size takes you a very long way. That doesn’t need helm.
So Linux is a mess? And we should rethink how rpm and deb work?
Or all Deb issues go away if you stop using Linux?
People forget that Helm is a package manager first and foremost (the only one for Kubernetes). It also happens to include a templating mechanism. The templating part has its issues, but until we find another package manager, I don't see Helm going anywhere.
Linux is indeed a mess, yes. RPM and Deb are both awful formats stuck in the 90s, with even worse package managers on the top. Even with the legacy of those, installing a package does not involve templating a whitespace-sensitive language with a mediocre template language.
> Or all Deb issues go away if you stop using Linux?
Never had any issues with debs on FreeBSD. Or NixOS if one likes the Linux kernel.
The idea that helm is even a package manager is fanciful at best, in any case.
The front page of helm.sh literally says "The package manager for Kubernetes". If it was advertised as "the best templating engine for K8s" or something similar I would agree with you.
People try to abuse Helm.sh as a fancy templating engine. And the testament to that is all the articles "Helm vs Kustomize vs JSonnet vs ..."
I take your image as the FROM for my own Dockerfile, tweak a few settings, maybe alter the CMD, and then run my image instead of trying to do some sort of ad absurdum variation on a Twelve Factor App.
I would say, helm is at it's best when you need to _publicly distribute_ a complex application, AND the majority of the users on the receiving end don't care about the complexity.
You don't need Helm if your manifests are simple. But when your manifests become complex, helm will make it even more complex by turning each field in the plain k8s manifests into a toggle in a values.yaml file.
yes, it's nice to do env-var-substitution, and --set is not that dumb.
For internal applications, it's in an awkward place of being both too complex and too simple, and in a lot of cases what you really want to do is just write your own operator for the complex cases and use kustomize for the simple cases.
Most of the problems with updating and installing helm charts go away if you manage it with something like argocd to automatically keep everything up to date.