But how do you package the controller + CRD? The two leading choices are `kubectl apply -f` on a url or Helm and as soon as you need any customization to the controller itself you end up needing a tool like helm.
I love how in Helm I can just look at the templates and figure out what values I need to change to get what I want, and I love each environment only needing a single values file to see all the customizations for it.
People complain about it being a template language, but that is exactly what you need!
Isn't the "weird syntax" just either Yaml files or just JSON Patches, which is a pretty easy standard?
>having to look at a bunch of different files to see what is going on
I consider that a feature, not a bug. prod/larger-memory-request.yaml makes it much easier for me to see what goes into deploying the prod environment instead of for example the test environment.
A template is much easier to read. I had zero experience with go templating, but was able to figure out what it all meant just by looking at the templates... they still looked like kubernetes resources
As for looking at a bunch of different files, if you like having a "larger-memory-request" file, you can still do that with helm... you can use as many values files as you want, just include them in precedence order. You can have your "larger-memory-request" values file.
Keep your customizations flat and compile them to yaml+grep to find out what’s getting overridden and where.
Helm's problem is that `values.yml` is basically the API and every helm chart provides its own (often incomplete) interface with poorly documented defaults. Some of those can spread over 3k+ lines and it's utterly overwhelming to figure out what to do with that.
templates are only good if your templates can remain simple and do not need to expose most of the output fields. my experience developing a chart for wide distribution has been very much that your templates will not remain simple (and will turn into an incomprehensible mess, since you'll need them to handle tasks templates are fundamentally poorly suited for) and that there is always someone, somewhere, that needs some particular resource field exposed in values.yaml. the
> As a result, the number of possibilities for configuration is often unreasonably large and complicated, mimicking the actual resources they want to create, but without any schema validation!
bit from the op is incredibly true. values.yaml grows, over time, to have every field in the objects it generates, just organized differently, without validation, and with extra complicated relationships with other settings
kustomize allowing you to provide a base set of resources that users can apply their own patches to avoids that config surface bloat problem entirely
The challenge has historically been that controllers are a lot harder to write, but I think that story has improved over the years
and all the customizations just end up being yaml merges from a configmap string or CRD if you're lucky
Maybe it's just us, but our operations team puts pretty hard restrictions on how we're allowed to talk to the K8s API directly. We can turn a regular Deployment around as fast as we can write it, but if we needed a controller and CRD update it'd take us like three days minimum. (Which, I even sort of understand because I see the absolute garbage code in some of the operators the other teams are asking them to deploy...)
If a product team sees itself needing to deploy an operator, it's likely the platform is subpar and should be improved, or the product team is overengineering something and could do with rethinking their approach.
As in most cases, a conversation with your platform/ops/devops/sre/infra team should help clarify things.
Probably even better is to ship a controller and a CRD for the config.
Doing it that means you ship a schema for the parameters of the config, and that you have code that can handle complexities of upgrades/migrations that tools like kustomize and helm struggle or fail at altogether.