> You can easily write tools that generate json/yaml for you based on your needs. For most applications you're deploying you'll need:

Can you with yaml? Helm tries, but isn't that great, jinja ( depending on how its done ( jinja in yaml as in ansible or jinja to yaml as in SaltStack)) does an ok job, but fundamentally managing a language which uses spaces for logic with code is hard, and certainly YAML wasn't created to be managed that way.

JSON i agree, you can easily convert your language of choice's data structures to it. But do you have to? It adds another layer of complexity, and you can have unexpected bugs ( since you have a logic layer that creates JSON, which could be wrong due to a logical error).

HCL is a DSL that needs learning and few use outside of Hashicorp products, but IMHO it's not that hard ( docs are great, tutorials are plenty) and it's worthwhile to have data+logic in the same human and machine readable layer.

In any case, Nomad accepts JSON for everything ( mostly for when non-humans interact with it).

Helm, in my personal opinion, is one of the biggest mistakes of the kubernetes ecosystem. Not embracing Jsonnet, Cue, Dhall, or even an in-language DSL like Pulumi has set the usability of kube back quite far.

Helm isn't that connected to the broader Kubernetes system. It was popular in the early days when Kubernetes knowledge was sparse and things were still being figured out. None of the official Kubernetes docs really mention or describe using helm, and I see more getting started tutorials are starting to avoid it too. The official kubectl tool supports a -k mode which applies Kustomize logic and I'd argue Kustomize is probably the official way forward with app deployments by the Kubernetes project.

Yet, the embedded kustomize version is years behind standalone.

https://github.com/kubernetes-sigs/kustomize/issues/1500

$DAY_JOB replaced helm with custom tool that does equivalent of kubectl apply by uses Jsonnet as templating engine.

I intend to write an OSS version of it because Jsonnet really is probably the best fit here (though Cue/Dhall are interesting I think Jsonnet is easiest to get others to adopt).

I also did something like this a few years ago (with Go templating instead of Jsonnet), but we recently replaced it with Kustomize (https://kustomize.io/) as it worked quite well for the basic use case of getting shared Kubernetes configs to deploy code to different environments at different times.

The fact that trivial things like ingresses don’t have transformers is a PITA.

I've done the same in the past. The important thing is making it so it's easy to build up abstractions and to work in a sensible deployment flow.

kubectl is close but doesn't understand secrets/variables, the concept of clusters, or changing image tags programmatically. It's difficult to get right.

> kubectl is close but doesn't understand secrets/variables, the concept of clusters, or changing image tags programmatically. It's difficult to get right.

Isn't that the problem kustomize (which kubectl now has native support for applying) is trying to solve? I ask that honestly, because I haven't used kustomize in anger in order to compare its workflow to that of helm

I did the full gambit of helm 1/2 tiller/tiller-less and kustomize. Personally kustomize is only marginally better than helm.

I need to check out kubecfg, I haven't used it yet.

My mistake. I meant kubecfg which is built off of jsonnet.

Yeah this is all stuff this tool understood, multiple clusters, managing kube contexts, external secret storage or k8s secrets, native git integration, etc.

Definitely agree that it's difficult to get right but the authors of the tool definitely struck a really good balance.

Hopefully if I implement it myself I can do their original design and implementation justice.

No need to build it, there's already kubecfg: https://github.com/bitnami/kubecfg

We use this heavily at InfluxData to manage multiple clusters in multiple regions of multiple clouds. One of the maintainers of kubecfg recently joined InfluxData as well.

A team at my workplace uses Jsonnet for this problem and it seems to work very well. It's a nicely human-readable DSL for generating JSON in a way that respects JSON structures - you can construct lists of objects (deployments, services, etc.) by looping over templates and composing modifications.

That seems a lot better than either SaltStack's use of Jinja (using text-based templating on a structured format) or Ansible's (using YAML as syntax for an imperative programming language that has Jinja-evaluation functionality).

HCL is JSON compatible, and Nomad can load JSON natively.

I assumed that was a possibility but that still means "HCL" isn't a benefit since kubectl can load json from the CLI and from apiserver.

It can only load json with the api, the nomad CLI can't launch json jobs.

Yes it can.

"For machine-friendliness, Nomad can also read JSON-equivalent configurations. In general, we recommend using the HCL syntax."

https://www.nomadproject.io/docs/job-specification#job-speci...

But who would use the Nomad CLI for serious work? I use the Terraform provider for Nomad to launch all my jobs.

You may not use it for automation. But you usually do much exploratory work before doing the automation. That's where the various CLIs for all kinds of things come in really handy.

Me? It's handy to be able to dump -> modify -> load jobs when I'm poking at things. No one said anything about serious work, though, they said "load json natively".

A thousand times this.

With Kubernetes, you can even use one of the many native clients (and/or protobuf) which makes converting from something else a breeze, avoiding the JSON/YAML entirely. You get static typing, you can write tests, etc.

Well, it's same thing with Nomad - there are native SDKs, protobuf and a classical JSON REST API for everything.

Unless you start messing with Operators (which are hot right now) for desired state configuration within your cluster, which can, theoretically, be written in anything, but realistically tend to be written in Golang.

I think operators are great and true to the k8s spirit. You describe your desired state (e.g. a postgres database) according to a CRD. And the operator will make your yaml definition become reality. I think the declarative approach is a big improvement over the imperative step-by-step setup. I rather like to describe how I would like things to be, instead of having to create the desired state myself.

May be. But operators are a relative pain to develop. I prefer the HCL approach. You can have modules instead of operators, and while HCL overall might be painful, it's less painful, IMO, than constantly changing operators. My team spends a significant amount of time chasing deprecations of operators and associated helm charts.

To be clear: I like operators! I was just responding to the parent post; JSON/YAML aren’t the only ways to express things in the Kubeverse.

Yep, exactly. KubeDB is a great example of this.

HCL alone is a good reason to prefer Nomad over any YAML mess that exists.