Draft, Gitkube, Helm, Ksonnet, Metaparticle, Skaffold – Comparison of K8s Tools

Re:Draft, Gitkube, Helm, Ksonnet, Metaparticle, Skaffold – C... by _5meq on Friday March 30, 2018 @11:38AM (#16716974)

I'm going to use this opportunity to get up on my soap-box and talk about helm, the "recommended" way to install kubernetes packages.

Helm is not a useful abstraction, and needs to change in a BIG way before I consider using it over k8s_raw and ansible.

My main complaint with helm is that it doesn't allow one to develop decent abstractions over the core kubernetes resources. IMO it needs to move away from `gotpl` and implement the ability to use more sophisticated templating libraries.

I think this block of text elucidates my point:

```

metadata:

  name: {{ template "drupal.fullname" . }}

  labels:

    app: {{ template "drupal.fullname" . }}

    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"

    release: "{{ .Release.Name }}"

    heritage: "{{ .Release.Service }}"

```

This block of code is in every single resource in every single helm chart. And it's one of MANY similar blocks of code that appear in basically every helm chart. I do not particularly enjoy writing the same massive YAML file over and over again.

Configuration management with helm is difficult. It would be nice to be able to declare "required" fields within an application, and enforce those fields with an error message if they are not provided. Currently, there is no way to do this other than having a "required" template that iterates through your "requiredFields" field and calls `{{ .Values.fieldName | required }}`

Interacting with the helm API server is difficult, which makes integrating helm with other configuration tools difficult. Specifically Ansible. This is also partly k8's fault ( And my fault, I suppose, because I could write it myself... ), because port-forwarding into the k8 server isn't implemented in any languages other than GoLang.

[ Parent | on HN ]

Re:Draft, Gitkube, Helm, Ksonnet, Metaparticle, Skaffold – C... by atombender on Friday March 30, 2018 @12:00PM (#16717217)

Agreed. Unstructured text templating to generate structures YAML is a terrible idea, which was previously shown with Ansible and Salt. (Salt provides a way to generate its YAML declaratively using Python, which seems to me a much better idea. Not sure about Ansible?)

I think Jsonnet is an intriguing idea here. Jsonnet is not well known, but apparently popular inside Google. It has variables and iteration, and allows you to cobble together structures by merging them, iterating over them, and so on, and the output is JSON, which happens to be what the Kubernetes API uses anyway. (There is a system that uses Jsonnet with Kubernetes, called Ksonnet, but that tool seems bafflingly overengineered to me.)

I've been dwelling on another idea: A system where you simply push all your templates -- what Helms calls a chart -- to Kubernetes as a single CRD (e.g. "kind: Template"). The templates reference variables, which you then push separately as another CRD ("kind: Vars"). Then the final component is a controller that listens to changes to templates and variables, and whenever one changes, expands the templates, compares them with the current manifests, and applies any differences.

To achieve controlled rollouts and rollbacks, you have a system on top that's similar to Helm, but it can be completely separate. You version the templates/vars as releases, not the underlying manifests generated by them.

[ Parent | on HN ]

Re:Draft, Gitkube, Helm, Ksonnet, Metaparticle, Skaffold – C... by wstrange on Friday March 30, 2018 @12:54PM (#16717741)

Helm Version 3 will adopt CRDs for release and application state. There will no longer be a server side component (tiller).

It should be much easier to integrate other tools or templating systems with V3.

https://github.com/kubernetes-helm/community/blob/master/hel...

[ Parent | on HN ]

Re:Draft, Gitkube, Helm, Ksonnet, Metaparticle, Skaffold – C... by lobster_johnson on Friday March 30, 2018 @02:45PM (#16718734)

Interesting, but I never saw Tiller as an issue. Quite the opposite, I want less client and more server.

[ Parent | on HN ]

Re:Draft, Gitkube, Helm, Ksonnet, Metaparticle, Skaffold – C... by _5meq on Friday March 30, 2018 @12:17PM (#16717389)

I've used both ansible and salt extensively.

Ansible is similar to salt, but is imperative as opposed to declarative. I've found I prefer Ansible for the scale I work at. Salt has a better eventing and provisioning story though, and I can see the value in declaring your environment's "state" using pillars and states. There's just a lot more to Salt and I like Ansible's simplicity.

In Ansible/k8s_raw, you can declare your k8 resources in Ansible as real yaml, or you can template YAML using the jinja2 templating language, or you can generate YAML using a python library and use THAT yaml.

Ansible is a full-scale configuration management and provisioning library that happened to implement a Kubernetes module, whereas Helm is only a package manager.

[ Parent | on HN ]

Re:Draft, Gitkube, Helm, Ksonnet, Metaparticle, Skaffold – C... by bacongobbler on Friday March 30, 2018 @01:14PM (#16717902)

Have you taken a look at some of the Helm 3 proposals? One of the points that talks about this exact use case is the concept of a "library chart", which is a way to define a block of code once and import it into another chart. In your case you could then define a block of code which will spit out a set of kubernetes labels that is shared across all your charts.

https://github.com/kubernetes-helm/community/blob/master/hel...

Your other point about it being difficult to interact with the kubernetes and Tiller APIs in any other language than Go is true, though it isn't necessarily a Helm Thing. This is a Kubernetes Thing. Tiller's going away in Helm 3 so the side effect should be that Helm will be much simpler to extend.

[ Parent | on HN ]

Re:Draft, Gitkube, Helm, Ksonnet, Metaparticle, Skaffold – C... by pyronicide on Friday March 30, 2018 @11:54AM (#16717149)

Check out ksonnet, it is closer to what you're looking for. (https://ksonnet.io/)

[ Parent | on HN ]

Re:Draft, Gitkube, Helm, Ksonnet, Metaparticle, Skaffold – C... by p3llin0r3 on Friday March 30, 2018 @12:03PM (#16717248)

KSonnet looks great, but I think it needs to mature a bit more.

PLUS, underneath the hood of Ksonnet is a massive amount of jsonnet code that I'm not extremely familiar with, and I like to learn my tools very thoroughly before I begin using them. So it's a big task to get started with it.

[ Parent | on HN ]

Re:Draft, Gitkube, Helm, Ksonnet, Metaparticle, Skaffold – C... by markbnj on Friday March 30, 2018 @12:03PM (#16717247)

I've been using it in production for close to a year and I run into the same issues. My biggest problem is really with the package manager metaphor and the fact that writing charts that can actually be installed and used in all/most use cases is really hard. To give a recent example: a chart for a log collector daemonset that allows you to pass in an external configmap name if you want to specialize the mounted .conf, but does not parameterize the 'checksum/config' annotation that is used to prompt a deployment update if the configmap changes.

Another is the inability to access cluster metadata from a template, so if something needs to know the podCidr or serviceCidr for example you have to inject that through a side channel.

I'm sure all of this will improve over time, as it has been, but right now I'm more attracted to ksonnet.

[ Parent | on HN ]