To follow this - what do you feel K8S provides on top of terraform?

We used K8S on a large project and I felt like it really, really wasn't necessary.

Nothing. We use Terraform to provision a simple auto-scaling cluster with loadbalancers and certs, does exactly the same thing but there is no Docker and k8s. Few million lines less Go code turning yaml filed into seggfaults.

Consistency and standardized interfaces for AppOps regardless of the hyper-cloud I use. Kubernetes basically has an equivalent learning curve, but you only have to do it once

They operate at different layers. K8s sits on top of the infrastructure which terraform provisions. It's far more dynamic and operates at runtime, compared to terraform which you execute ad-hoc from an imperative tool (and so only makes sense for the low level things that don't change often).

So terraform is a higher-order, meta-Kubernetes. It's very rarely used, but who provisions the cluster itself? That's terraform.

So terraform creates the cluster, DNS and VPC. Then k8s runs pretty much everything.

How are you deploying the workloads into the cluster? Manual kubectl or Helm, GitOps with something like Flux, something else?

Spinnaker. Huge. Clunky. But excellent if you can justify it.

Alas, still rely on bash for that. Practically a one liner.

Mainly just kustomize piped into kube apply.

But, but, but. Having to a create a one-off database migration script imperatively.

Ooh!

Some others are Tekton, Argo, KNative You could also use Jenkins with K8s deploy plugin (from MS Azure Devops team)

Why would you want to provision your own k8s cluster, if you can use EKS, AKS or similar?

I don't. I use a cloud cluster. But that still has to be provisioned? You need to choose size, node pool, VPC, region, etc.

EKS, GKE and the like have a number of limitations. For example: they can be pretty far behind in the version of K8S they support (GKE is at 1.15 currently, EKS at 1.16; K8S 1.18 was released in at the end of March this year.

One example I worked on myself: when you need to train lots of ML models and need lots of video cards. Those would be damn expensive in the cloud!