Let me ask you a question if I may: I have a server app I’m running on k8s, it’s stateful and external users connect to it (over grpc). Users connect to a specific server which is identified by ID, this is important and can’t be easily rearchitected. I also expose an API to spin up servers for new ids which users can then connect to. At present I’m doing this by having my “controller” api (which is just an app running in a pod which exposes a rest api then makes calls to the k8s api) create a new deployment+service per server and then add an ingress for that service. I’ve been considering taking the k8s functionality from my controller and putting it in a new CRD + controller. Is there a better way to do this? I feel like I’m fighting the system somewhat with my current approach and yet I don’t see a simple answer using vanilla k8s primitives. I feel like what I want is scalable statefulsets that automatically create an service+ingress to each pod created but nothing seems to offer that. My use-case seems simple and common, so how is everyone else solving this problem.

You may want to look into operators. It seems to address your requirements nicely.

totally agree :) I have been running Kubernetes in production for several years now on multiple projects. So far I had no need to use any CRDs or even Helm. Just well maintained deployments, services, secrets and ingress manifests.

I've started trying to learn more about kubernetes by setting up some personal services and it seems like everything I find online points to some helm chart to install.

I think because I don't fully understand k8s, yet, helm seems too magical. It's abstracting away what I already don't understand and doesn't feel right.

For now, I've been focusing on writing my manifests and applying them with kubectl to get a feel for what's going on under the hood. Maybe a time will come where it's a tool that I'll need to reach for, though.

So how do you run stateful workloads? E.g. postgres or mysql?

Generally speaking, you probably shouldn't run databases in k8s unless you need to scale them dynamically, or cloud disk performance is acceptable to you. Having said that, k8s is getting a lot of support for running databases on local disks. A lot of newsql databases (CockroachDB, TiDB, Yugabyte, Vitess, etc) are shipping with strong k8s support/integration.

As a matter of fact, we recently released the 1.0 GA version of TiDB Operator: https://github.com/pingcap/tidb-operator