Upgrades that get in bad states, ya, I've had to deal with it before, and it definitely sucks. The "atomic" goal that Helm 3 tries for is better in my opinion, but still has rough edges.
What have you been using as an alternative that doesn't require you to write your own manifests for every dependency?
We found a lot of success also with Kustomize. I did not enjoy the ksonnet detour as much - it was nice to override a prototype instead of having to screw with yaml templating, but jsonnet is not a good language.
Helm 3 had some designs on a lua API, which I thought was a promising idea. I've been out of the loop since moving on to a company that is back in EC2-land (which really just means that another team manages the thing that approximates Kubernetes...).
I agree with you about an addon manager, but then something has to be the source of truth for that, and the "KubeApps" type apps were all using Helm under the hood anyways.
I played around with Kustomize for a bit at one point, but honestly had a bit of difficulty understanding how it loaded and applied the different "layers". I've been looking more recently at it as a "make some quick customizations to this rendered helm chart" tool as of late.
This seems to be a big problem with container orchestrators these days. I was deploying something to ECS and while 90% of the config could be accomplished with environment variables, there are still a few cases where it was expected that I would provide files. On a local machine with Docker and some config mgmt like Chef or Ansible, no problem - template out those files and volume mount them in. In ECS? No easy to way to do it without baking your own container or using EFS, which for one config option seemed like major fuckin' overkill. Ended up bastardizing the entrypoint with some judicious base64'd bullshit and got it going without either, but christ, how fucking hard would it be to just provide a native mechanism for templating config files out?
For applications I built myself, I would often just roll the possible configs for us as files in the container, and you pick one with a single env var (realistically this was like "dev", "beta", "prod"). Not great, but meant I could avoid the clusterfuck that is specifying all my AWS configuration in env vars on a per deploy basis.
ConfigMaps could sort of help with this, but I found what you found - having to base64 a config file as a field in a ConfigMap and make sure it gets mounted correctly just seems pretty baroque for configuring something with a file. And at the time, ConfigMap changes didn't even cause a redeploy, so your pods would for their lifetime keep on using a stale config!
However Kubernetes does have a lot to offer, especially with the easy of hosted Kubernetes solutions these days. Helm makes installing tools and dependencies you may need almost trivial.
One of the nice things about Kubernetes is that it is relatively easy to scale from a small deployment to a larger HA deployment when (and if) you need to.
> Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed.
This sounds like more of a management, requirements, and overbuilding problem, rather than a Kubernetes problem. Even without Kubernetes, if they wanted an HA devops set up, they still would have built one. Honestly without Kubernetes it could have ended up more expensive.