It’s nontrivial to get nomad to be resilient. For example, if your template uses keys from consul or vault, even with “noop” mode, and consul or vault are unreachable, Nomad will happily start killing running containers that had been rendered correctly in the past, because it can’t re-render the template. This pattern has only recently been addressed by “nomad_retry” but there have been several bugs with it, and 1.6.2 will currently and happily kill all running containers on a whim if some template resource can’t be reached. Under the hood this uses consul-template, which does support infinite retry, but getting nomad to use consul template safely is non trivial.

Node_pools just landed (2023!), and are still broken when using Levant (another abandoned nomad tool - think kustomize with a much more horrifying DSL).

Bonus nightmares: they’re still trying to get cgroups v2 working, the current version finally doesn’t DOS your backend services in an infinite loop, the authors are constantly surprised to hear you’re running a recent version, nomad-server leaks memory for fun Sunday afternoon outages, the UI outright lies, deployments get stuck forever because “progress_deadline” is more of a suggestion than a deadline, nomad-autoscaler is not highly available, crashes very easily, often scales faster than its “cooldown” window, and is simply stupid. AWS Karpenter feels a full decade into the future.

And all that so I can write my own Nomad specs instead of installing a vendors helm chart in a network isolated namespace. Boo.

To reiterate, I’ve liked Hashi for years and years, and I don’t have any ill will towards them. Just shocked at how poor nomad is compared to k8s.