Recent version of Nomad has introduced many QoL features like secrets and service discovery, though logging collection remains unsupported.

> though logging collection remains unsupported.

Do you happen to have a link to the issue (I guess or docs, if it's their official stance)? I'd enjoy reading how they ended up in that circumstance

Logcollection doesn't come out of the box, and you're kinda on your own to set it up, this article (although a bit dated) shows you what you must go through:

https://atodorov.me/2021/07/09/logging-on-nomad-and-log-aggr...

It also doesn't help that Nomad handles everything from Docker containers to managing regular applications.

Also worth noting that nomad’s closest competitor (Kubernetes) does not do this either. It’s a huge oversight in both ecosystems.

We may be talking about different things, but GP's comment said "unsupported" whereas in kubernetes it's just not part of the control plane offered by kubernetes itself, in exactly the same way that cloud load balancer or even cloud auto-scaling isn't provided by kubernetes itself

Having read the sibling's comment, I think I better understand the situation since Nomad isn't a container orchestrator and thus there isn't one homogeneous place from which _to_ collect the logs, but in any sane k8s setup that's not true since both dockerd and containerd have mechanisms to influence that behavior and k8s is perfectly able, and supports, scheduling per-Node utility plumbing stuff like that without drama

Nomad has equivalent mechanisms for collecting logs to Kubernetes. None are built in, in either platform.

Would you like to provide a link? It would help me a lot. Thanks.

https://developer.hashicorp.com/nomad/docs/job-specification... covers much of it from the collection side

Yes k8s does not provide a out-of-box logging collection solution but since everything is in containers, we can piggyback on Docker or Containerd via deamonset to get all the logs.

Nomad on the other hand support various payload(native exec, exec via chroot, containers, even Firecracker VM by community support), so doing logging collection by end users is trickier. It worth noting that Nomad UI(a official web admin panel) has log tailing utility built-in so maybe partial work has already been done. The developers may have other concerns.

The related issue is https://github.com/hashicorp/nomad/issues/10220

All of the work has been done; not partial work.

I’ve started playing with Nomad recently just to see what it’s all about. There are some annoying things with networking, but overall it’s been pretty fun so far. What made you consider it harmful?

It’s nontrivial to get nomad to be resilient. For example, if your template uses keys from consul or vault, even with “noop” mode, and consul or vault are unreachable, Nomad will happily start killing running containers that had been rendered correctly in the past, because it can’t re-render the template. This pattern has only recently been addressed by “nomad_retry” but there have been several bugs with it, and 1.6.2 will currently and happily kill all running containers on a whim if some template resource can’t be reached. Under the hood this uses consul-template, which does support infinite retry, but getting nomad to use consul template safely is non trivial.

Node_pools just landed (2023!), and are still broken when using Levant (another abandoned nomad tool - think kustomize with a much more horrifying DSL).

Bonus nightmares: they’re still trying to get cgroups v2 working, the current version finally doesn’t DOS your backend services in an infinite loop, the authors are constantly surprised to hear you’re running a recent version, nomad-server leaks memory for fun Sunday afternoon outages, the UI outright lies, deployments get stuck forever because “progress_deadline” is more of a suggestion than a deadline, nomad-autoscaler is not highly available, crashes very easily, often scales faster than its “cooldown” window, and is simply stupid. AWS Karpenter feels a full decade into the future.

And all that so I can write my own Nomad specs instead of installing a vendors helm chart in a network isolated namespace. Boo.

To reiterate, I’ve liked Hashi for years and years, and I don’t have any ill will towards them. Just shocked at how poor nomad is compared to k8s.