The free version of Nomad is missing, IIUC:

- no quotas for teams/projects/organizations

- no preemption (ie higher priority job preempts lower priority job)

- no namespacing

So generally it's somewhat useless in organizations where there are multiple different teams that should be able to coexist on a cluster without stepping on eachothers' toes, or even where you want a CI system to access the cluster in a safe manner.

Preemption is going OSS! https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md

We fully intend to migrate more features to OSS in the future -- especially as we build out more enterprise features. As you can imagine building a sustainable business is quite the balancing act, and there's constant internal discussion.

(I'm the Nomad Team Lead at HashiCorp)

This is going to make our AI team very happy because they can just dump experiments into the cluster at low-priority so those'll be done when those are done.

It's also going to make the operators very unhappy because it'll be harder to monitor actual memory utilization (allocated memory vs memory really in use) in order to plan cluster extensions. Are there some tools around or work planned to make this kind of scaling and utilization easier?

I was going to say "yes!" because we do have some telemetry/metrics improvements coming up, but then I realized those won't address your use case. It seems like aggregating cluster resource usage by job priority is what we need for that. Our UI might be able to calculate that, but I don't think our metrics include job priority so there's no way for your existing tooling to display it.

Please file an issue or link me to an existing issue of you have time. This seems really compelling!

Enterprise version has all of those things ;)

For my personal use I don't need them and for a business Enterprise won't break the bank. You'd be surprised how much your management might be interested in having an escalation path if everything goes south and you need a hot fix. I can vouch that Enterprise support is worth it if you're on a small team that can't spend all day on this stuff

We run everything in Nomad for all our teams, but we don’t use any of the features you mention, and it’s not causing issues. We wrote some metrics for how much memory all the allocations are claiming, and we throw another EC2 instance in the cluster when we’re running out. Works pretty well. Preemption seems like a nice feature, but I imagine getting teams to coordinate their preemption values would be a political nightmare.

For me the ability to squeeze in some jobs in between the cracks at best effort priority is crucial (ie. any sort of high latency batch processing / experiments). I wouldn't want these extremely low priority jobs to compete with, I don't know, an actual customer facing service.

Also, we run on bare metal, so there really isn't a way to request extra capacity within seconds.

Preemption is coming in the next release in OSS :)

A few long-standing feature requests I've noticed a lot:

* No autoscaling

* Can't reserve entire CPU core: https://github.com/hashicorp/nomad/issues/289

* No way to run jobs sequentially: https://github.com/hashicorp/nomad/issues/419

Autoscaling is in Tech Preview as of March: https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-t...

Disclaimer: I work at HashiCorp in Product Management.

Nice! It's definitely been one of the bigger holes.

I hope you'll also consider sequential tasks in a single jobfile. Without it it's kind of awkward because you have to be passing shell scripts around just to run a setup step before your actual workload.

We’ve also started to implement this with a new task dependencies feature in the latest release. Check out: https://www.hashicorp.com/resources/preview-of-nomad-0-11-ta...

https://learn.hashicorp.com/nomad/task-deps/interjob

There’s more work going on there to further improve the feature in upcoming releases.

Disclaimer: I’m an engineer on the Nomad team.

For us it's not a feature missing, but the mindshare is low, and there's not really any prebuilt examples of how to run and maintain services long term reliably.

Does it have "easy" side-cars/init/post? Last time I checked those were missing for example.

It does! Here is the relevant documentation and examples:

https://www.nomadproject.io/docs/job-specification/lifecycle...

https://learn.hashicorp.com/nomad/task-deps/interjob

Ill check that out, I really think K8s is becoming a bit too complex for many customers. Thanks!

The main missing feature is that it's not kubernetes.

You have an extra word there.