When migrating from a non-containerized deployment process to a containerized one, there are a lot of new skills the employees have to learn. We've had 40+ employees, all who are basically full of work, and the mandate comes down to containerize, and all of these old school RPM/DEB folks suddenly need to start doing docker. No big deal, right? Except...half the stuff does not dockerize easily requires some slightly-more-than-beginner docker skills. People will struggle and be frustrated. Folks start with running one container manually, and quickly outgrow that to use compose. They almost always eventually use compose to run stuff in prod at some point, which works but eventually that one server is full. This the is the value of swarm - letting people expand to multi-server and get a taste of orchestration, without needing them to install new tools or learn new languages. Swarm adds just one or two small new concepts (stack and service) on top of everything they have already learned. It's a god send to tell a team they can just run swarm init, use their existing yaml files, and add a worker to the cluster. Most folks start to learn about placement constraints, deployment strategies, dynamic infrastructure like reverse proxy or service mesh, etc. After a bit of comfort and growth, a switch to k8s is manageable and the team is excited about learning it instead of overwhelmed. A lot (?all?) of the concepts in swarm are readily present in k8s, so the transition is much simpler
We generally avoid mounting volumes at all costs. The challenge of mapping host uid:gid to container uid:gid (and keeping that mapping from breaking) proved painful and not worth the effort
> The only serious downsides is having to use the HCL DSL ( https://github.com/hashicorp/hcl ) and their web UI being read only in the last versions that i checked.
1. IIRC you can run jobs directly from UI now, but IMO this is kinda useless. Running a job is simple as 'nomad run jobspec.nomad'. You can also run a great alternative UI ( https://github.com/jippi/hashi-ui ).
2. IMO HCL > YAML for job definitions. I've used both extensively and HCL always felt much more human friendly. The way K8s uses YAML looks to me like stretching it to it's limits and barely readable at times with templates.
One thing that makes nomad a go-to for me is that it is able to run workloads pretty much anywhere. Linux, Windows, FreeBSD, OpenBSD, Illumos and ofc Mac.
I know HCL gets a lot of hate around here but I find just dandy and I'd rather write HCL over YAML any day of the week.
Also to just put another thumbs up for Nomad because it's been absolutely fantastic for us and has been so much easier to manage and deploy. We've begun testing edge deployments with it as well and so far looks very promising.
Consul and Vault are also two Hasicorp products I can't live without at this point either. Vault is probably one of the first things I deploy now.
Anyway rambling, I think a lot of people just immediately jump to Kubernetes without really ever giving Nomad a look first.
I agree that the format Kubernetes uses is probably a bit overcomplicated, which is why I've also sometimes used the aforementioned Compose format with tools like Kompose to save some of my time.
In that regard, it's perhaps not the fault of YAML itself (even though the format has its own complexity problems), but rather of the tool using it.
What personally keeps me away from HCL for the most part is the fact that a lot of the time you can find Compose files for various pieces of software online and only have to change some parameters for the most part to get stuff running vs having to rewrite all of it in a different format.
Are there any Compose to HCL tools out there yet?
And yeah, there is not much examples of HCL files for different software, but most of the time there is not much to adapt. Usually when i need to run something, i just take some random HCL file i have and substitute the docker image, edit the amount of resources and add storage and config via templates. And done. It might be confusing/intimidating at first, but like with anything after some time you should be ok.
In my experience, there are actually two platforms that do this pretty well.
First, there's Docker Swarm ( https://docs.docker.com/engine/swarm/ ) - it comes preinstalled with Docker, can handle either single machine deployments or clusters, even multi-master deployments. Furthermore, it just adds a few values to Docker Compose YAML format ( https://docs.docker.com/compose/compose-file/compose-file-v3... ) , so it's incredibly easy to launch containers with it. And there are lovely web interfaces, such as Portainer ( https://www.portainer.io/ ) or Swarmpit ( https://swarmpit.io/ ) for simpler management.
Secondly, there's also Hashicorp Nomad ( https://www.nomadproject.io/ ) - it's a single executable package, which allows similar setups to Docker Swarm, integrates nicely with service meshes like Consul ( https://www.consul.io/ ), and also allows non-containerized deployments to be managed, such as Java applications and others ( https://www.nomadproject.io/docs/drivers ). The only serious downsides is having to use the HCL DSL ( https://github.com/hashicorp/hcl ) and their web UI being read only in the last versions that i checked.
There are also some other tools, like CapRover ( https://caprover.com/ ) available, but many of those use Docker Swarm under the hood and i personally haven't used them. Of course, if you still want Kubernetes but implemented in a slightly simpler way, then there's also the Rancher K3s project ( https://k3s.io/ ) which packages the core of Kubernetes into a smaller executable and uses SQLite by default for storage, if i recall correctly. I've used it briefly and the resource usage was indeed far more reasonable than that of full Kubernetes clusters (like RKE).