It was all quite a while (3-4 years) ago, so I don’t remember much details, except that Swarm was then not exactly capable of working in IPv6-only environments. It was significantly easier to navigate Docker’s codebase than Kubernetes’.
Ultimately, my solution was that I understood that I didn’t need any kind of orchestration there, just HA/failover.
A quick search turned up this: https://www.greencentury.com/google-parent-alphabet-pledges-...
Looks like a possible case of greenwashing to me.
My experience with bare-metal Kubernetes hasn't been that bad. On my home cluster with very underpowered J4105 CPUs, I get about 5% CPU usage on the master node at idle, and 2% on workers.
I had high hopes back in my earlier days of them offering pluggable KV stores so folks could make tradeoffs if etcd doesn't align with their goals and risks but as best I can tell they've declared it a non-goal
That said this is a three node cluster running a handful of containers, with services accessed by my wife and I. So not exactly a large scale production experience.
You'd have to do a lots of custom morphing if you want that kind of interface over docker swarm.
For smaller setups, simplicity of docker swarm might be useful but I am curious to know from people who are running clusters of 25 to 50 nodes.
This is the voice of absolute delusion. Swarm is a mess. Nomad actually scales.
You will spend quite some time figuring the labels out, but once you have them for one service, it works for many.
A new solution that might be even better is Kamal by 37 Signals. It runs all their production stuff, so it must be capable. Can’t wait for a version 2 where, I’d suspect, by then there will be more community contributions for some more diverse setups merged in.
My tldr now is nomad should be considered harmful.
Do you happen to have a link to the issue (I guess or docs, if it's their official stance)? I'd enjoy reading how they ended up in that circumstance
https://atodorov.me/2021/07/09/logging-on-nomad-and-log-aggr...
It also doesn't help that Nomad handles everything from Docker containers to managing regular applications.
Having read the sibling's comment, I think I better understand the situation since Nomad isn't a container orchestrator and thus there isn't one homogeneous place from which _to_ collect the logs, but in any sane k8s setup that's not true since both dockerd and containerd have mechanisms to influence that behavior and k8s is perfectly able, and supports, scheduling per-Node utility plumbing stuff like that without drama
Nomad on the other hand support various payload(native exec, exec via chroot, containers, even Firecracker VM by community support), so doing logging collection by end users is trickier. It worth noting that Nomad UI(a official web admin panel) has log tailing utility built-in so maybe partial work has already been done. The developers may have other concerns.
The related issue is https://github.com/hashicorp/nomad/issues/10220
Node_pools just landed (2023!), and are still broken when using Levant (another abandoned nomad tool - think kustomize with a much more horrifying DSL).
Bonus nightmares: they’re still trying to get cgroups v2 working, the current version finally doesn’t DOS your backend services in an infinite loop, the authors are constantly surprised to hear you’re running a recent version, nomad-server leaks memory for fun Sunday afternoon outages, the UI outright lies, deployments get stuck forever because “progress_deadline” is more of a suggestion than a deadline, nomad-autoscaler is not highly available, crashes very easily, often scales faster than its “cooldown” window, and is simply stupid. AWS Karpenter feels a full decade into the future.
And all that so I can write my own Nomad specs instead of installing a vendors helm chart in a network isolated namespace. Boo.
To reiterate, I’ve liked Hashi for years and years, and I don’t have any ill will towards them. Just shocked at how poor nomad is compared to k8s.
Early Hashicorp was incredible. They were open source stewards and looked like an up-and-coming Redhat or Canonical. Their products were ground-breaking and truly huge value adds to the open-source eco system. But they got extremely popular (mostly thanks to Terraform, which took off and drew more attention to their other products).
But since going public it has been clear that they are trying to get money and enterprise customers at any cost.
Terraform itself has felt like it was in maintenance-mode since hitting version 1. Terraform providers frequently break. For production I have found you need to pin providers down to the patch level, because i've had multiple problems just in small patch-level updates over the recent years. They famously started rejecting any open-source contributions that didn't provide a business value for Hashicorp. Since TF hitting v1, almost all the attention seems to be towards Terraform Cloud and Terraform Enterprise. At Hashicon, it feels like every talk is just propaganda pushing these products. It is all they care about anymore.
Nomad was a product that Hashicorp was very excited about for a while and then they seemed to just abandon on the side of the road in their quest for enterprise dominance (probably after learning that most Enterprises are all-in on K8s and Nomad is more beneficial to fast-moving start-ups).
Vault was an incredible tool, especially in the Open-Source space. But in the past few years they have really split the open-source and licensed versions of Vault significantly which made the open-source version feel more like a burden to Hashicorp. The last time I talked to Hashicorp about Vault (we were deeply considering a switch to Vault at work last year), they treated the open-source self-hosted solution as a "trial" of the real Vault, and that is what it felt like. Almost everything we ran into during setup, they would respond back with "oh that's fine in the enterprise version".
Overall, they have peeled back on all but the absolute minimum efforts required to support their open source versions of products and have been an entirely enterprise focused company for a while. And I can't blame them, sure you need to make money. But I can't help but point to organizations like RedHat and Canonical as examples of what Hashicorp could have been.
At this point I feel like the parent that watches their kids, talking about how they missed out on reaching their potential, thanks mostly to what seems like greed or over-ambition. "I'm not mad, I'm just disappointed" comes to mind when I think of Hashi. I have high-hopes for OpenTofu to fill the Terraform void. I've moved past Vault and am using one of the big hyperscalers' secrets management tools (which I enjoy a lot less, but is cheaper and less complicated). I use kubernetes instead of Nomad, which again is fine, it has become the standard anyway. So i'll be just fine... but I'm dissapointed at you Hashicorp. That's all.
TL;DR: they put me through an interview gauntlet, took my entire day, and ignored me for weeks until I aired my frustrations in their contact us form.
I haven’t used their products since but always heard great things about them. That enterprise push is fierce.
[1]: https://blog.webb.page/2018-01-11-why-the-job-search-sucks.t...
Interviewing sucks for everyone on either side, and efforts to improve candidate experience on my end have often been met with resistance in the company. One thing I've done is try to be as clear as possible at the end of interviews where someone stands. If it doesn't go well, I try to relate what didn't go well and how it applies to the position we're hiring for. If it goes well, my interviewers and I try to send out invites for the candidate to schedule the next interview within 10 minutes of completion, and I tell them that during the call.
So many companies are so risk-averse though, that they act like interviews are a poker game. I've generally had nothing but positives with my approach, and yes, someone may sue us at some point for being blunt and upfront, but my experience so far is people appreciate honest feedback if you deliver it kindly on the spot.
Every person I interviewed with at Hashicorp seemed to enjoy the conversation and the live coding challenge wasn't particularly challenging. Ah well.
Rejection is protection and redirection.
I tried doing that last time I hired and I think generally speaking makes a good experience for candidates, BUT when you have those 2-3 individuals that have really high opinions of themselves it can get tiring for the hiring manager (myself), because they won't accept the answer anyway and will keep asking and ultimately give some bad review in Glassdoor etc
And yet, as an enterprise customer, their sales team is terrible. Incredibly unhelpful with problems, not willing to discuss pricing, ultimately lost the deal due to how unresponsive they were, and how they basically told as that they wouldn’t be any more likely to fix all the bugs we were hitting if we paid.
The reason they had to go this route is they are getting trounced by their competitors and it’s all their own fault
The BUSL is not OSI approved and places restrictions around usage, in an effort to sell some product or service off of a project and prohibit competition. It is not free as in beer either, if you hit such usage.
The source is available (hence source available being another terminology) but even Hashicorp has dropped "open source" from nearly every relevant page in understanding that their products no longer have an open source core (see instead "Vault Community" or "Terraform Community").
The general consensus is that no, the BUSL is not OSS as OSI has not blessed it as such: https://opensource.org/licenses/
Yes, source being available is not enough to be called open source.
Edit to add: https://github.com/hashicorp/vault/graphs/contributors?from=...
Unfortunately it doesn't seem like there are going to be open source alternatives to Vault, Consul, or Nomad (that last one is hilarious to me: nomad was a good product until hashicorp stopped investing in it, now that it's closed source it basically has a snowballs chance in hell of being adopted).