I don't want to sound bad, but to me all of these sound like a misconfiguration and lack of understanding how consul works tbh. But i don't know the full context so eh, these things happen.
Re:Nomad vs. Kubernetes
by still_grokking on Friday October 15, 2021 @10:16PM
(#28885033)
And what is the replacement for it?
We rearchitected. At one workplace, we built and distributed our own service. At another, we shifted to semi-automated more static lists of servers for roles; those servers were much less dynamic.
Re:Nomad vs. Kubernetes
by still_grokking on Saturday October 16, 2021 @12:25PM
(#28888947)
There are exceptions but most of the time replacing an off the shelf std. solution with something self made looks like NIH syndrome to me.
The exceptions are the few cases where you know you will forever only need some strict set of features and your own solution can provide them by way more simple means than the in comparison "fat" off the shelf solution.
I'm not sure service discovery in a cluster is one of those cases.
etcd
Re:Nomad vs. Kubernetes
by still_grokking on Saturday October 16, 2021 @12:26PM
(#28888960)
Never heard that someone moved form Consul to etcd. It's always the other way around.
* in a legacy system, a server, perhaps with the leader node, filled up disk space and became unhealthy, the consul agent kept reporting it and itself as healthy, and failover and gossip generally wedged
* in a dev environment, after we replaced some servers in the cluster, the other nodes noted cert changes and refused to work with the new servers
* second-hand, in self-hosted installations, it caused a number of hard-to-troubleshoot outages
* something about circular dependencies and going by "wait _n_ seconds" rather than by healthiness
It was reliable enough that it could gather a really significant blast radius, and it had different gnarly failure modes, so documentation could be irrelevant from case to case.