News for hackers. Stuff that matters.
Re:Fly.io outage, recently deployed apps down, deployments d... by atonse on Wednesday March 15, 2023 @09:23PM (#35176990)
How is this possible? How is consul not self—healing? It just seems so brittle in a way even database clusters aren’t.
Re:Fly.io outage, recently deployed apps down, deployments d... by candiddevmike on Wednesday March 15, 2023 @09:35PM (#35177080)
Consul is a lot more than just a database cluster, and that may be part of its problem.
Re:Fly.io outage, recently deployed apps down, deployments d... by throwawaaarrgh on Wednesday March 15, 2023 @10:17PM (#35177364)
All distributed decentralized systems are brittle. The only people who don't think this are people who haven't run them at scale.

Also, "self-healing" isn't really one thing. There are hundreds of different problems that can take out such a cluster, and every single one of them needs its own "self-healing" mechanism. These systems are literally the most complicated kinds of systems.

Re:Fly.io outage, recently deployed apps down, deployments d... by namaria on Sunday March 19, 2023 @06:43AM (#35217729)
"Should have self healing" can be expanded into "should have systems to address the underlying system failure modes", which starts to shed some light into why distributed systems will always run into failure modes.
Re:Fly.io outage, recently deployed apps down, deployments d... by mdaniel on Wednesday March 15, 2023 @10:32PM (#35177475)
I haven't been responsible for babysitting consul, but I have been responsible for etcd for years and if consul's problem is anything like etcd's it's because members have identity and if one of them goes toes up then etcd will wait forever for the snowflake to come back to life, and if that's not how the underlying infra is configured, that's very very bad. Mix it mTLS into this story and it gets worse

I stayed away from the so-called "stacked" control plane of etcd inside kubernetes because it can make a tiny fire into a sharkfirenado but recently I've heard discussions of k3s (which uses dqlite) managing the etcd members and then "formal" kubernetes managing the workloads pointed at that k3s-stacked-etcd but I haven't tried it yet in order to know how theory and practice differ

Re:Fly.io outage, recently deployed apps down, deployments d... by cheeseprocedure on Wednesday March 15, 2023 @11:34PM (#35177836)
Consul’s autopilot feature makes life a little easier by automatically reaping failed instances:

https://developer.hashicorp.com/consul/tutorials/datacenter-...

Paired with cloud discovery, it makes for a tolerable operational experience when instances are expected to occasionally disappear.

Re:Fly.io outage, recently deployed apps down, deployments d... by grrdotcloud on Wednesday March 15, 2023 @11:30PM (#35177813)
I've worked with Vault and clusters, not Consul specifically but generally there's a healthy cluster state until something happens putting the cluster into an inconsistent state.

Generally there's a master node or multiple nodes in agreement. If the cluster cannot agree on it's current state the entire system may run multiple versions or be completely unavailable or provide inconsistent response bringing down other systems that rely on it.

Inspection itself is hampered by elections or syncing state or other process/race related/caching/ddossing itself or other services.