News for hackers. Stuff that matters.
Re:Fly.io outage, recently deployed apps down, deployments d... by mrkurt on Wednesday March 15, 2023 @08:52PM (#35176781)
This has been a rough week, and I'm sorry we broke peoples' apps. We had a big Nomad outage on Monday, and then a suspiciously similar Consul outage today. Both tipped over faster than we could detect and mitigate, and we ended up having to do serious surgery to build entirely new Consul/Nomad clusters.

There's nothing to brag about here, I just wanted to let y'all know we're listening (even when things aren't on the HN front page).

Re:Fly.io outage, recently deployed apps down, deployments d... by wjossey on Wednesday March 15, 2023 @09:37PM (#35177095)
This stuff is hard. As someone who runs infra teams for a living, these are the worst kinds of weeks.

Hang in there. You all will learn from this and be better for it. Your architecture will improve. Customers will give you a second chance. This too shall pass.

Sending positive vibes.

Re:Fly.io outage, recently deployed apps down, deployments d... by mrkurt on Wednesday March 15, 2023 @10:18PM (#35177373)
<3
Re:Fly.io outage, recently deployed apps down, deployments d... by srhyne on Thursday March 16, 2023 @08:02AM (#35180828)
Lovely response. Ah, kindness. So refreshing to see.
Re:Fly.io outage, recently deployed apps down, deployments d... by atonse on Wednesday March 15, 2023 @09:08PM (#35176896)
We had mysterious consul outages (and related nomad outages) causing us to never deploy our new hashicorp stack to production.

Shame cuz we were excited about our nomad+consul+vault setup and invested a lot of money into building it. But just didn’t have the time or enough depth of expertise to babysit it.

Re:Fly.io outage, recently deployed apps down, deployments d... by Mizza on Wednesday March 15, 2023 @09:15PM (#35176944)
From my experience with the Hashi stack, I don't think it's a coincidence that Fly has a lot of downtime and are a major Hashi user. Terraform makes excellent bait though.

Still love using Fly, please add static assets hosting/CDN.

Re:Fly.io outage, recently deployed apps down, deployments d... by atonse on Wednesday March 15, 2023 @09:23PM (#35176990)
How is this possible? How is consul not self—healing? It just seems so brittle in a way even database clusters aren’t.
Re:Fly.io outage, recently deployed apps down, deployments d... by candiddevmike on Wednesday March 15, 2023 @09:35PM (#35177080)
Consul is a lot more than just a database cluster, and that may be part of its problem.
Re:Fly.io outage, recently deployed apps down, deployments d... by throwawaaarrgh on Wednesday March 15, 2023 @10:17PM (#35177364)
All distributed decentralized systems are brittle. The only people who don't think this are people who haven't run them at scale.

Also, "self-healing" isn't really one thing. There are hundreds of different problems that can take out such a cluster, and every single one of them needs its own "self-healing" mechanism. These systems are literally the most complicated kinds of systems.

Re:Fly.io outage, recently deployed apps down, deployments d... by namaria on Sunday March 19, 2023 @06:43AM (#35217729)
"Should have self healing" can be expanded into "should have systems to address the underlying system failure modes", which starts to shed some light into why distributed systems will always run into failure modes.
Re:Fly.io outage, recently deployed apps down, deployments d... by mdaniel on Wednesday March 15, 2023 @10:32PM (#35177475)
I haven't been responsible for babysitting consul, but I have been responsible for etcd for years and if consul's problem is anything like etcd's it's because members have identity and if one of them goes toes up then etcd will wait forever for the snowflake to come back to life, and if that's not how the underlying infra is configured, that's very very bad. Mix it mTLS into this story and it gets worse

I stayed away from the so-called "stacked" control plane of etcd inside kubernetes because it can make a tiny fire into a sharkfirenado but recently I've heard discussions of k3s (which uses dqlite) managing the etcd members and then "formal" kubernetes managing the workloads pointed at that k3s-stacked-etcd but I haven't tried it yet in order to know how theory and practice differ

Re:Fly.io outage, recently deployed apps down, deployments d... by cheeseprocedure on Wednesday March 15, 2023 @11:34PM (#35177836)
Consul’s autopilot feature makes life a little easier by automatically reaping failed instances:

https://developer.hashicorp.com/consul/tutorials/datacenter-...

Paired with cloud discovery, it makes for a tolerable operational experience when instances are expected to occasionally disappear.

Re:Fly.io outage, recently deployed apps down, deployments d... by grrdotcloud on Wednesday March 15, 2023 @11:30PM (#35177813)
I've worked with Vault and clusters, not Consul specifically but generally there's a healthy cluster state until something happens putting the cluster into an inconsistent state.

Generally there's a master node or multiple nodes in agreement. If the cluster cannot agree on it's current state the entire system may run multiple versions or be completely unavailable or provide inconsistent response bringing down other systems that rely on it.

Inspection itself is hampered by elections or syncing state or other process/race related/caching/ddossing itself or other services.

Re:Fly.io outage, recently deployed apps down, deployments d... by jen20 on Thursday March 16, 2023 @07:43AM (#35180698)
The simpler explanation is that running products designed for LAN usage on a WAN is a fundamentally bad plan, as the folks over at fly acknowledge, even in this thread.

Meanwhile, hundreds of thousands of Consul, Nomad and Vault clusters used appropriately work perfectly well…