Re:Rapid growth, lessons learned and improvements at Fly.io
by tptacek on Monday March 06, 2023 @10:15PM
(#35051105)
I don't disagree that there are opportunities to introduce topology. I do disagree that there are opportunities to benefit from distributed consensus. If a server in ORD is down, it doesn't matter what some server in SJC says it's hosting; all the ORD instances of all the apps on that server are down. If that same ORD server is up, it doesn't matter what any server says it's running; it's authoritative for what it's running.
Of course, OSPF has topology and aggregation, too.
At any rate: I didn't design the system we're talking about.
Re:Rapid growth, lessons learned and improvements at Fly.io
by dastbe on Monday March 06, 2023 @11:38PM
(#35051629)
> I do disagree that there are opportunities to benefit from distributed consensus
there's some benefits to static stability and grey failure, but sure, whatever. the important bit is to have clear paths of aggregation and dissemination in your system.
that being said
> it doesn't matter what some server in SJC says it's hosting
it kind of does matter doesn't it? assuming that server in SJC is your forwarding proxy that does your global loadbalancing, what that server is aware of is highly relevant to what global actions you can take safely.
Re:Rapid growth, lessons learned and improvements at Fly.io
by tptacek on Monday March 06, 2023 @11:52PM
(#35051718)
My point is just that there isn't a consensus algorithm that needs to get run to know which of the two proposals to accept.
Re:Rapid growth, lessons learned and improvements at Fly.io
by injinj on Tuesday March 07, 2023 @01:09AM
(#35052147)
It doesn't need a raft consensus algorithm, but corrosion does converge to a consensus, doesn't it? In the OSPF example, that does needs to converge to a state that is consistent and replicated on all the routers, otherwise loops and drops will occur. I'm curious if any convergence benchmark has been done that compares raft to corrosion.
to be pithy about it, going full-bore gossip protocol is like going full-bore blockchain: solves a problem, introduces a lot of much more painful problems, and would've been solved much more neatly with a little bit of centralization.