News for hackers. Stuff that matters.
Re:Fly.io Postgres cluster down for 3 days, no word from the... by quickthrower2 on Friday July 21, 2023 @02:43AM (#36810724)
Fly is in my “try later book” from a year or two ago. I remember it was hard to deploy anything due to downtime so gave up. Sad that stuff like this still happens.

You shouldn’t need to multi region a postgres yourself - they should have at least 2 data centre redundancy for the region and it just works.

Hope they get some magic sauce to become better at this.

Re:Fly.io Postgres cluster down for 3 days, no word from the... by throwawaymaths on Friday July 21, 2023 @04:53AM (#36811584)
> Hope they get some magic sauce to become better at this.

When I saw them describe their multiregion SQL replication architecture I thought "what crazy person thought this wouldn't eventually open up a spider's nest of distributed systems errors?"

Re:Fly.io Postgres cluster down for 3 days, no word from the... by api on Friday July 21, 2023 @09:25AM (#36813480)
CockroachDB does this, but that's the result of over 10 years of heads down hard-ass engineering and it's still slower than Postgres because distributed sync is not free. That means you have to provision it properly and with enough resources.

Their license would require a company like fly.io to pay them though, so I'm sure this resulted in fly.io instead trying to whip up an improvised infrastructure on the back of stock Postgres. I bet this cost them a whole lot more than paying CockroachDB would have, but devs have been conditioned that you should never ever pay for software even if it's the result of tons of deep engineering and solves massive brutal problems for you.

P.S. I don't work for CDB but I would absolutely consider them and we may end up using them at some point. They let you do a ton for free. They only charge for stuff you need if you get really really huge or if you are running a SaaS reselling DB services like fly.io would have been doing.

Re:Fly.io Postgres cluster down for 3 days, no word from the... by tptacek on Friday July 21, 2023 @10:49AM (#36814468)
Our multiregion SQL replication architecture is the standard Postgres multiregion replication architecture. We do single-write-leader, multiple reader replicas, like everybody else does.
Re:Fly.io Postgres cluster down for 3 days, no word from the... by throwawaymaths on Friday July 21, 2023 @11:51PM (#36823069)
This is not standard. I see now that it is legacy, but I think it still demonstrates a bit of poor judgement. I believe it was before you were at fly, tptacek

https://fly.io/docs/getting-started/multi-region-databases/