News for hackers. Stuff that matters.
Re:Zero downtime Postgres migration, done right by luhn on Friday June 11, 2021 @12:31PM (#27475384)
Braintree (IIRC) had a really clever migration strategy, although I can't seem to find the blog post now. They paused all traffic at the load balancer, cut over to the new DB, and then resumed traffic. No requests failed, just a slight bump in latency while the LBs were paused.

This app apparently had robust enough retry mechanisms that they were able just eat the errors and not have customer issues—Color me impressed! I'm not sure how many teams can make that claim; that's a hard thing to nail down.

Re:Zero downtime Postgres migration, done right by scrollaway on Friday June 11, 2021 @12:38PM (#27475473)
I remember reading something like that about Adyen. That might be why you're unable to find it.
Re:Zero downtime Postgres migration, done right by rattray on Friday June 11, 2021 @04:13PM (#27477881)
I wonder if that was it (Adyen and Braintree are competitors).

Here's the post: https://www.adyen.com/blog/updating-a-50-terabyte-postgresql...

Discussed here: https://news.ycombinator.com/item?id=26535357

They architected their application to be able to tolerate 15-30min of postgres downtime.

Re:Zero downtime Postgres migration, done right by nezirus on Friday June 11, 2021 @01:02PM (#27475811)
pgbouncer has PAUSE comnand, which can be used for seamless restarts/failovers or similar
Re:Zero downtime Postgres migration, done right by simonw on Friday June 11, 2021 @03:21PM (#27477400)
I think that's covered in this talk (I've not watched the video though): https://www.braintreepayments.com/blog/ruby-conf-australia-h...
Re:Zero downtime Postgres migration, done right by pgr0ss on Friday June 11, 2021 @03:38PM (#27477545)
Re:Zero downtime Postgres migration, done right by luhn on Friday June 11, 2021 @05:50PM (#27478923)
Aha, that's the post I was looking for!
Re:Zero downtime Postgres migration, done right by xyst on Friday June 11, 2021 @02:30PM (#27476907)
probably had a good circuit breaker design. would love to read the article though.