News for hackers. Stuff that matters.
Re:Zero downtime Postgres migration, done right by fullstop on Friday June 11, 2021 @12:34PM (#27475426)
I work for a small company and it would be devastating if our database was down for a few hours.
Re:Zero downtime Postgres migration, done right by silviogutierrez on Friday June 11, 2021 @12:48PM (#27475637)
Unplanned? Probably. But maybe it's not as a dire as you think? Azure Active Directory (auth as a service) went down for a while, globally, and life went on. Same with GCP and occasionally AWS[1]

I'm not saying there's no downside, I'm asking against the downside of engineering cost. And that itself carries risk of failure when you go live. It's not guaranteed.

[1] AWS is so ubiquitous though, that half the internet would be affected so it makes it less individually noticeable.

Re:Zero downtime Postgres migration, done right by dividedbyzero on Friday June 11, 2021 @03:43PM (#27477588)
So don't host on AWS, wait out their next big outage and then take down your app for that big migration, so you can hide among all the other dead apps. Half joking, of course.
Re:Zero downtime Postgres migration, done right by silviogutierrez on Friday June 11, 2021 @03:50PM (#27477651)
That's brilliant. Have the PR ready to go and merge/deploy only when that happens. Just host on us-east1 and it'll happen eventually.
Re:Zero downtime Postgres migration, done right by willcipriano on Friday June 11, 2021 @12:50PM (#27475663)
In my view that's just saying at some point in the future the company will be devastated. Nothing has 100% uptime.
Re:Zero downtime Postgres migration, done right by hobs on Friday June 11, 2021 @12:50PM (#27475660)
OTOH I worked for several small companies that had "business hours" software - if you had to take downtime on the weekend or after hours you'd be looking at impacting <1% of users.
Re:Zero downtime Postgres migration, done right by jbverschoor on Friday June 11, 2021 @12:49PM (#27475647)
I doubt that.. aws is down sometimes too. Card payments are sometimes down, even 112(911) is sometimes down.

I hope you have everything redundant, including the CEO

Re:Zero downtime Postgres migration, done right by xcambar on Friday June 11, 2021 @02:11PM (#27476689)
> I hope you have everything redundant, including the CEO

Made me laugh, I'll sure reuse it!

Re:Zero downtime Postgres migration, done right by fullstop on Friday June 11, 2021 @12:58PM (#27475754)
The database has a hot standby and two warm standby instances, redundant hardware, network infrastructure, all of that.
Re:Zero downtime Postgres migration, done right by chefkoch on Friday June 11, 2021 @12:54PM (#27475706)
Then hopefully everything is HA in your company.
Re:Zero downtime Postgres migration, done right by fullstop on Friday June 11, 2021 @01:00PM (#27475779)
It absolutely is, BGP with geographically diverse paths, databases, app servers, etc, are all redundant. It's hosted in-house, so there is a cold standby database in AWS which would only be used if, say, an aircraft crashed into our server rooms.
Re:Zero downtime Postgres migration, done right by ikiris on Friday June 11, 2021 @01:17PM (#27476005)
It always amazes me when someone says they can't ever be down, and then says they only serve from one physical location actively.

One of these things is not like the other.

Re:Zero downtime Postgres migration, done right by fullstop on Friday June 11, 2021 @01:44PM (#27476338)
We have everything in place to run from AWS if needed but do not operate from there because of cost.

This goes both ways -- there have been many AWS outages which have not affected us. I hear what you're saying, but we've had only one instance of extended (hours) downtime in the last 20 years.

Re:Zero downtime Postgres migration, done right by xcambar on Friday June 11, 2021 @02:15PM (#27476725)
The more experienced I become, the more such down to earth solutions seem OMG SO MUCH MORE reasonable than all the bells and whistles of "modern" engineering practices.
Re:Zero downtime Postgres migration, done right by ohyeshedid on Friday June 11, 2021 @09:16PM (#27480449)
Out of curiosity, what do you use in AWS to keep a running live backup of the data you'd need to migrate in the worst case scenario?
Re:Zero downtime Postgres migration, done right by fullstop on Friday June 11, 2021 @10:25PM (#27480876)
For postgres we send WAL files to a server in AWS which processes them. To bootstrap the database initially we sent zfs snapshots, and those WAL files are applied on an ongoing basis. If our data center were to die a horrendous fiery death, we could lose, at most, about 3 minutes of data although monitoring shows that it's closer to 30s under normal operating conditions.

For the app servers, saltstack is used and we synchronize that repository with what is needed to reproduce a production environment in AWS.

Obviously we'd have to provision servers, etc, but it's all possible in a worst-case scenario.

Re:Zero downtime Postgres migration, done right by jnpatel on Friday June 11, 2021 @01:11PM (#27475936)
Could you share what domain your company is in? Just curious what has such HA requirements