News for hackers. Stuff that matters.
Re:Updating a 50 terabyte PostgreSQL database (2018) by ComodoHacker on Monday March 22, 2021 @04:53AM (#26538283)
This indeed means downtime on database level. But their system design allows processing transactions with DB down for some time. I guess they are caching account balances and limits in some in-memory DB. This is a common design choice for payment processing systems.

Sadly, PostgreSQL doesn't have a built-in solution for online upgrade yet. This is one of the few points why commercial DBMSes are often worth their money.

Re:Updating a 50 terabyte PostgreSQL database (2018) by easton on Monday March 22, 2021 @09:16AM (#26540335)
I wonder what the break even point is for just that one feature. 15 minutes of downtime per year, which you can schedule and maybe work around, or five-six figures to Oracle or Microsoft for them to already have solved this problem, but otherwise you don’t get much else (that you need).
Re:Updating a 50 terabyte PostgreSQL database (2018) by ComodoHacker on Monday March 22, 2021 @09:28AM (#26540476)
It could be more than one downtime per year, if we take security patches into account.
Re:Updating a 50 terabyte PostgreSQL database (2018) by darksaints on Monday March 22, 2021 @09:45AM (#26540691)
Security patches are minor upgrades...the upgrades only cause significant downtime if there are large changes in on-disk representation, which shouldn't happen for a security patch.
Re:Updating a 50 terabyte PostgreSQL database (2018) by zozbot234 on Monday March 22, 2021 @12:12PM (#26542807)
> Sadly, PostgreSQL doesn't have a built-in solution for online upgrade yet.

Surely you could just set up a logically-replicated standby server running the new PG version and then hot-failover to the standby, turning it into the new master. That should preserve availability throughout the upgrade.

Re:Updating a 50 terabyte PostgreSQL database (2018) by 411111111111111 on Monday March 22, 2021 @12:55PM (#26543391)
That's how they did it in 2015. Read the article if you want to know why they decided to change this in 2018.

Changing master for upgrades works great if youre not at their scale though.

Re:Updating a 50 terabyte PostgreSQL database (2018) by darksaints on Monday March 22, 2021 @09:41AM (#26540630)
> Sadly, PostgreSQL doesn't have a built-in solution for online upgrade yet. This is one of the few points why commercial DBMSes are often worth their money.

I was about to say the same, as much as it pains me to say it. I wonder if Citus or EnterpriseDB has any solutions to a problem like this? Can Citus run temporarily with heterogeneous nodes? E.g. some nodes upgrading with other nodes operating?

Re:Updating a 50 terabyte PostgreSQL database (2018) by mathnode on Monday March 22, 2021 @11:46AM (#26542448)
MariaDB has offered online schema changes in InnoDB for several years now with continous improvement.

Their (currently commercial only) engine, Xpand (previously known as Clustrix), offers distributed SQL as well as online schema changes.

No need for an unncessary database migration if you are already on LAMP and the like to get these fancy features. However handling any kind of change should be handled with care, especially on such a large dataset. Test your rollback procedures!

Disclaimer, I work for MariaDB.

Re:Updating a 50 terabyte PostgreSQL database (2018) by peterthehacker on Monday March 22, 2021 @12:30PM (#26543060)
This is about an online database version upgrade not a schema upgrade / migration.
Re:Updating a 50 terabyte PostgreSQL database (2018) by jdmichal on Monday March 22, 2021 @11:32AM (#26542245)
> I guess they are caching account balances and limits in some in-memory DB. This is a common design choice for payment processing systems.

They are a payments processor, not an issuer. That means they are on the merchant's side of the transaction, not the purchaser's. Their role in authorization is simply routing. The issuer is responsible for performing the authorization. The only balance they need to track is how much money they owe the merchant.

Re:Updating a 50 terabyte PostgreSQL database (2018) by datavinci on Monday March 22, 2021 @10:02AM (#26540881)
Sadly, PostgreSQL doesn't have a built-in solution for online upgrade yet. -> Hey coould you explain by what u mean here? Thanks.
Re:Updating a 50 terabyte PostgreSQL database (2018) by greggyb on Monday March 22, 2021 @11:18AM (#26542057)
There's no way to upgrade from one Postgres version that keeps the database up and responding to arbitrary queries for the duration.