Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by sverhagen on Monday October 02, 2023 @01:54AM
(#37734454)
I think I saw an Uber-engineer's presentation (YouTube; can't find it now anymore) about their test environment having the ability to automatically fork the database for every new test run, such that they could easily "revert" (in their case ignore/walk away from) the database after a test run. I find that an appealing idea, since regular test runs in the places where I work, tend to create cruft data in the test enviroment that is lopsided and voluminous in ways that do not properly represent the production system anymore. (And no one wants to troubleshoot performance issues that only ever exist in the test environment.) I have to imagine that pg_branch might be part of building a solution similar to this? Could people speak to some practical applications for this? Maybe I'm thinking about entirely the wrong one here...
[ Parent
| on HN ]
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by petepete on Monday October 02, 2023 @02:24AM
(#37734600)
> about their test environment having the ability to automatically fork the database for every new test run, such that they could easily "revert" (in their case ignore/walk away from) the database after a test run
I think this should be really straightforward by using one database as a template for another. This is how Postgres's create database works and it defaults to Template1
https://www.postgresql.org/docs/current/manage-ag-templatedb...
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by jasfi on Monday October 02, 2023 @03:03AM
(#37734832)
The intention of pg_branch is to optimize this process. They mention two problems:
1. The template database can't have active connections during the CREATE DATABASE process.
2. The CREATE DATABASE from a template database can be slow for large DBs.
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by ilyt on Monday October 02, 2023 @07:21AM
(#37736754)
> The template database can't have active connections during the CREATE DATABASE process.
Why would that be a problem ?
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by dalore on Monday October 02, 2023 @07:39AM
(#37736892)
If you are trying to branch prod (or even a staging db) then you can't
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by ComputerGuru on Monday October 02, 2023 @12:23PM
(#37740353)
Right. But the obvious solution is to just not do that. Create a template and base production/staging off it, then use that template. Or use the same bootstrap/migration process to create the template as you do staging.
Regular tests shouldn’t assume access to staging/prod, anyway.
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by conceptme on Monday October 02, 2023 @01:11PM
(#37741207)
So how do you test your migration? Will work fine with 10 records but perhaps not with 10 million.
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by ComputerGuru on Monday October 02, 2023 @06:06PM
(#37745105)
I don’t consider that “a regular test” but something db-specific that would have a custom workflow.
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by workingjubilee on Monday October 09, 2023 @08:48PM
(#37827341)
Eliminating "custom workflows" seems like rather the point of such an extension.
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by rokkitmensch on Tuesday October 03, 2023 @03:39PM
(#37756543)
Run it in a transaction, and if it fails, abort the transaction. nbd, did this for years in prod. Discovered some interesting assumptions about sequential IDs in the DB tests while running it against prod under peak load, too!
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by dalore on Monday October 02, 2023 @07:39AM
(#37736887)
If you have lots of data, this can take some time
The appeal of this project is that it uses CoW (copy on write), so in theory it should be fast to copy as it only maintains writes.
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by vasco on Monday October 02, 2023 @06:28AM
(#37736322)
We have something like this with gitlab "review apps". Each MR launches an env with its own db that applies migrations and fake data inserters, so each MR has its own db at current state of MR.
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by prpl on Monday October 02, 2023 @09:26AM
(#37737777)
Salesforce does this for customers:
https://developer.salesforce.com/blogs/2022/09/sandboxes-on-...
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by Zanfa on Monday October 02, 2023 @04:20PM
(#37743884)
A common pattern in Ruby-world is running each test case in a database transaction that is rolled back. A simple way to keep a clean slate in your test database.
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by vb-8448 on Monday October 02, 2023 @04:44PM
(#37744161)
It's also quite common in python and I guess in any other language. It's great for unittests, but not suitable for integration or end-to-end tests.
Re:Pg_branch: Pre-alpha Postgres extension brings Neon-like ...
by sverhagen on Tuesday October 03, 2023 @01:49AM
(#37748369)
Been there, done that, but please beware, this means that you're missing out on some nuanced behaviors that you would get if your subject under test would actually commit the transaction. I moved away from this entirely. A more-reliable setup is to let the test commit real transactions, and cleanup afterwards. You might be able to do that by calling "delete" on all repository instances, in the tearDown method of your tests. Or you could use something like Flyway to reset and reinitialize the database entirely before every test. That sounds expensive, and it certainly depends on the kind of project we're talking, but if this is a moderately-sized microservice, and you're running tests scoped to just that microservice, the setup shouldn't be too crazy, and this may actually run pretty smooth. (We run our database in a Testcontainers instance.)