News for hackers. Stuff that matters.
Re:Using short lived Postgres servers for testing by whalesalad on Thursday June 27, 2024 @02:51PM (#40813738)
Echoing the other comment here, https://testcontainers.com/ is a great tool for this. (Or just vanilla docker in general)

In our core makefile we have `make pytest` to run standalone unit tests, and then `make pytest_db` to run full database tests using a docker container. The test suite fires up a completely fresh DB, runs migrations against it, and then the test suite proceeds as usual. On a per-module basis a simple fixture is used to determine if the db should get reset for each test, each file, or the entire module.

Works great on Github actions, too.

When the test is done, the container and all noise is automatically cleaned up. No state left behind.

Re:Using short lived Postgres servers for testing by pletnes on Thursday June 27, 2024 @04:04PM (#40814490)
Do you populate with test data, and if so, how?
Re:Using short lived Postgres servers for testing by vhodges on Thursday June 27, 2024 @04:27PM (#40814733)
Yes, when the container is started you get the db url connect string, you connect to it, load the schema and any test data/seeds.

For me I just run the ddl (sitting in a .sql file) and run my tests. They run with an empty db but I could have sample data too the same way.

Re:Using short lived Postgres servers for testing by DasIch on Thursday June 27, 2024 @04:12PM (#40814577)
> On a per-module basis a simple fixture is used to determine if the db should get reset for each test, each file, or the entire module.

That sounds interesting. How do you determine, if the db should get reset?

Re:Using short lived Postgres servers for testing by whalesalad on Thursday June 27, 2024 @05:14PM (#40815195)
At this time we basically reset the db after each module run, but there is some flexibility.

Inside a test module, the developer has the ability to dicate whether the entire module should share the same db state, or if each individual test should get clean slate.

Here is an example of our `test/db/conftest.py` file that is auto-loaded when the specific db test dir is invoked. This essentially bootstraps everything: https://gist.github.com/whalesalad/6ecd284460ac3836a6c2b9ca8...

The main fixture is `database` which is set to autouse at the session level. This produces one db for the entire test session. This could easily be augmented to be scoped to the module or even function level.

You will see `truncate_db` which is currently set to autouse at the module level, so the tables are truncated after each module is finished. This can be totally customized for your own purposes.

I combined a few files to produce this example, but I still think it is pretty concise. We are using PostgreSQL 15.x at this time. Our migrations are perhaps unconventional - we just have a directory in the root called `migrations/` with files like 001_foo.sql, 002_bar.sql ... and we manually run them on deploys. So those same migrations get run sequentially all at once on boot.

The part here that would need to be modified for others are the `service.db` monkeypatching parts. That is our core DB module that everything else utilizes for grabbing a psycopg2 conn from the pool. The test monkeypatches these methods so that the rest of the codebase 'just works' and is handed this conn from the test container versus the traditional one.

Re:Using short lived Postgres servers for testing by canadiantim on Thursday June 27, 2024 @03:55PM (#40814396)
Seems like a great setup, thanks for sharing.
Re:Using short lived Postgres servers for testing by JBorrow on Thursday June 27, 2024 @04:26PM (#40814718)
Yes, I use this kind of setup in a number of places. I am also a fan of running the full test suite using SQLite (if you are using a compatible ORM) to make sure where you are (or not) running into DB-specific behavior.
Re:Using short lived Postgres servers for testing by esafak on Thursday June 27, 2024 @04:28PM (#40814737)
With testcontainers you can use exactly the same database you do in production.
Re:Using short lived Postgres servers for testing by pvorb on Thursday June 27, 2024 @05:13PM (#40815184)
If your application only uses one database in production, you probably don't want to use an ORM, because that might prevent you from using all of your db’s features.
Re:Using short lived Postgres servers for testing by esafak on Thursday June 27, 2024 @07:48PM (#40816429)
The testcontainer is agnostic to how you use it; it's a real docker container.
Re:Using short lived Postgres servers for testing by JBorrow on Friday June 28, 2024 @12:47PM (#40822438)
Sure, but you may want to use different databases as your application may be expected to be deployed in different environments. I like to keep some of my apps db-agnostic, so I test with postgres in a testcontainer and sqlite outside of one to ensure consistency.