I’ve only ever worked in very small teams, where we didn’t really have the resources to maintain nice developer experiences and testing infrastructure. Even just maintaining representative testing data to seed a test DB as schemas (rapidly) evolve has been hard.
So how do you
- operate this? Do you spin up a new postgres DB for each unit test?
- maintain this, eg have good, representative testing data lying around?
we have the same strategy for testing against Kafka., etc.
where we care about data, we seed the db with data for a specific group of tests. Otherwise, we just nuke the db between each test.
Prior to doing this, we'd use in-memory db for tests, and a real db for runtime, using JPA / Hibernate to make things transferrable. But this was leaky, and some things would pass in tests then fail at runtime (or vice versa)
TestContainers has been so much better, as we're running against a real version of the database, so much smaller chance of test and runtime diverging.
Yes.
Typically you'd keep the database container itself alive, and you would run the schema migrations once at startup. Then your test runner would apply fixtures for each test class, which should set up and tear down any data they need to run or that they create while running. Restarting the database server between each test can be very slow.
The test data is a harder problem to solve. For unit tests, you should probably be creating specific test data for each "unit" and cleaning up in between each test using whatever "fixture" mechanism your test runner supports. However, this can get really nasty if there's a lot of dependencies between your tables. (That in and of itself may be a sign of something wrong, but sometimes you can't avoid it or prioritize changing it.)
You can attempt to anonymize production data, but obviously that can go very wrong. You can also try to create some data by using the app in a dev environment and then use a dump of that database in your tests. However, that's going to be very fragile, and if you really need hundreds of tables to be populated to run one test, you've got some big problems to fix.
Property-based testing is an interesting alternative, where you basically generate random data subject to some constraints and run your tests repeatedly until you've covered a representative subset of the range of possible values. But this can be complicated to set up, and if your tests aren't fast, your tests can take a very long time to run.
I think at the end of the day, the best thing you can do is decouple the components of your application as much as possible so you can test each one without needing giant, complicated test data.
Generally I've seen a new database (schema in other dbs?) in postgres that is for testing, i.e "development_test" vs "development". The big thing is to wrap each of your tests in a transaction which gets rolled back after each test.
> maintain this, eg have good, representative testing data lying around
This is much harder. Maintaining good seed data - data that covers all the edge cases - is a large amount of work. It's generally easier to leave it up to each test to setup data specific to their test case, generalizing that data when possible (i.e if you're testing login endpoints, you have all your login test cases inherit from some logic specific data setup, and they can tweak as needed from there). You will end up with duplicated test setup logic. It's not that bad, and often you don't really want to DRY this data anyways.
That being said, if you have the time and resources to maintain seed data it's absolutely a better way to go about it. It's also beneficial outside of tests.
Every place I've ever worked which tried this has managed to get a production database deleted by somebody running tests.
Why not?
But, the one important thing is, do not give people direct access to production. And for the few that must have it, it should not be easy to connect to it.
That's just a different way of saying "Every place I've ever worked at use production for testing" :-/
TBH, any place using the same credentials for test and for production have bigger problems than would ever be fixed by mocking.
... is that good? Hell no. But it's a much more common version of reality than you're assuming is happening.
Sure I've made an assumption, but in my defence it's a single reasonable assumption: that one wouldn't be running tests in production.
If you have a leftover `.env` that contains production credentials, you were running your test in production.
Anyway, the missed point is that you can’t just do anything in tests and just expect the best of intentions to ensure it doesn’t backfire. One must also consider the security situation, infrastructure, secrets management, shared environments, and more. It’s not as simple as just plopping down a test database and expecting everything to go smoothly. You wouldn’t be that careless with other things, so don’t do it with tests, and don’t rely on “don’t run tests in production” as your only safeguard.
Obviously, there are some caveats, e.g.:
* While this approach works perfectly for some tests (load testing, performance testing, …), it does not work for others (e.g. unit testing).
* You have to be careful about PII, and sanitize your data.
Just run PostgreSQL on your local machine, connect to that, setup a new schema for every test (fairly cheap-ish) inside a test database.
def Test1:
setupdb()
obj1 = createObj1()
obj2 = createObj2()
have = doStuff(obj1, obj2)
if have != want: ...
def Test1:
setupdb()
obj = createObj1()
have = doOtherStuff(obj1)
if have != want: ...
1. It's easy to create the objects you need
2. Your creation functions are well tested so that the rest of your tests can rely on them.
If you have spotty coverage or just poorly defined creation semantics, or it's a bunch of calls to functions all over the place just to set up your test data, then this doesn't work.
But the solution typically isn't "write a bunch of JSON mock test data", it's to solve those problems.
idk if you've solved this, but PG doesn't like to bind to 0, so you have to manage ports. And I've had issues with processes sticking around if the test driver has crashed (I dont currently, but i'm turning off setsid in postgres).
Start it once across a bunch of suites, and have each suite manage its DB state. Done deal.
1. Init the DB with some "default" data - configuration, lookup tables, etc
2. Each test in the test suite owns its data. It creates a new account and inserts new records only for that account. It can for example create users on this account, new entities, etc. It can run multiple transactions, can do rollbacks if needed. It is important to only touch the account(s) created by the test and to avoid touching the initial configuration. There's no need to clean up the data after the test finishes. These tests can run concurrently.
3. Create a separate integration test suite which runs sequentially and can do anything with the database. Running sequentially means that these tests can do anything - e.g. test cross-account functionality, global config changes or data migrations. In practice there aren't that many of those, most tests can be scoped to an account. These tests have to clean up after themselves so the next one starts in a good state.
Other approaches had tons of issues. For example if each test is wrapped with a transaction which is later rolled back then testing is very limited - tests cannot use transactions on their own. Savepoints have similar issue.
> - maintain this, eg have good, representative testing data lying around?
This can be tricky, but usually my advice is to never even being trying to do write seed data in the database unless its very static. It just gets annoying to maintain and will often break. Try to work out a clean way to setup state in your tests using code, and do not rely on magic auto increment ids. Some of the more effective ways I have found is to f.ex. have every test create a fresh customer, then the test does work on that customer. Avoid tests assuming that the first object you create will get id == 1, makes it very annoying to maintain.
There's times when a big test fixture can provide value, but it's very context dependent and almost never for smaller/micro tests.
However, it’s not always possible.
For example:
- you use oracle db (takes minutes to start, license, hope the containers run on ARM fine, etc.) - sometimes an in memory fake is just much faster, and can be an official db on its own for people to try the product - your storage might be only available through a library by a third party provider that is not available locally.
Doesn't solve your testing-data question, but it'll save you from spinning up a new DB.
When you write mock data, you almost always write "happy path" data that usually just works. But prod data is messy and chaotic which is really hard to replicate manually.
This is actually exactly what we do at Neosync (https://github.com/nucleuscloud/neosync). We help you anonymize your prod data and then sync it across environments. You can also generate synthetic data as well. We take care of all of the orchestration. And Neosync is open source.
(for transparency: I'm one of the co-founders)
A major benefit of doing separate schemas for each test is that you can run them in parallel. In my experience, unless you have a metric ton of migrations to run for each test, the fact that your database tests can now run in parallel makes up (by a lot!) for the time you have to spend running the migrations for each test.
EDIT: usually we also make utilities to generate entities with random values, so that it's easy to make a test that e.g. tests that when you search for 5 entities among a set of 50, you only get the 5 that you know happen to match the search criteria.
Once that gets a bit too slow, running migrations once before every suite and then deleting all data before each test works really well. It's pretty easy to make the deleting dynamic by querying the names of all tables and constructing one statement to clear the data, which avoids referential integrity issues. Surprisingly, `TRUNCATE` is measurably slower than `DELETE FROM`.
Another nice touch is that turning off `fsync` in postgres makes it noticeably faster, while maintaining all transactional semantics.
This one can be very easy, depending on the kind of data you're working with. Many places shall simply dump a part (or the whole if it's not too big) of the production DB into dev and pre-prod environments.
Now if there are sensitive, non-encrypted, data that even the devs cannot see, than it can get tricky (but then arguably they cannot see the logs in the clear either, etc.).
But yeah: a recent dump of the prod DB is good, representative data.
I've worked at places where pre-prod had a daily dump of the prod DB. Simple.
No need to spin up a new server, not in a container, not in a new directory, not at all. It's pointless busywork with too many extra points of failure.
Nothing is stopping you using an existing server and creating a new DB, which takes about 1/100th the time that starting up a new server (whether in Docker or otherwise) takes.
Secondly, I don't actually do unit-testing on the database layer - there's little point in it. I test workflows against the database, not units!
What I do is create multiple 'packages' of tests, each with multiple tests. A single 'package' creates a temp db, runs its tests sequentially and then drops the temp db. Each package will setup itself with SQL statements.
This lets the tests perform tests of actual workflows, instead of testing in isolation that an object can be (de)serialised. IOW, I can test that the sequence of `addUser(); setProfilePassword(); signIn(); viewInfo(); signOut();` work as expected, and that `removeUser(); signIn();` fail with the correct error.
I’ve never been thrilled by tests that rely on mocking — it usually means you need to re-express your module interface boundary.
Mocks for me fall into the class of software I affectionately call “load-bearing paint.” It’s basically universally the wrong tool for any given job but that really doesn’t stop people. Putting in a data class or similar model object and a delegate is usually sufficient and a much better tool.
Your way means you only ever have siblings. With an orchestrator pulling results out of one module and pushing it into another.
Maybe you mean "less surface to mock", which is irrelevant if you generate your mocks automatically from the interface
You define fakes in this case, not mocks https://blog.cleancoder.com/uncle-bob/2014/05/14/TheLittleMo...
For what it’s worth, I find the distinction between mocks, fakes, spies, stubs, dummies and whatever completely useless in practice, the whole point is to control some data flows to test in isolation, it’s really all that matters.
Fun thing this kind of bikeshedding comes from Bob Martin, who famously has nothing worthwhile to show as an example of his actual work. Every time I read something from him, I get more and more convinced he’s a total fraud.
Personally, I like seeing "dummy" name passed into a function and understanding just from the name that no methods are expected to called on it during the test. Or seeing "fake_something " and understanding that a manual (perhaps test-specific) implementation is used. It is a minor point but such chunking is universally useful (there is enough stuff to keep track of).
My original point was that the architecture can facilitate so called classical unit tests over mocks (make the latter less necessary) https://martinfowler.com/articles/mocksArentStubs.html#Class...
I find mocks useful for testing conditions that are on the periphery and would be a decent amount of trouble to set up. For instance, if I have a REST controller that has a catch all for exceptions that maps everything to a 500 response, I want a test that will cause the DAO layer to throw an exception and test that the rest of the "real stack" will do the translation correctly. A mock is the easiest way to accomplish that.
And then you mock the delegate?
Every tests seeds all the data needed to run (user, org, token), it requires an initial setup but then you just reuse it everywhere, and voila. No side effects, no mock to maintain, it also test your auth and permissions, almost 1:1 with prod.
Can also be used to test version updates of your DB.
I've been doing E2E testing using 3rd-party APIs for a decade now, and this has yet to be a significant problem. The majority of my APIs had a dedicated sandbox environment to avoid "rate limiting, bans, and other anti-abuse mechanisms". The remainder were simple enough that the provider didn't care about users exploring on the live API, and were usually read-only as well.
Did I run into the occasional flaky failure, or API stability issues? Sure. But it was very rare and easy to workaround. It never devolved into becoming "untestable" or "full manual QA"
My other teams that relied on mocks suffered from far worse problems - a ton of time being spent on manual-QA, and bugs that leaked into production, because of mock-reality mismatches.
I don't find mocks for this kind of thing very helpful, because what you're really testing for are things like changes to how an API changes over time -- you need real API calls to see this.
you also need to test the mocks against the real thing separately.
What you're describing sounds like a fake to me.
It's important to have a contract testing layer in place to make sure your test doubles are still behaving like the real thing, though.
I thought interested mostly fizzled out over the years after the initial hype died down
That was always the go-to for me when testing against 3rd party services, especially because the tests would then survive the offboarding of the engineer who set them up with their personal credentials.
If your test suite relies on live Github PATs or user-specific OAuth access tokens, then you can either figure out how to manage some kind of service account with a 'bot' user, or live with things breaking every time someone leaves the org.
Services that incur a per-request charge, or consume account credits, are another problem. Especially if they don't have sandboxes.
https://github.com/dolthub/dolt
We use the reset functionality to speed up our tests.
Actually that's wrong too. The production database will be different than the "testing Postgres instance", leading to bugs.
It turns out that whatever testing solution you use, if it's not the actual production instance and you're not using real production data, there will be bugs. Even then there's still bugs.
This is the simple truth: you can't catch all the bugs. Just put in Good Enough testing for what you're doing and what you need, and get on with life. Otherwise you will spend 99% of your time just on testing.
It never happened to me to be honest. This reads an argument for "if you can’t do perfect, just do it badly" but it’s nonsense. Running tests against a local Postgres instance with the same major.minor version and same extensions as your prod instance WILL work.
And testing your storage layer against the database is probably the most reliable safety net you can add to an app.
A team I worked with recently said the same thing. But, as I predicted, they ran into bugs because the CloudSQL Postgres was different than their Dockerized Postgres, even though it was the same core version.
There will always be testing problems you can't anticipate. Especially with systems that are not your own code. Just be ready to adapt your testing when it doesn't work as expected, and don't invest too much in the testing if it's not worth it.
I’ve been ready for 5 years as it’s the duration I’ve been testing my storage layers with 90% integration tests (the 10% being the hard to reproduce error cases, these tests have low value but are easy to test with mocks so I still test them). The only issue I’ve encountered was with time zones (shocking), and it made me ensure I got full control of the time zones in my app, in my deployment, in my local Postgres and in my prod Postgres, so net benefit in the end.
- Access to different parts of the database are limited in CloudSQL since it's a managed database. This makes some features of automated tooling (like migrations) not work on CloudSQL (i'm not saying migrations don't work, i'm saying some features do work, some don't). Sometimes elevated permissions can fix it, but some aspects are just walled off.
- There are schema differences from the stock Postgres (I don't remember specifics). No support for custom tablespaces.
- Import/export can often lock up a CloudSQL instance, whereas it might work fine on a local instance.
- You only get Read Committed transaction isolation.
- Operation is different (load, logging, replication, nodes, etc) and this affects performance, and performance can lead to bugs if there's an expectation of a certain run time or performance that doesn't match up with the development experience. Often times some job will have run faster on a laptop than in the cloud, and that leads to weird corner cases where production has an issue due to weird performance and it has to be triaged in the cloud and a fix applied to avoid it. Performance issues don't sound like a bug, but if you have to change your code to make it work in prod, it's a bug.
- To access CloudSQL they want you to use a proxy app, and that can have inconsistent results compared to a dev connecting directly to a Postgres instance. Even something as simple as handling reconnects is often not considered when all you have is a local instance that never needs reconnecting.
- There's a limited selection of Postgres extensions supported. And I could be wrong but I think the version of the extensions is pinned for each version of Postgres core used in CloudSQL.
To all of this you might reply "well none of that affects me", and that's fine... but it does affect other people, and that's important to note when you're telling people on the internet there will be no problems.
If my prod were using CloudSQL, I'd use CloudSQL for tests too. Haven't noticed so many differences between Heroku Postgres and stock.
After figuring it out, I swapped us over to running all the tests that hit the database against the real database, in a testcontainer, with a RAM disk for minimizing query latency. It was about a day's worth of work, and turned up a few other bugs that hadn't bit us in production yet, too. Also sailing past our test suite because the test double failed to accurately replicate the behavior in question.
Total time to run CI went up by about 10 seconds. (For local development you could chop that way down by not starting a fresh server instance for every test run.) Given how many person-hours we spent on diagnosing, resolving, and cleaning up after just that first defect, I estimated the nominally slower non-mocked tests are still a net time saver if amortized over anything less than about 50,000 CI runs, and even then we should probably only count the ones where an engineer is actually blocking on waiting for the tests to complete.
That said, there was a time when I thought test doubles for databases was the most practical option because testing against real databases while maintaining test isolation was an unholy PITA. But that time was 5 or 6 years ago, before I had really learned how to use Docker properly.
The same logic does not apply to all ORMs, of course, which do not all benefit from the same quality of professional dev support that EF receives from MS. But that's my main point from above: the correct design decision depends on the context. For services written in other languages with other ORMs or raw SQL, I absolutely will spin up a full Postgres test container because it is indeed trivial (have one running in the background on my laptop right now in fact). It just isn't necessary in the specific context of EntityFramework code.
I love EntityFramework, it's easily the best ORM I have ever used but it has a few cross-platform footguns that require testing against the actual database service you're using.
I ended up a few times with discrepancy in the format (excel, .net and windows ) because someone changed it.
Can you please explain what you did with a RAM disk to speed them up?
Those rate limits, bans, and other anti-abuse mechanisms are things that would be good to uncover and account for during tests. Better for the test suite to detect those potential failures than the production deployment :)
If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at worst. Some APIs are very stable, and testing against the API itself can hit rate limiting, bans, and other anti-abuse mechanisms that introduce all kinds of instability to your test suite.
Use the right tools to solve your problems.