All of the major performance problems at scale I've dealt with came down to a super giant table that keeps getting locked.
The fundamental problem with rspec + ActiveRecord is for each test: you create db state (tons of inserts), run the test (more db reads and writes), and then tear down the state. This is very expensive when you have 100,000s of tests each taking 500ms.
Rails/rspec does not make it easy to stub db state with ActiveRecord.
Use transactional fixtures as much as possible. Minimal db usage between tests.
Transaction is started. Run Test 1 Transaction rolled back. (quick)
Test 2 can be run without the expensive fixtures.
Repeat for all tests.
Any test can make use of what is in that database. So you tend to put a bunch of stuff that is useful for most tests.
If a specific test needs an unusual setup, then it can do whatever is needed to load data. Generate data, load extra fixtures, etc. You can do this on a per test basis, or a group of tests, etc.
An option is to reload everything between each test. Not recommended.
Normal usage is to rollback any changes a test did, so that each test has a clean slate. This tends to be plenty fast. Parallelism can usually be done.
If speed is still a concern, you can get fancy and load up multiple databases with fixtures and divide tests between them. Maybe have 2-8 sets of tests running at same time.
It's been a few years, but I've inherited test suites that take 14 hours and been able to get them down to a few minutes with above techniques.
Something like Guard can be setup to automatically run tests relevant to what you're working on. So you don't have to run full suite each time.
Wat? I mean test times aren't great but I've got several thousand specs for what I would call a medium-sized application that run in <1m using TurboTests or <3m using rspec alone. I can't imagine a "tiny" rails app that takes an hour to test. Maybe you're doing a lot of browser rendering or something? My app is API only.
Gems like Fabricator or FactoryBot also make the "create db state" even more excessive, because developers would be lazy and use factory methods that create more than they actually need for the test.
For example, one project I stored a tree-structure that could be modified by api calls. Each test required re-creating the 15 node tree + each node's relational data (think node = user and all users must have a profile, authorization scheme, etc.).
I never figured out how to avoid resetting the db state for each test case.
The two main options to mitigate are:
- just work with in-memory objects (FactoryBot using build or build_stubbed, or just MyModel.new) and stub/mock finders as needed,
- or there's this thing people like to hate that was introduced in Rails 1 called test fixtures, which allow you to prepopulate the database with some initial data and then your test just runs in a transaction and rolls back.
The best suggestion I've heard recently (though I can't recall where to give credit) for managing complexity w/ fixtures is to write an extremely minimal fixture set aiming for 2 instances per model with minimal data required to be valid, and then customize it as part of your test. For a blog example, that means you have two Posts in your fixtures and would write a test around drafts by updating one of them to draft status (be it state, toggle, or publication date) and then test your query, or controller, or whatever. This keeps the data churn inside each test minimal, while keeping the out-of-scope data minimal as well -- if there's only 2 blog posts, all your tests can assume there will be one you care about and one you don't, which is useful for testing filtering/search/visibility/authorization/etc while remaining pretty consistent.
Mocking the finders is obnoxious, since ActiveRecord returns a relation class, not an array. Plus stubbing out relational coupling isn't easy (e.g. the user model might have a `company_name` method that delegates to the company table.).
> - or there's this thing people like to hate that was introduced in Rails 1 called test fixtures, which allow you to prepopulate the database with some initial data and then your test just runs in a transaction and rolls back.
Fixtures and Factories have the same issue are vulnerable to inserting unnecessary data for the test. Models may have validation requirements that must be satisfied to be stored, but completely unnecessary for the test. For example, all users need a `company_id` with a foreign key constraint, so to test _anything_ on user, you have to insert a valid company as well.
Maybe I need to re-evaluate fixtures though, since they would be simpler to run than a factory.
Since the inserts only happen once for the whole test suite, the marginal cost of adding more fixtures is minimal, so it makes sense to just make the fixtures as complete as possible.
In addition to that, the test suite performance was regularly monitored. And like any performance issue, we'd instrument and then fix issues as they came up.
The fundamental design problem for rspec is for each test you create db state (tons of writes), run the test (reads and writes), and then tear-down (truncate). Each test is at least adding an authorized user (and then removing it) at the simplest and at the worst creating complicated db state to support relational models that must be added and torn down with each test (database_cleaner).
Gems like Fabricator or FactoryBot also make the "create db state" even more excessive, because developers would be lazy and use factory methods that create more than they actually need for the test.
I don't exactly agree that it's a design problem. I think there is a lot of value in testing at a high level (controller tests) and having them touch the full stack (down to the DB). Gives a lot of confidence that the code will work in production. Throwing money at it (running the tests in parallel) pretty much solves it.
I have seen FactoryBot use get out of control (creating 8x as many records as you'd expect). That's a really easy way to slow down a test suite :). One way I've found to fix that is by adding tests for the factories, and asserting they are only creating what you want them to.
On model tests, another thing GitHub did well was encourage using `.build` when possible to avoid writing the the DB.
There is a lot of wasted work. Each (similar) test would insert and then delete the same rows, but make a little tweak.
Other languages and frameworks (java, python or golang) don't need to test the db and run super quick. Most db systems do not actually need to be tested. e2e testing can be done manually via curl or qa.
> Throwing money at it (running the tests in parallel) pretty much solves it.
Tests still are slow locally.
> One way I've found to fix that is by adding tests for the factories
That is interesting. I had a meta-test that would limit the number of sql queries a test could trigger.
Is this like when people say you don't need to test functional code?
---
Every tiny rails app that uses rspec ends up with +1hr test times due to tight coupling between activemodel and the db connections.
Do you know of any good blog posts the describe how to do rails testing at scale?
> Companies doing grunt work style web development don't waste their time on flavor-of-the-month.js.
Rails devs have to deal with DHH's-flavor-of-the-month.coffee (coffeescript, asset pipeline, webpacker, import_maps)