When your TDD approach revolves around integration tests, you have complete freedom to add, remove and shift around internal components. Having the flexibility to keep moving around the guts of a system to bring it closer to its intended behavior is what software engineering is all about.
This is also how evolution works; the guts and organs of every living creature were never independently tested by nature.
Nature only cares about very high-level functionality; can this specimen survive long enough to reproduce? Yes. Ok, then it works! It doesn't matter that this creature has ended up with intestines which are 1.5 meters long; that's an implementation detail. The specimen is good because it works within all imposed external/environmental constraints.
That's why there are so many different species in the world; when a system has well defined external requirements, it's possible to find many solutions (of varying complexity) which can perfectly meet those requirements.
Unit tests on the other hand are useless at identifying complex issues like race conditions; they're only good at detecting issues that are already obvious to a skilled developer.
* I restart the system from scratch between each test case but if that's too time-consuming (I.e. more than a few milliseconds), I just clear the entire system state between each test case. If the system is truly massive and clearing the mock state between each test case takes too long, then I break it up into smaller microservices; each with their own integration tests.
I mistunderstood your Integration tests. I understand them as something like checking from the end user contract. If so, without TDD, it won't be possible to track down the bug easily as with TDD.
Mock based testing however is the only thing I've ever encountered that forces me to think very, very clearly about what my code is doing. It makes me inspect the code and think about what dependencies are being called, how they're being called, and why they're being called.
I have found that this process is extremely valuable for creating code that is more elegant and more correct. I value mock tests not for the tests that I end up with at the end, but for the better production code that I wrote because of them.
Or they don't. If the tests just continue to work, then you ask yourself whether the change should have broken them. If not, you go on your way with a fair amount of confidence. But if the change should have broken the tests, then you have to look at why it didn't...
While yes, unit tests do have a maintenance burden, they are often reproducible, less flaky, give you targeted debug information, and run extremely fast.
There are heavy costs to integration and e2e testing that often gets dismissed by developers who often have not experienced the fast feedback loop a good fat unit suite gives you.
E2E UI tests are definitely hard to debug though, in most cases. They still have value, but I’m a fan of the “testing pyramid” here. A small number of E2E UI tests, a decently large number of integration tests that each try to test a single contract between 2 services, and a tonne of unit tests to cover sad paths, detailed behaviour, etc.
A few months ago, I did a contracting job and the team I worked with used the "mock everything" approach, without even one integration test, which to me seemed crazy (especially for a component which was calle "integration layer").
I tried hard to find the advantages in this approach, studying what the rationale and the best practices were, and questioned my previous assumptions. In the end, I had to confirm those assumptions: even if there were hundreds of tests and they all passed, many logical errors weren't caught.
Even worse, they gave a false sense of confidence to the team, and made refactoring super-slow. But the team leader was super-convinced it was the best idea since sliced bread.
Additionally, people with large state machines with too-complex sets of possible states (games, big frontends without a top level state management system) tend to only unit test because it’s frequently too much of a pain to set up an integration runtime environment. Places with lots and lots of manual QA testers.
A lot of people overuse mocks when testing. In fact, using mocks enforces coupling between different methods because a lot of people use them to assert that a method with a specific name was called with specific parameters or they create one that asserts a method with a certain name returns a certain value. So when one wants to refactor, they not only have change the code; they need to update all the mocks that reference it as well.
I've found that a better way to structure code is to take the result of an external dependency and pass it in as a parameter to a method that will process it. Then when I unit test that method, I just pass in what I expect from that dependency and assert on the return value of that method. I don't try to unit test the outer method that calls the dependency by creating a mock call for it.
> Then I started writing integration testing while working on converting a bunch of code recently, and it has been eye-opening. Instead of testing individual models and functions, I was testing the API response and DB changes, and who really cares what the code in the middle does and how it interfaces with other internal code?
It makes it easier to isolate the cause of the error rather than having to search to the entire call chain to find it (especially if it's a logic error that doesn't result in an exception). Plus, integration test suites take a lot longer to run and can have timing issues due to caching or other reasons which can result in sporadic failures.
I am fan of Gary Bernhardt ever since viewing this talk.
For GUIs, the proper approach is to unit-test the functionality underneath, not the GUI itself.
After all, any program out there does something. If you know what it's doing, you know what it should do, and what it shouldn't. That means you can test it.
After all, writing the test needs to be possible, to start with.
So adapters, views, commands and what have you need to exist only to fulfill such purpose, and even then, their interactions with the GUI layer don't get tested.
So one is creating them, without knowing if they are the right elements for the GUI layout.
Hence why testing, 100% behind it, TDD not so much.
This isn't to say that integration tests aren't great, too. They all tell different stories. I like unit tests, as they force me to think about modules as units. If a unit is hard to test or brittle, it may need more attention to its design.
As the codebase gets larger and more complex (and interesting!), I want unit tests to fail because of small changes. That's actually useful feedback, whereas the simple, brutal failure of an integration test is just not granular enough to quickly help me understand the details of the change.
For future tests and projects, unit tests do have a place and I'll make better use of them.
A good learning experience, none the less.
They serve two different purposes.
Unit tests scale a lot better. That's why most generally use a pyramid structure: lots of unit tests, a moderate amount integration tests, and a few end-to-end tests.
Mentally, something like clojure’s ring framework make this easier to grasp: the abstraction it provides is dictionary-in, dictionary-out. Once you have something like this, there’s no need to spin up a web server to do integration testing: you just shove a bunch of dictionaries and and make sure the output dictionaries are what you expected.
A good approach is to use each technique where you get the most bang-for-buck: use unit tests only for pure functions, and decouple your system in a way that integration tests can be reduced to simple data-in data-out (which also makes them very fast)
If your integration tests can be reasonably be set up with a couple containers, great, but not every system is that flexible. And not every data store is that simple to provision.
Knowing about that bug is really valuable knowledge and not adding a test for it is basically like throwing away that knowledge instead of sharing it for future people working on the project.
Can you or others speak more about this? I was taught that verifying function calls for spies/mocks was good practice. But, I encountered this problem just the other day when I refactored some Java code for a personal project. Everything still worked perfectly, but, exactly as you said, the intermediate function calls changed so the tests would fail due to spies/mocks calling different "unexpected" functions.
I'm an intermediate programmer so can someone with more experience fill me in with what's best practice here and why? Do I update the test code to reflect the new intermediate function calls? But this whole approach now seems silly since a refactoring that doesn't affect the ultimate behavior of the function that is under test will break the test and that seems wrong. So do I instead not verify function calls when using spies/mocks? In that case, what is the use case for verifying spies/mocks?
If you have a function that fetches data, you shouldn't test that its hitting the data layer, only that the correct data is returned. This way when you improve the function to not hit the data layer at all under some conditions, your tests will keep passing.
On the other hand, if that function is supposed to log metrics or details about its execution, you should test that ,as it is part of its contract and can't be inferred from the return value.
The important part is to remember that not every function is a component, and not even every class by itself. Where to draw the boundary between components is itself an explicit design decision, and should be made consciously, not mechanically.
Instead of mocks, some people prefer to build fakes / stubs which are versions of a dependency which are "fully operative" in a sense but with a simplified internal implementation. For example a repository that keeps entities in memory. (Not the same as an in-memory database! The fake repository wouldn't use SQL at all.)
Tests would check the final state of the fakes after the interactions, or simply verify that the values returned by the tested component are correct.
The hope is that fakes, while possibly more laborious to set up, allow an style of testing that focuses less on the exact interactions between components, and therefore is less brittle.
Some links:
- Mocks aren't stubs https://martinfowler.com/articles/mocksArentStubs.html#Class...
- From interaction-based to state-based testing http://blog.ploeh.dk/2019/02/18/from-interaction-based-to-st...
Also, many of the assertions that are often put into unit-tests (especially at stub/mock boundaries) are better formulated as invariants, checked in the code itself. Design-by-contracts style pre/post-conditions is a sound and practical way of doing this. When this is done well, you get the localization part of low-level unit testing even when running high-level tests. Plus much better coverage, since these things are always checked (even in prod), not just in a couple of unit tests. And it is more natural when refactoring internal functions to update pre/post-conditions, since they are right there in the code. When a function disappears they also do.
I don't like the term "integration" tests though, as they hint at interactions between systems being the important thing to test. Integration between services / subsystems are just as much a detail as internal function calls. If using the real system during test is too complicated or slow, maybe it should be simplified or made faster? Only when that is not feasible do I build a mock.
The trick for me is focusing on what the unit does from a consumers perspective. Avoid testing implementation details (unless they are important side effects), and test the behavior that does not change. If you do that, then refactoring becomes easier, because tests will only break when the contract of the unit changes.
We have a lot of data that explicitly shows that automated unit-testing doesn’t work. One good example is one of our ESDH systems, which changed supplier in a bidding war, partly because we wanted higher code-quality.
It’s a two million line project, we paid for 5000 hours worth of refactoring and let the new supplier spend two years getting familiar with it and setting up their advanced testing platforms.
So far we’ve seen some nice performances increases thanks to the refactoring. It has more problems than it did before though, even though everything is tested by unit tests now and it wasn’t before.
We have a lot of stories like this by the way. I don’t think we have a single story where unit-testing actually made the product better, but we do have tested systems where we couldn’t say because they always had unit-testing.
Ironically we still do TDD ourselves on larger projects. Not because we can prove it works, but because everyone expects it.
At best you have data that shows that a poor unit testing implementation failed to deliver. A bad experience doesn't prove a whole tech strategy doesn't work when the whole world shows otherwise.
Because your users should not be your testers and you must catch bugs before deploying them in production.
You may be able to deploy in 3 minutes but it takes way more to debug and fix your bugs.
Testing helps the project to catch bugs prior to deploying them and also provides the infrastructure to avoid regressions.
Write tests. Not too many. Mostly integration.
Use system/integration/functional/unit tests wisely.
For precise stateless stuff, like making sure your custom input format parser/regexp covers all edge cases, I prefer unit tests - no need to init/rollback database state.
If a unit test fails when the code under test hasn't functionally changed then it means that your unit tests are flaky.
Unit testing to me seemed akin to drinking 8 glasses of water every day. A lot of people talk about how important it is for your health, but it really tends to get in the way, and it doesn't seem to really be necessary. Too frequently, code would change and mocks would need to change with it, removing a good chunk of the benefit of having the code under test.
Then I started writing integration testing while working on converting a bunch of code recently, and it has been eye-opening. Instead of testing individual models and functions, I was testing the API response and DB changes, and who really cares what the code in the middle does and how it interfaces with other internal code? So long as the API and DB are in the expected state, you can go muck about with the guts of your code all you want, while having the assurance that callers of your code are getting exactly out of it what you promise.
Unit test suites would break all the time for silly reasons, like someone optimizing a function would mean a spy wouldn't get called with the same intermediary data, and you'd have to stop and go fix the test code that was now broken, even though the actual code worked as intended.
Integration tests (mainly) only break when the code itself is broken and incorrect results are getting spit out. This has prevented all kinds of issues from actually reaching customers during the conversion process, and isn't nearly so brittle as our unit tests were.