I've been occasionally pushing coworkers to restructure code and tests into a higher level style, defining an API boundary and calling it from the rest of the code (as if you're writing a library except it's inside your codebase instead of installed), then writing tests to that API. That does make for some easily refactorable code where the tests don't need to change at all.
That pseudo-library style is kind of what "unit test" originally meant: a business unit, not a code unit. Examples simplified it too much and people copied the style instead of the substance, and the original meaning was lost.
For another example, some of our recent projects have been on updating daemons, those have also been really good candidates for this since there are clear entry and exit points.
Very interesting. It sounds a bit like what happened with the term "Hungarian notation".
I don't consider it a problem, but it is to those who want to see big numbers.
As for hardware-related problems and race conditions, testing at a higher level of abstraction seems like it'd help more than it'd hurt - in the former case ensuring graceful handling as part of the tests, and in the latter case making the race conditions more likely to hit (and hopefully fix).
It’s a bit of the old classic “reading from the file never returns an ‘access denied’ error, until it does”. There’s ways and means, but let’s not pretend this is simple.
In todays development world, the unittest is primarily a developer tool to help speed up development. One should not be afraid to delete unittests if they are doing refactorings. But the long term value is always in the integration tests.
The important thing is not the size of the code affected by the test, the important thing is that a test should verify a single requirement isolated from other requirements.
I believe the original distinction between unit test and integration test was that integration test was for when parts developed in isolation by seperate teams was finally integrated. But this tended to be a disaster, which is why continous integration was adopted.
Obviously, when reading the rest of the papers, they are clear on the fact, that it is the specifications that the programmer have developed for, that should be tested. That was at the time, synonyms with individual sub routines, and as such it was both the smallest unit and a logical testable unit at the same time. Since then, we've come a long way with programming smaller units with better composability.
I'm not sure I agree that the original meaning of a unit test was lost. Perhaps, only one part of the definition was carried over to modern development practices. In any case, I always stand by the fact that long term value is in tests of the "API". Everything else is implementation details.
When I was at BigCompanyA, we had 95% coverage requirements as a management fiat. The company generally was very management-decree-driven. People would unit test individual methods and helper functions and each “unit” of code. If you wanted to change an API, you had to dig through a sea of broken tests because each little bit of code was individually tested. We literally had unit tests that validated one line methods for string concatenation of a prefix (in Java). Management said add tests with each commit so everyone added tests without thinking about what is valuable.
At BigCompanyB, our testing was engineering driven, not some management metric being tracked. The goal was to test “public interfaces” and ensure that these tests captured all the helper methods along the way. This helped catch dead or extraneous code if you couldn’t write a test to exercise a particular path. Changing an API didn’t require changing a bunch of silly tests. We still had equal >95% coverage, and it was very useful.
Basically you need to actually think about what makes sense to test and write logical tests.
It depends.
I've written a number of apps where I'm fearless with making changes because I trust the tests I wrote. One of the apps has been running for 7 years and it's had a lot of big updates, refactors, etc.. I'm breaking every rule there is on jinxing things but there hasn't been a single bug introduced due to those updates and there's ~85% coverage. It's only a ~3k line Flask app though, but it does get deployed straight to production with no staging environment and gets hundreds of requests per day. It writes to a DB, Redis, interacts with multiple 3rd party APIs and sends out webhooks so it does quite a few "external" things.
I've never been a fan of TDD and personally I think tests that really hit your DB, Redis, etc. help a lot more than mocked out unit tests or a billion unit tests and nothing else. I tend to write more tests that really test things together. Not full blown Selenium style tests, but I do really write to the DB and other data stores in tests and often use a framework's built-in test client for making HTTP calls. Everything can still be really fast too (~100 tests in 2 seconds is my usual rough guide for having an assortment of "real" tests with Flask).
I've come to the conclusion that the idea that "unit tests" should test functions/objects in isolation with completely mocked dependencies is based more on the slow speed of those dependencies in the past than what actually makes for good tests. Now that we have faster computers and storage devices, and easy/fast store creation, we should move past this.
Obviously, this is very dependant (no pun intended) on the dependency in question, but as a minimum, anything with SQL should have a test that hits a real SQL DB (PG in docker for example) at some point.
External dependencies (such as PostgreSQL, Redis, what-have-you) are a pain though. I feel pretty strongly that just a single command ("go test", "cargo test", "npm test", etc.) should run all the tests on all platforms, reliably, with a minimal of fuss and messing about. Things like docker-compose or whatnot quality as "a fuss and messing about".
That said, in the current job, we have a dedicated Postgres container that comes up with with a just file command. The setup, test, and teardown of schema all happen within the standard test system of the tested platform (pytest or go test specifically).
You could quickly make the changes you want not worry about what else breaks, then run the tests. The tests that you expect to fail would fail, and you fix them. Then you find other tests that you didn't expect also failed; you'd review the impact of your changes on those parts of the system and make appropriate changes until the tests were happy.
Because the testing was so thorough (and quality), you had high confidence that nothing unexpected was broken.
It was "move fast & break [tests], then fix them and deploy" :)
https://engineering.atspotify.com/2018/01/testing-of-microse...
> Has anyone ever actually found this to be true?
Yes, but you have to have the right kind of test coverage, and that's the tricky part.
Yesterday, I refactored a bunch of functions that changed a bunch of unit tests. However, since we also have integration/system tests on that code, I'm confident that I haven't broken the code as a whole. Without those system tests, I would not have confidence that the change would be successful, and probably would not have refactored.
In another codebase that hadn't been touched for a year, as part of a feature change I refactored an SQL statement to what I thought was a more optimal design and immediately broke a bunch of tests. Based on that, I was able to understand the original intent of the SQL, and updated it in line with the feature change. I added test scenarios for the new feature, but left the existing scenarios as is.
Without those tests, would have broken the system in a subtle way.
In most other cases, we had tests that only covered the happy path and did little more than slow down builds and make refactoring more difficult. In other words, they made things worse. E2E tests, in particular, are the Afghanistan of web development. I’ve seen more time wasted getting useless tests green than I’ve seen wasted on any other programming exercise.
Of course if you need a higher standard of guarantee you can try model checking or other formal verification techniques.
I am always in favor of tests around the boundaries between system components (regardless of what you call them; integration tests, API tests, etc.) The more robust the better.
If I'm working on a system with robust (ideally generative) integration tests covering the interface of a component, I feel I have near-complete freedom to rewrite any aspect of the component I want with high confidence.
Stable interfaces that the user cares about (e.g. CLI, public API, RPCs) make for the highest-value tests.
How have you seen this done well? In my experience, this usually ends up being some hello-world type simplicity that doesn't really represent the real world use corner cases.
For public examples, I'd point you to (e.g.) Jepsen.
I'm not going to deny it requires a pretty high level of time/money to implement. But done well it's super powerful.
Funnily enough, "unit tests" is what this was called originally. The boundary is what "unit" referred to. In my experience, these days most people just call them "tests". That they happen at the boundary is implied as by this point it is generally agreed upon that testing anything else is a waste of time at best and sometimes even detrimental.
Which is why nobody can settle on what "unit", "integration", etc. mean in the modern age. There is nothing related to testing in need of additional qualification to communicate to other developers what isn't already communicated with "tests" alone.
If you have to rewrite tests that means you've changed the user experience in ways that are not backwards compatible.
Which is sometimes valid, but not exactly what is being talked about here. The discussion here is more about changing the code in ways that makes the code better, but still delivers the same user experience – possibly with new features added, but not where anything is taken away.
That is only true for integration tests. You can rewrite a set of local functions without changing user behavior, and then you need to change tests, such refactors becomes a pain when you have too many unit tests but are really easy when you have many integration tests.
If changes to "local functions" calls for tests to be rewritten, that means you've exposed "local functionality", even if by accident, to the outside user. Which means it is not actually local functionality, but something you have exported and are committed to maintaining. Rewriting the tests is not the correct course of action. You need to fix the code that you just broke as the functional contract was violated. With any luck that hard lesson will teach you to be more careful next time.
But you are right that deleting the tests isn't an option as it will break the contract that was entered into with the users of the code. Of course, you can't modify the tests for the same reason, so...
There may be a black art to recognizing what you overlooked, but otherwise it is pretty straightforward. Tests are your documentation. The angles needed to be covered are exactly what the user needs to know to use your software – how to use it and, when using it, what happens both in the expected case and when failure occurs (especially what happens when failure occurs!).
Calcification is of little concern as the inverse is breaking changes, and the user does not want to deal with your breaking changes. You can put your mind at ease knowing that once you commit to a documented feature, it should remain there until the end of time.
Please explain this to Product Managers. :)
But only at companies that don’t give a shit about test coverage %. The percent kills the purpose of the tests, which is certainty.
We have a 300kloc monster, and I find that going up from 0 to 60% test coverage has given me appreciably more confidence that the system still works after any change.
Sure, the test code is almost twice that size, and breaks almost as often as the code, but my confidence in the system itself is definitely higher.
To prevent people from writing tests that need too many mocks we now have explicit dependency injection. So much easier to reason about stuff, and prevent’s people from not testing the important bits.
Obviously tests are going to break when you change the code's APIs and functionality. That's expected, and does nothing to boost your confidence in the part of the code you're working in. The point of tests is to improve your confidence that you're not doing things that have an unexpected impact somewhere else (hence the Bob Martin quote in the article about tests being useful even if you have good design).
Tests aren't an alternative to thinking. You still have to consider what changes you're making, and what tests should break as a result. That's just part of your code change. When tests that shouldn't have broken start failing that's when they show their value.
Unit tests which dependency-inject mocks of other parts of your codebase are 99% worthless.
Source: Spent many years writing the latter and they caught close to 0 bugs. Moved to Elixir ecosystem where integration tests are the norm, and they catch bugs on a regular basis.
I never understood the fascination with Unit Tests. For Testing to be useful the code being tested should have a certain degree of Complexity (algorithmic/behavioural/state-transition/etc.). But what i see from most unit tests is mere "busy work" as if mock-testing a trivial class/api/etc. would somehow make your code better. A similar criticism is also applicable to TDD based programming.
Me neither. The legacy forms of testing that "unit" was trying to differentiate from have died out. All tests are unit tests today, which is why most developers just say "tests". "Unit" is already implied and adds no additional information not already understood by "test" alone.
Same goes for integration tests. Integration tests, as it was historically defined, died out. It seems anyone still trying to use the term today is using it to also just mean "testing", or what was historically known as unit testing. Like unit, a pointless qualification.
> I work in a place with 99% test coverage requirements
Sounds like your problem is bad tests. In fact, I guarantee they’re bad tests because that’s exactly what coverage requirements lead to
Has anyone ever actually found this to be true?
I work in a place with 99% test coverage requirements and it's honestly still a super brittle system that everyone is afraid to make big changes to
In my experience, automated tests don't promote quality engineering, they just calcify whatever quality exists
And before you get into it, TDD is not a panacea for this problem