I’m the first to say that if you have no tests and no idea how to get start, go with end-to-end tests first and take it from there. That doesn’t mean that guidelines that help you to have less brittle and more idiomatic unit tests are bad per se IMHO.
Bit offtop, not debunking the content of this post though!
There were lots of brittle mocks. I think we even did mock out the Ruby equivalent of the HTTP client.
Every change would inevitably break tests (not functionality!) and developers were spending more than half their time wrestling with the test suite.
All of us could have used this article.
Please always strive to improve your practice. Commit things that aren’t perfect, but try to internalize best practices over time. Gradually raise your own code quality bar as you get better.
Learning from posts like the OP is super important if you don’t have a good senior Eng on your team. A group of smart juniors can easily code themselves into a hole of left to their own devices. I know this, because I did this.
Not sure if I’d have been smart enough to take my advice though. ;)
I find that this invariably happens because the team tries to test implementation at a very low level rather than behavior at a high level.
I blame an overemphasis on unit tests over integration testing (e.g. J B Rainsberger's inane rant) and tutorials that imply that, for example, if you build a class you have to have some tests for that class.
The other thing Juniors don't get is prototyping. I regularly build something quickly to get the problem set into my head. If it's good enough, I'll leave it. If it's got issues, I'll throw it away and start fresh now that I know what I'm solving. Juniors don't work fast enough or effective enough to do things like that without putting timelines at risk. Plus, they frequently want to do some cargo cult methodology of the week like TDD.
Are you saying you have to tell the developers not to test business logic, or that you have to fix the tests they write?
Test code should fail when an important underlying mechanism regresses. If it always fails on the next phase of the project because some hapless idiot needed to cover every line, the signal to noise ratio gets weaker and the purpose of the test code is lost.
Same goes for prototyping. I don't see much of a correlation between willingness to prototype and seniority, either.
Railing against code coverage is at this point just as dogma as insisting on it.
What I'm railing against is the idea that seniority is a prime indicator to the effective strategy parent comment insists on, which simply doesn't mirror my experiences. What I see is juniors picking up the habits of their superiors. They're learning this dogmatism from somewhere.
Willingness is one thing, but getting yelled at for learning and doing your best is another. I never treat my Juniors like that, and encourage a 75/25 working/improving split. Still, several frequently get anxiety about how long it's taking them to clear tickets.
I've found that E2E tests make for a pretty bad case of hill-climbing where tests are concerned. To the point where I think that's the pathology of the testing icecream cone.
It's easier conceptually go to 'up' the testing tree from unit tests to E2E tests, than it is to come down. People do what they are most familiar with, and if the E2E tests are first, then they are the most familiar. They are also glitchy as hell, and I see disdain for that get applied to all tests, not just the human tragedy that Selenium turned out to be. Also the ways in which unit test habits are considered 'bad' in functional tests are either less consequential or easier to walk away from. I'm not entirely sure which it is, or if it's both, but it's definitely less painful to relearn.
We also overvalue the E2E test versus human testing. I can't count how many times someone has come to tell me a login button is broken, 20 minutes before the E2E tests fail and tell me the same thing. Computers arbitrating code regressions are half the point of CI, so if your tests aren't arbitrating, they aren't doing their job, and should be fired.
Low level tests are the reverse - easy to get started, hard to maintain.
If youve got a really well oiled framework e2e tests are spectacular but getting to that point can vary between easy and "harder than the actual project itself" and if you half ass it (which most people do) it ends up a useless flaky mess.
I've only seen this with projects in maintenance mode. Any major functionality changes tend to land teeth-first in the E2E tests, and after a few cycles of that you start dragging your feet a bit on major changes. E2E tests tend to lock in assumptions about the structure of your application, not just the minutia (which is not without its own set of problems).
It is more difficult to get people to write maintainable E2E tests than it is getting them to write maintainable application code, so I've retreated and retrenched.
In most cases we could literally rewrite the whole app in a different language and barely change the tests.
E2E tests do require a solid infrastructure though. If you have tooling problems you cant fix (e.g. you hit selenium's dark corners way too frequently because your web app is even a bit quirky) then maintainence cost can quite quickly exceed any benefit you derive.
Ive also hit this problem of "Id have to create my own equivalent of selenium for interacting with this app's interface" a few times and yeah, the required engineering effort to do it well explodes to the point where its just not worth it.
It all depends on the language and the project, but my experience is usually the opposite. It's often much easier to write an end to end test. To be able to write a unit test you usually have to architect the code to make it easy to mock, etc - and that takes significant skill for most real world projects.
E2E tests have a lot of downsides, but being challenging to write is probably not one of them.
My experience agrees with this. Adding unit tests to untestable code (think: class has 10 dependencies that it news up being the first branching of a depdendecy graph to be untangled) requires refactors that just don’t get done with the busyness of other priorities.
That's been my experience. My point is that the reason the project has E2E tests and not unit tests is because it was easier to add E2E tests.
(To be fair, my projects are low on JavaScript which somewhat changes the trade-off calculations, but I haven’t seen any assertions about project types above.)
Tests are code too. Like the code being tested, they can be shitty at times, but they are still code and should be treated with the same level of care.
Which is not to say "your tests should be as DRY as a desert". That's frequently a poor fit for test code's lifecycle. But it does mean you shouldn't do stuff that wouldn't pass code review elsewhere, just because "it's just a test". Someone later has to read, understand, and maintain that code - treat it as such.
Basically all the test did was prove that the mocking library worked well. Almost no actual project code or business logic was tested.
Company was not happy to hear.
> "Because often the end goal is about writing clearer code even with shitty tests."
If you're worried about people being afraid to write tests, this really scares people away. Nothing makes me want to write a test for my changes less than a 2000 line test file with 50 mocks.
Maybe for some strict definition of "testability", but a lot of people in the Python world use patching to circumvent anything that might reasonably be called "testability" in the pursuit of "confidence" and some puritanical desire to avoid changing the code base to make it testable. This makes everything a complete mess for anyone who comes along later to extend the code or increase test coverage, as the tests now depend on the private internals of the code and the patching logic is magical and unintuitive.
If your "testability" remark precludes patching, then I'm more bought in, but I do worry that some will read this and use it to argue for patching.
That said, experience tells me that mocking delivers a lot less confidence than using the real deal, although there are some cases (e.g., network errors) that effectively require mocks _somewhere_.
At at an earlier company, we had very little patching or mocking. We spun up all the internal services when running the tests. Internal calls were never mocked. External calls were sometimes mocked (if test accounts / API keys were not convenient.) We preferred testing the "real thing." Tests ran slower but in general I had much more confidence tests actually did something useful.
Most of the time for third-party services, I'll just fake the service client for unit tests. For example, if I'm writing a BookStoreApp that has a BookStore dependency (where BookStore is an interface), the concrete implementation for which is a PostgresBookStore, then I'll write my PostgresBookStore client library with unit tests that include a real postgres connection but then my BookStoreApp tests will use a FakeBookStore (which is probably a thin wrapper around a dict, but which behaves like a BookStore should modulo persistence). Finally, I'll have considerably fewer API tests which actually do stand up the complete backend service (including a PostgresBookStore). This way I get a high degree of confidence and also speed of test execution--it's also nice not to have to stand up the entire world just to run one test (in particular, it's nice not to have to debug unrelated connection/auth issues).
Useful if you have a reference implementation that is obviously correct but slow, and a fast one where it may not be correct.
Heh. I've long felt a need to write a blog post touching this very idea, but have been too lazy to.
I think a lot of people (including younger me) would dive right in to things like unit tests without too much thought. Now I focus on:
- What should we test? (Think about the feature you are adding and why - what aspects need to work?)
- Why? should we test it?
- Should we test it? (really the same as above). Common to see junior folks end up testing 3rd party libraries.
- How should we test it? Mocks? Unit tests? Integration tests? Something else?
- Is it even testable in an automated fashion? If not, ponder over what makes it hard to test. Does that need to be rectified?
- How can we be sure we're testing what we think we are? Extremely common to see people write tests that appear to pass, but are testing the wrong thing (buggy test). It's very rare that I see a colleague intentionally introduce a bug in the feature to see if the expected test will fail.
- What is the cost of this feature having a bug? If it's very costly, you may want to have multiple types of tests, and possibly even a human testing it each release. If the cost is almost zero, and you won't need to fix it quickly if a customer reports it, and it's hard to write an automated test for it - just skip it. If this mentality bothers you, know that this is recommended by the ISO standard for SW in cars - even if a bug can end up in loss of life, if you can show it's extremely unlikely, you are conformant if you don't test it.
You should first answer these questions, and they will then guide you into whether to write unit tests, or E2E tests, or something in between. I generally find that people who dive into testing without answering the above write crappy tests (buggy, pointless, etc). Answering these can lead to better mocks (or a decision not to rely on mocks).
In summary, step back and forget everything you've learned about testing before answering these questions. Pretend you don't know the difference between unit tests, integration tests, etc. You've written (or plan to write) a feature.[1] What would you like to test in an automated fashion, and how can you do it? Feel free to use any tool that works for your project.
[1] You're not writing code. You're writing a feature.
Is there a place where tests/unit-test-methodologies are discussed, or codified?
How learn?
All rules should be in service of the above. I don't know that there is a "most people agree with this" set of rules, but if you read any source on how to test, think about how each rule affects the two goals. Senior developers tend to have developed an instinct for "things like this have failed one of the two rules" but it is hard to codify that instinct into rules.
The PRACTICAL rule is always have unit tests. Start with writing tests for bugs you actually encountered. Use whatever testing framework is popular in your current language, particularly if it ships with the language. Always try to test things in the most straightforward way possible. Add automation to run all tests regularly.
BUT keep in mind, test code is code. You're now writing code twice, once for the code, once for the test. Either or both can be wrong. In the future, both may have to change. Tests improve reliability, but are not free.
---
What is the PRACTICAL learnings of CREATING tests?
Meaning, how does one succeed at learning to employ the same sentiment above usch that one does well from ground zero?
(integrate tests into your design, but what does one need to know in order to do such?) [ELINSW: Explain like im a newbie software engineer]
----
I understand the philos... I am talking about IS THERE A LEARNING CHANNEL TO BECOME AN EXPERT AT DEVELOPING UNIT TESTS, or is it something that is lit. learned on job?
Writing unit tests for software that is testable is easy. The hard part is to write testable software. But then there’s people who will argue that making software testable often leads to “test damage”.
As always, trade-offs & personal preference.
If you listen to Kent Beck, Ron Jeffries, Martin Fowler and so on, you could be pardoned for the impression that unit testing is an idea invented by Kent Beck in 1997 with JUnit, that has taken the world by storm since.
That's complete and utter BS. Perl 5.0, released in 1994, came with Test.pm, which made it easy to write tests. Better yet, when you installed from CPAN it would BY DEFAULT run the unit test suite to be sure that your environment worked as expected. (This is a critical idea that I wish had been adopted by other repositories...)
This was Larry Wall encouraging people to follow the practices that he did. Perl 1.0, released in 1987, came with a unit test suite that ran by default before you installed it. In fact he knew the technique because he'd adopted it with his best known previous program, rn. And he knew to do that because the practice was already accepted.
Of course the idea was not original to him. See http://secretsofconsulting.blogspot.com/2008/12/how-we-used-... for the reminiscences of someone who was doing unit testing back in the punchcard era. He should know. He was Manager of Operating Systems Development for Project Mercury (the first NASA program to put humans in space). And THEY were doing unit testing back in the late 1950s, early 1960s.
Even he was not first. https://www.amazon.com/Digital-Computer-Programming-Daniel-M..., written in 1957, already described unit testing.
But, despite how widely known all of this was, Kent Beck reinvented it. And a dogma quickly emerged about this "new idea". Which included some good ideas and some terrible ones.
Extensive use of mocking I would consider a terrible idea. That it was terrible was obvious to me the first time I saw it. Replacing an external dependency with a stubbed out implementation is often an excellent idea. But mocking is a horrible way to write that implementation. I'd far, far rather deal with tests that create a SQLite database and test against than than something that does extensive mocking of what they think a database should do.
That said, I would never trust a learning channel like the one you describe. I've learned the hard way that people who consider themselves experts on unit testing tend to believe that more tests are always better, and have no idea of the impact of excessive unit tests on a codebase. So a channel like the one you describe is going to attract people whose opinions I'm suspicious of.
Instead aim to be expert at developing software. Unit testing will prove to be an important component. But nothing about unit testing makes sense when separated from the development that you are attempting to accomplish.
For a very enlightening example, consider http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-s... to compare someone who is an expert on unit testing (Ron Jeffries) with someone who is an expert on software development (Peter Norvig).
I think that's why people don't read blog posts like this - such wording is rude and patronizing.
If changing one line of code regularly requires me to update mocks in 100 test cases, those test cases may have negative value.
Kent Beck
Do whatever that will increase confidence in code and testability. If I want to make it easier for me to develop, I'm gonna do anything that will enable me to get shit done. Test takes 0.1 second instead of 0.001? Whatever. I can improve it if needed, when I get more knowledge or I can discard it completely.
When I stopped caring about 'should I use mock, or inject closure, or interface...' I started writing more tests and producing better software. Because often the end goal is about writing clearer code even with shitty tests.