If you're building simple things, tests can be overkill.
If you're building complex things, tests pay for themselves rather quickly.
Most software in the world implements business rules defined by people who are way less smart than your average engineer. And yet our software is more complicated than ever.
Not everything can be simple.
Mathematical algorithms are an example where there's necessary complexity and unit tests backstop defects when you refactor code.
They move logic to a different library, break it into 10 libraries, introduce layers of abstraction, try increasingly complicated View/Model architectures… And ultimately the code is no easier to understand or manage than when they started.
It is nearly impossible to find a complex domain that doesn't naturally break into smaller subdomains.
That's because "domain" is human concept. The ones that are too complex and don't have natural boundaries within them just don't stick.
There are two main reasons software becomes complicated:
1. It is over-engineered at the beginning of it's lifecycle.
2. It is under-engineered in the middle of it's lifecycle.
Interestingly, unit tests contribute to both of these.
For an engineer it's always hard to admit that your system is complex because you failed at engineering. It's much easier to claim that you're working in a complex domain: you get to shift blame and boost your ego at the same time.
I'll give you a concrete example: AML (anti money laundering). If you build a fintech with zero dependencies, you will need to deal with this. It's not manufactured complexity—it's an authentically complex problem which stems from 1) the actual law and 2) the risk profile the business is willing to assume. Now, you can come up with clever ways to make the rules of your AML policy composable and easy to change, but the reality is that you will immediately hit an ocean of edge cases that require ever more nuanced mitigations. You can make keep subdividing and decomposing the problem, but you will end up with an architecture diagram which is (wait for it) complex. So yeah, you shouldn't over-engineer before you launch, but eventually, your AML system will take on a certain amount of unavoidable complexity.
Now try to do all of the above WITHOUT unit tests, and see how far you get.
> introduce layers of abstraction, try increasingly complicated View/Model architectures
> you can come up with clever ways to make the rules of your AML policy composable and easy to change
These are the exact design decisions that are typically accompanied by unit tests.
And yes, if we're being pedantic, an AML system would need unit and integration tests—but the question is whether or not tests as a whole are useful. So unless you're arguing that an AML system can be made so simple that no tests are required and it "fits in your head," let's agree that testing has its place.
Not in my experience. Overengineering and unit-testing always come hand in hand, likely because both are the "enterprise" way of doing things.
> the question is whether or not tests as a whole are useful
Huh? Nobody ever asked this question.
Questions I'm always debating are:
1. Whether unit tests are useful (no in 99% of cases)
2. Whether more tests = better (no)
3. Whether you should invest in automated testing before manual becomes painful (no)
> unless you're arguing that an AML system can be made so simple that no tests are required and it "fits in your head,"
Some tests will be required. But yes, it should fit in your head, otherwise you cannot reason about it. Building AML system that you can't reason about is a road to failure.
It doesn't mean that you need to load implementation of every single rule in your head. But you need to fit, for example, the idea of rule engine and the constraints of rules working together. If you build rules that can do anything with no constraints (depend on each other, override each other), then you're screwed.
That said, mathematical algorithms:
1. Rarely need refactoring. If you often refactor your math, you probably mixed it with something else (unless your business is math).
2. Rarely have state. You're always dealing with values, not entities.
3. Rarely have dependencies. There's nothing to mock, so the boundary between unit tests and integration tests is very blurry.
4. Usually have a formal oracle of correctness.
So yeah, math is a place where unit tests might be ok (if you truly have a lot of complex math).
When teams don't have good test suites (meaning mostly automated, if not entirely, and reasonably comprehensive), then you can watch as over time you get either increasing error rates or a massive slow down in team performance. Even for small changes, having no tests or primarily manual end-to-end/integration style tests means that you have little chance, unless it's a particularly good code base, of determining the actual effects statically (by analysis) or dynamically (because your tests will take too long and are likely mostly ad hoc so you will miss edge cases).
Teams that don't care (or are pressured into not caring) will deliver crappier and crappier software over time. Teams that do care will eventually stop delivering software at all, or at least no more meaningful updates.
All for lack of confidence.
> you can watch as over time you get either increasing error rates or a massive slow down in team performance
I hear this story repeated over and over and yet I consistently witness the opposite of what you're describing: teams start investing in unit testing and they noticeably slow down and ship more bugs at the same time.
What can give you confidence is simplicity. Being able to fit the entire system in your head. Being able to say what are the side effects of your changes. If you feel like you're losing those, it's time to take a step back and rethink your architecture. But of course, nobody has the balls to take a step back and think, to admit that you failed at something. It's so much easier to just slap a bunch of unit tests on your big ball of mud and hope for the best.
I was writing about tests generally, in case you were wondering.
"Tests generally" and unit tests are pretty much the opposite camps in testing world.
I've written a Photoshop file format parser twice, and trust me—between byte alignment changing constantly throughout the spec, at least three different string encodings, sub-formats and errors in the spec—it does not "fit in your head." Certainly not enough to confidently add features without worrying that you've broken the blend mode parser.
And of course, if you can and do fit a complex system in your head when no one else on the team can, congratulations: you've made yourself into a bottleneck and flight risk.
Or it has good boundaries and I never have the need to load the whole system in my head.
You're not loading the whole OS kernel in your head every time you call fopen(), do you?
> Photoshop file format parser
Would you say that PSD format is an example of good design? Is complexity inherent to the problem domain, or is it accidental?
You are forced to replicate complexity created by other engineers. Woe is you.
My bet is that Adobe also has a lot of unit tests, and contrary to what unit-testing advocates claim, it resulted in a terrible design you are dealing with.
I strongly disagree on the ratio. If you're building Mastodon client, most of your effort should be spent on features and UX, not on OAuth and the like. If half of your time is spent on those, either you're at the beginning of the project, or you're at the beginning of your career. Or maybe you just care about engineering itself more than the end product.
Here's a talk where Jack Diederich mentions it: https://youtu.be/o9pEzgHorH0?t=870
It is indeed complex: 200+ classes. Plenty of unit tests and mocks.
And then there's his implementation:
- 0% unit test coverage.
- Lightweight at less than 200 lines, including blanks and docstrings.
QED.
I've done three or four major refactors on a FOSS project of mine. The refactors went without any hitch because of an almost excessive test suite.
What I didn't expect is that the tests gave me so much confidence to improve, extend and refactor the code. Before it felt so fragile, like the whole thing could crumble if I don't quadruple check my work. With the tests in place, I changed the entire underlying data type from a struct to a native simd vector in an hour. Made the changes, fixed a couple of bugs, merged the branch, boom.
I don't strive for anything close to 100% coverage, but the handful of tests that cover my critical code have saved me so much time.