Unit testing is by far the most valuable form of testing during development, in that it catches 90% of bugs before the code touches hardware. Also, after development, unit tests serve as regression tests that catch unintended changes made by others.
Mocking, whether or not through dependency injection, is absolutely necessary to enable testing when targeting bare-metal platforms, and very helpful to speed up testing when targeting an operating system. Dependency injection in particular usually constrains code to be written in a way that, although perhaps seemingly more complex at the surface, is a lot more maintainable long term.
Integration testing is also a very good idea. The thing with integration tests, though, is that they get bigger and more complicated and slower over time. Relying solely on integration testing will likely become a development bottleneck if your system grows in complexity. If you can run an integration test suite deterministically and efficiently on a development system, then the software under test probably isn't that complicated, relatively speaking. Also of note is the tendency for mocks and dependency injection to help out when integration testing as well; being able to completely abstract away hardware allows for faster than real-time deterministic simulation at a system level.
If you're doing well without unit tests and folk generally seem to be happy, then good on you. Unit tests are a pain to write. I'd suggest that you might be severely limiting your full potential as a software developer, though.
Maybe it's you who hasn't taken the red pill.
Safety critical software would be even more safe if you used formal methods. Formal methods verify logic to a degree of 100% while integration tests dig out unintended side effects or leakage.
Let me put it another way. In Computer science there's a field called formal methods. With formal methods it's possible to do things like dependent typing. These techniques and technologies can verify your program to 100% correctness. This is not unit testing to correlate correctness to a certain degree but actual 100% verification.
Such a technique has pitfalls in the sense that it is brutally hard. I am saying that there are programming methodologies that can be used to go above standard typing and below dependent typing where the program is not verified to 100% correctness but the correlation of correctness is so high that you can forego unit tests.
Some languages like Haskell and Rust already sort of enforce these programming methodologies / technologies on you such that most unit tests become redundant.
You need a lot less unit tests for a haskell program than you do for javascript.
Therefore a program that relies to heavily on unit tests is a sign that the programming methodology is flawed. Something is wrong.
- It's a must-have for business logic, because errors can lead to actual dollar costs (and looked at by the SVPs). Any maintenance change should require a unit test for this type of logic. This way our code confidence in deployment of a new feature or change goes up.
- Catching regressions. We've had errors we've missed in the past, and as our team changes and team members come and go, the want to make sure we are not making the same mistakes. Thus, automated testing on regression tests helps our confidence in code stability goes up.
- System failures. This is more obvious as an integration, but it's good to have these tests in place because once again they can be traced to actual dollar amounts. ALso helps confidence level of deployments by junior/new employees.
- Maintenance of legacy code over YEARS. I think most people skip unit tests because they are writing new code, which gets tested over and over during development, but adding a new feature to massive legacy code is still an aspect of our jobs. I don't remember how my code worked last month let alone 3 years ago, so unit tests help not only in confidence levels, but in speed of development because I wrap mocks around any services that I don't want to wait for or should not be triggering for this rapid-iteration development.
Each circumstance is different, but I think unit testing goes beyond just knowing the domain well.
It's easy to write something small and correct from scratch (like a greenfield microservice). It's far harder to maintain that over several years and several changes of hands.
If your coding methodology was good, the type checker is enough to verify these contracts.
A language without a type checker will have more need of unit tests.
Unit tests ensure that everything you thought about still works. Types ensure that ALL types are compatible.
What types do well is the everything case, you cannot forget something with types. However they only cover compatibility not correctness. You can have a type correct program that is wrong.
There are formal methods can can get to correctness. If you have the right constraints. The only problem is you can have the wrong constraints. Within that limit though it shows the program works correctly in all cases.
Unit tests prove only the cases you thought about work. While this is far below the infinite number of cases the above prove, they are still very useful because it is generally very easy to reason about the one case you do choose. If you choose the cases well, the majority of possible cases are "close enough". Thus a few well chosen unit tests can give you assurance that your code works correctly in the important cases.
For things that live outside of this domain the term integration test becomes relevant. If you are testing for unknown side effects it is no longer a unit test but an integration test.
This is not my point.
Unfortunately I can't go into detail about my point because it's just too long to talk about. I glossed over it and called it "Programming methodology."
In the context of correctness, with the right programming methodology, programming language and type checker you can use this and your intuition alone to forego the need for unit tests. This does not apply to integration tests.
Anyway without going in to deep the right programming methodology involves reducing complexity. It's similar to how if a programming doesn't have nulls you can't have runtime errors caused my nulls. If you use the right programming language or right methodology you can reduce much of the complexity of the program to the point where unit tests are redundant to your intuition.
Another good example of a language that pulls this off is Rust or Haskell. You don't have to go as far as COQ but the previous two languages force you into a methodology that ensures correctness to a degree that many unit tests become redundant.
Or to put it a different way, unit tests guard against an incorrect specification. Code can be proven correct and wrong at the same time. Unit tests make it much easier to reason about if what is specified to happen is what should happen.
As an aside, I'm not against blurring the line between unit tests and integration tests.
Dependent types and/or automated proof checkers should bridge the gap of intuitive reasoning. Basically the dual relationship where code verifies the type and dependent types verify the code should be strong enough to forego unit tests and guard against "incorrect specification."
>As an aside, I'm not against blurring the line between unit tests and integration tests.
Sure that's fine. But then the type of test and the context of what I am referring to is strictly tests that only cover pure mathematical logic.
Things outside of that bound cannot be covered by proof nor by programming methodology and are also not the type of test I am referring to.
For me this is the best side of unittests: their validation of “correctness” is just one aspect, but their job as a guarantee of decoupling and a living documentation is at least as important. Even a test that is only written and compiled but never run provides great value for architecture and documentation.
If you want to write unit tests for documentation that's a different argument. I'm not against that.
Unit tests famously don’t prove correctness (the absence of bugs) only the presence of bugs, so I guess it’s wrong to compare them to actual verifying actual correctness.
Unit tests are like statistical samples of all possible test cases. So unit tests while they don't verify correctness overall they can correlate with correctness. That is why you only need a couple unit tests to correlate with. a degree of correctness with a domain that's almost infinite. The phenomenon that enables science also enables unit tests to work.
That being said, there are programming methodologies of managing complexity that negate the need for unit tests. This is my main point.
I presume you're talking about proofs, or maybe something like Idris?
It's really too long to get into but the programming methodology I'm talking about has to do with reducing complexity of your program to the point where your intuition along with the type checker should render unit tests mostly redundant.
For example a language with no nulls renders all unit tests that test for null runtime errors unnecessary. Add exhaustive pattern matching into the mix and you'll render All tests that cover runtime errors unnecessary because your program simply can't have a single runtime error (see elm.)
Both of the examples above involve either the language itself or a programming methodology or both. Rust and Haskell are two languages that force a lot of "methodology" onto your programming where a significant amount of unit tests that you would otherwise need in say python or C++ are rendered unnecessary.
Another way to look at it is using a highly restrictive form of programming. The more restrictive the less opportunity for errors to the point where your intuition and type checker alone should be able to allow you to forego unit tests.
There's more too it like segregating all IO from core logic but I've already written too much. Suffice to say what I'm talking about falls within a subset of the domain of a semi-popular style of programming. I don't bring up the name because that will only detract me from the main point.
Everything including nulls, index-out-of-bounds is unnecessary and can be elegantly handled by the type system.
But complex business logic like building fire codes, tax law, network standards, traffic rules etc aren't that simple. You are quickly into Agda/Idris non-mainstream territory if you want to represent some tax law in types. It's basically "if you have two or more dependents and make under 10k then the tax is 4.5% of the income over 5k minus...". It's of course possible to represent anything using a complex enough set of types - but I haven't seen a good example of a complex piece of domain logic that didn't need tests. I'm a big fan of e.g. "Domain modeling made functional" and similar, but usually the most complex calcultions are just that: calculations. You can encode things like "the input must be positive" or "the output is guaranteed to be positive" with types easily, but when you want to encode that "given the input 3 the output must be 16 on a thursday" your are beginning to probably see diminishing returns if doing it with types instead of e.g. property based testing.
Let me put it this way:
Given an exact set of tax laws, network standards or traffic rules I can implement this perfectly and easily with types, intuition and programming methodology alone. (Another hint: don't use mutable variables or even better forego variables and go point free; it eliminates a whole class of bugs that has to do with state.)
Implementing a specification is trivial even when the specification is complex. It's simply a mapping from english to programming logic. There is little complexity here.
I think what you're getting at is a sort of derivation of the specification from a macro goal.
For example, what is the best way to tax society so that society has maximal benefit and zero loop holes for bad actors? What is the best way to create a set of traffic rules and network standards so that throughput is maximized, fair among everyone, and (most importantly) 100% bug free?
This is hard and specs derived from a macro goal can be bad or wrong. This also usually happens before programming begins.
Bad specs are not something that unit tests are really used for. Unit tests are used for logic errors. To catch where a programmer implemented a spec incorrectly.
While I can see some logical value for using unit tests to test for macro level goals this stuff in practice happens at the IO level and is the domain of integration tests. You are not testing a "unit" when measuring a macro level goal, the philosophy of an integration test applies in this case because a macro level goal is basically operating at the integration level rather than the unit level.
Type checkers do not verify behavior. Unit tests verify behavior
Types do verify behavior - e.g. this function always returns an Int, that function only returns a UserId, this other function doesn't do any I/O.
What you really want is to be able to state properties, e.g. "for all x/y, mul(x, 0) = 0, mul(x, 1) = x, mul(x, 2) = x + x, mul(x, y) = mul(y, x)", and so on. And working with these kinds of statements (via e.g. quickcheck or whatever) might generate tests, but feels a lot more like working with types. You're making proofs, not examples.
Tests add real value for me. They are now usually the first "user" of my code. I often find design issues when writing tests. They also catch bugs (rarely but often enough to be useful) and make it easier to refactor with confidence. YMMV!
Most of the code I write doesn't need unit tests (automated or manual) because the way I programmed it reduces the complexity of the program to the point where my intuition and the type checker alone will cover correctness.
Basically another way to put is... the code doesn't have any errors in the first place so any form of unit testing isn't really needed.
Note that I am talking only about unit tests or tests that cover pure internal logic. Anything that touches IO or anything external becomes an integration test and That is not what I'm talking about.
I would try coding some solution in haskell with minimal IO. You should only need the type checker to verify correctness.
I don't know that I'd word it exactly like that, but one thing I've found over the years is any tests we have that rely on mocks basically just don't test anything at all.
If your whole test is "This gets called, then that, then that", you aren't actually writing a test. You're repeating what the code already says.
But your philosophy is correct and we are in agreement on that.
The thing I was referring to is the abstraction of dependency injection itself. It leads to code that is over complicated as the abstraction itself is not only pointless but adds unnecessary layers of complexity.
There are ways to target the philosophy of what dependency injection for unit testing is trying to accomplish without introducing the side effect of unnecessary complexity that comes with dependency injection. It's as simple as just separating IO functions from core logical functions. Don't bundle these two things together into an object or a single function, that's it. (Also don't use objects period, but that's another topic.)
I find it natural and convenient to think about "What dependencies on other parts of the system does this code have?" Expressing those dependencies explicitly feels like it reduces complexity, not adds it.
But, I'm not talking about pulling in a big fancy DI framework, just making dependencies explicit in your function/class parameters.
I will say that DI is sometimes used as a tool in overly-abstracted systems. An example that comes to mind is the ASP.NET MVC framework -- with DI and inheritance, literally any behavior can be overridden in fairly opaque ways. Trying to suss out the concrete behavior details is like swimming in quicksand. (Or it used to be that way, haven't touched ASP.NET in a while.)
As an aside -- I'm curious about your programming language of choice. I think DI is a lot more useful in some PLs than others. For example, I find JS code often uses imports to create complex graphs of implicit dependencies, and DI can help tame that complexity. But for other PLs like Python or Clojure, I basically don't use DI at all.
Now Java and C# are different because they are compiled. You need a DI tool to do dynamic dispatch if you want to mock.
You can't inject a nuclear reactor into a burrito in any language untyped or not because it will lead to an error.
The only difference is python/js the error will happen at runtime while in C$/java the error happens at compile time.
The main difference is the type. In typed languages you need describe am interface or a class of types if you want to do mocking or dynamic dispatch while in dynamic languages you don't need to explicitly define this as a type, the definition exists in how you use the object that is passed in.
Either way DI, however universal, in any language is bad practice.
Yeah, mocks are just a way to zero out the dependencies so you can focus on the unit you're actually trying to unit test. If all you're doing is zeroing out dependencies, you're not testing much. Every mock you bring in carries and assumption, and those assumptions need to be tested, too.
Cases like that really call for some small scale integration unit tests, where you actually bring in your dependencies and test them all working together.
EDIT: What I'm working on currently doesn't have a hardware component to it, so this might not be practical in the future
I find it valuable at times, depending on the context, but I agree that extensive mock testing like this can get quite useless real quick.
I think that's being too extreme and dogmatic. Unit testing and small-scale integration testing (implemented as unit tests) are a way of testing your assumptions and documenting and communicating your assumptions to others. Both are good engineering practices.
You just have to be pragmatic about how you use these practices. Teams definitely get too dogmatic and narrow minded about mock-heavy unit testing, and some people do manage to write code without writing any unit tests, however those are both extremes that are best avoided or of only limited applicability.
The benefit of unit tests is that certain things are more economical to test that way. You literally can test everything though integration tests of varying levels (or manual testing!), but it'll take more effort and run more slowly. If you want to thoroughly test a regex, just test the regex by throwing a dozen strings at it. Don't bootstrap the program and construct valid input requests to test each of those strings (but do do that to test that everything can work together, which includes a subset of those strings).
I use documentation for communication and documentation. Unit tests may do this but this is a side effect. The purpose of unit tests is to test your own logic and find mistakes you've made.
I'm saying there are ways to code where your intuition and type checker alone are enough to verify correctness without the need of a single unit test but with the same correctness as if you did verify your program with unit tests. You don't even know what this "way" is as I haven't even described it, so you can't even argue for or against it.
Dogmatic is a word for those who spout their point without understanding the reasoning behind the other point.
Or even write the test cases separately, then import them into a unit test runner that can check them all very fast as well as an integration test runner that can check them all very slowly.
- If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. There are other ways to make your code more modular without injecting new scope, logic and apis into an object.
Dependency Injection is one form of composition. One perspective to look at it, is that objects are like tiny scoped programs with variables and methods. To do a dependency injection is to put a program within a program.
Is this a good way to compose primitives? It's like parsing a python program inside a java program inside javascript. What are other ways to manage complexity and compose primitives without dependencies everywhere? What is a programming primitive? Is the most fundamental primitive of computing a function? Should we be composing bricks to form buildings or compose buildings to form bigger buildings?
Over the years I have come to reject some things that are seemingly fundamental to software engineering in the industry. One of them is unit testing. I don't believe that unit testing is harmful in itself, but excessive use of it is a sign of weakness in another area.
- if your code relies on unit testing for correctness than your coding methodology (or language) is error prone. A good programming methodology where the domain is well understood usually does not need a single unit test.
- If your code base utilizes dependency injection to make the code more "testable" with mocks, you are making the code 10x worse. There are other ways to make your code more modular without injecting new scope, logic and apis into an object.
This logic sort of applies integration tests which is basically testing the same thing as unit tests but touching functions that have IO. However it is a little different. If your program does Not need unit tests but needs integration tests it means things like:
- your type checker does not extend across systems
- the programming methodologies become more error prone as you move to other systems and you have no control over it.
- you lack understanding of the foreign system.
The thing with integration tests is that all of the above is largely inevitable and integration testing really becomes the only way to correlate (not verify) the entire system with correctness when you lack the integrated approach of a single programming language. Therefore because of this I am not against integration testing.
I recently green fielded a small micro-service at my company and I did not write a single unit test. It had one logic error and that was largely due to a flaw in the type checker. I can imagine many engineers being aghast at the whole concept.
So essentially I've had the opposite conclusion. Unit testing to me is a symptom of bad engineering practices. If you rely on it, something is wrong.