> It pains me when I see developers testing their functionality end to end, manually

On the other hand I know developers that only unit test. Hundreds of lines of mock code, dozens of tests. No documentation, no sample configuration for the final application and dozens of data races because the unit tests don't cover multi threading. But hey an unusable buggy mess is a net positive as long as it makes the test coverage statistic happy and that is one of those things reported to management.

While a bit exaggerated one of the downsides of unit tests is that they quickly become part of a metric without regards for when they ad value.

> Hundreds of lines of mock code

Mocking should be kept to a minimum and ideally avoided.

Mocking is most useful when adding tests to legacy code, but new code should be designed so it can be tested without the need for mocking.

Mocking own code should be avoided if possible, but it does not apply to external resources like database, http responses etc.

I don't think it's ever a good idea to mock a database. Invest in a test framework that can performantly spin up a fresh database instance and throw it away at the end of the test run instead.

Mocking outbound HTTP calls is worthwhile, but thankfully most testing frameworks I've used (at least in Python world) make that pretty easy.

I mock database by using sqlite in-memory database. It is not 100% compatible with my production database but still god enough for most of my tests. This way I have fresh database for each unit test and the tests are still fast.

Also can be pretty handy when there's hardware that needs something very specific to happen to work properly. Mocks can give you (some) confidence that when you poke the I2C bus, you won't hang it up.

But it's very easy to overdo it and end up testing irrelevant implementation details instead.

Hence why you need automated integration tests, where possible. Especially for regressions.

They tend to have much higher value than pure unit tests.

I agree that automated integration tests typically offer higher ROI than equivalent unit tests written with mocks, because mock code is so difficult to write and fragile.

I disagree with your assertion that integration tests offer higher value than unit tests in general. Don't judge all unit tests by mock code.

I will counter that integration tests tend to have a much bigger footprint: one test covers a lot more ground and so per line of code will catch more things.

But, the failures from integration tests are often a lot harder to read. Even half-good unit tests tend to show exactly where the error is, and make the fix cycle much shorter. It is also much harder to write exhaustive integration tests for all of the inputs that might happen.

I, as a long-term qa-ish person (I usually write testing systems) see real value in having a mixture of both system, integration (multiple systems), and unit tests, as well as end-to-end manual tests. Anything big enough needs all of these.

If errors from your integration tests are hard to understand, that just means you need to improve the debugging information emitted by your software such that your developer can debug it properly using it, no?

This is the point. Both are tools. They're both needed.

TBH mock code just sucks to write, no matter how good your unit tests are.

That is one of the big benefits I have found from writing code while planning to write unit tests for it (even if you don't write all of them). You will generally avoid patterns that will require a lot of mocks to test, which generally make that code easier to refactor and reason about. If you are able to make big chunks of your code into "pure" functions, they are now really easy to test.

Also agree.

I've seen code so mocked it was dubious that it could be testing anything, accurately at least.

A classic approach is that if you have to make some complicated API call with a bunch of arguments, you can put all the logic wrangling the arguments into a pure helper function and unit test the helper.

Agree.

I use strategy pattern with Java functional interfaces to avoid using magic mocking frameworks. That way, my mocks are just trivial impls of the plugin parts.

Setting up sample data is the biggest pain for mocking in my experience. But it's true however you're testing.

How do you accomplish that? I often end up in a situation where mocks are needed (like in the article).

Often times, you will have chunks of code that are either creating input for what would go into an interface/mock, or handling the output of it. You could pull those bits, even if they are relatively simple (maybe only a few lines, and only used in that one spot), and extract those out to a pure function to sidestep the mock.

Sometimes you can't really avoid it. In those cases, I would judge how much of the effort is testing your code, and how much is testing your ability to write mocks. Unless it is a really important functionality, or something complex enough that it would be easy to get wrong, I wouldn't bother with most tests that need extensive mocking. Instead handle those later with integration / e2e tests.

> and that is one of those things reported to management.

Yeah unit testing as I see it is just optimising for the success of middle management, at the expense of both the organisation's goals, and the developers.

You are making predictability the only focus, because that's the metric which middle management is evaluated by. At the expense of performance, end product quality, and work satisfaction. "It was done in exactly the excruciatingly slow and mind numbing boring way, and was exactly as shitty as designed" = success for management.

If you would try to increase performance, end product usability, and/or work satisfaction, you would have to put the predictability at risk, which means that the success of the middle manager will not be the highest priority at all cost anymore.

yep. I've worked at places with unit tests/e2e/and static typing. And I've worked at places with zero unit tests/no e2e/dynamic typing.

The push for more testing and static typing (to degree) was always managers. They trade predictability for momentum.

The reality in both situations is that bugs happen. I've not noticed a difference in frequency of "critical" bugs in either scenario. The only difference is that at the no unit test/no e2e places, getting your code into production was trivial. Which always meant when bugs happen, they are fixed much faster than the unit test-everything place. We're talking minutes vs. days here. Because at a risk-adverse business (that is, any place more than a few hundred people), you have multiple stakeholders that have to sign off if you want to go take a piss in the bathroom.

While I totally agree that process can go nuts and get in the way, I will disagree that testing always makes things slower. Once you are past the prototyping phase, and get into actual production phase (some projects never make this turn) some testing infrastructure can absolutely make thing faster. Rather than having to deal with all of the bugs that your fellow developers are just dumping into the main branch, they can help weed out some of the bugs before others have to deal with them.

Your comment is downvoted perhaps because it sounds so cynical, but after thinking about it I agree to a certain extent. Throughout my career I've noticed the biggest pushers for more unit testing are usually middle managers who love to see that coverage number goes up since it's an observable metric, but nuanced opinions like "more unit tests are not always good and can be detrimental" are generally not welcome.

> Hundreds of lines of mock code

No wonder they are not getting value out of unit tests.

https://blog.metaobject.com/2014/05/why-i-don-mock.html

Many developers seem to lack a focus on automation in the most basic sense of the word as well. While consulting, I see lots of backend developers constantly making a change, waiting for the server to reload with the new changes, switch to Postman/cURL/whatever, fire off their request and see if it's accurate. Rather than just having the request handler as a function that is under test with assertions, and have the test re-run on each change. So much time during a developers day is spent on just repetition, and they don't seem to care.

Plenty of time to check Reddit while waiting for the server to reload.

> Then I showed that a unit test can discover this

Yeah but the catch is that in order for your code to be "testable" you have to rewrite it completely in a way that comes with large sacrifices, making it much more confusing, abstract, larger and much harder to work with, in every way except testing it.

My experience is the exact opposite of this. Modular code that has well defined boundaries and responsibilities can be mocked and tested in isolation. It can also be reasoned about in isolation, reducing complexity. New functionality can be assigned to the proper module because each module has a well defined purpose.

I think the focus that everything should be testable is a wrong one, but almost every codebase has functionality that can be easily unit tested without major refactors. For everything else, integration tests can already help a lot.

Ideally, everything should be testable or replaceable. The non testable parts of your code have network effects on the testability of other parts of the code that are dependent on those parts. This is where dependency injection (e.g via a constructor or function parameters) really shines.

I totally agree that if you have to choose (and usually you do, unless you have unlimited time and funding), choose to implement end to end integration tests first.

The simple verification aspect of unit tests makes writing code easier and more purposeful in my opinion. At a most basic level it's kind of like a small informal proof. When the code base get's complex and larger in size this is invaluable if unit tests are written correctly. Tests have to make sense and be written well just like any other piece of code.

It also helps form thinking about test cases and logic before writing code. Before I used exhaustive unit tests I would just start writing code right away and often times this would result in constant refactoring as the code evolved or during debugging. But unit tests encouraged me to scaffold out abstract functions and write tests before implementing the code. It got me thinking about what the logic and workflows then implementing the functions meant having the tests pass as verification.

So if you an existing, functional, code base, I totally agree. Rewrites in general are rarely worth doing. If you have a new code base, it can be incredibly valuable to design the code such that externalities can be injected. Usually this means being able to swap out file system, RNG, system time, and networking layers. If your code uses a particular hardware platform, it can be useful to put the platform specific code behind an abstraction layer.

Not only does it make your code more unit testable, it makes your code much more ad hoc testable. Again, I would not refactor a large, non-disposable application just to achieve this.

Imho one of the benefits of unit testing is precisely that you have to write/refactor your code to make it more testable and in the process you uncover and fix flaws in lacking abstraction and isolation.

If you write code that is not testable, how do you test it? What you propose is false dichotomy really. Code that isn't testable is almost always confusing, large, hard to work with etc.

Once I wrote my third game in Unity, with unit tests coverage of about 80%. It was least buggy game in my entire career. It was also the hardest time as I had to start thinking more about creating testable components rather than messy spaghetti code I used to write before.

This is one of the best "side-effects" of testing, with the reasoning is that testable code is good code (as in modular, cohesive, etc...)

Games are likely to benefit significantly from unit tests: architecture is complex and likely to be improved by adding clean boundaries and layers for the sake of testability; supporting automation is likely to enable useful features (e.g. deterministic simulations); tests are likely to be meaningful and important.

Have you developed games at all, or are you just speaking from an armchair?

As a gamedev who writes unit tests, I agree with GP. Games benefit from unit tests. That, however, doesn't change the systemic incentives that prevent tests from being written. The ways that gamedev is budgeted/funded is a more important factor than more tests = a more stable/less buggy game.

What engine do you use that supports a unit test framework?

Since testing in game development is very much underrated, it would be interesting to hear more about this. Was this just as a hobby or is the game available somewhere? And have you ever done a demo/talk about how you've approached unit testing with Unity?

not unit testing but a good talk on one team's testing

https://m.youtube.com/watch?v=m5zrfTFKf_E

I’ve found that while I did spend a lot of time at first figuring out how to write testable code, I now am so much better at writing in this way that I am much more productive since everything has great test coverage. Saves me going back to modules I wrote 2 years ago to add features.

Property based tests have the benefit of being fun to actually think about (as far as testing goes) and also often exposing more bugs than unit tests (which are usually more like sanity checks than exploring the boundary of what will break or turn weird your functions).

Property based tests have caught 10-100x the number of buggy edge-cases than the handwritten unit tests I could have come up with. The material gains of getting a combinatorial explosion of test cases from property composition is extremely underrated.

However, there seems to be a general skepticism around property based tests in my experience. Any idea where it stems from?

IMO, you have to code in a certain way to make them ergonomic or even worthwhile - which is also true for example tests. They have some overlap with example tests but have a different advantage. Example tests document function/interface usage and typical or historical errors/exceptions. Property based tests are more about data integrity and invariants.

It requires more mathematical / analytical thinking and code IME.

It pains me when I see developers assiduously write unit tests for every line of their code and every single bug in the code ends up encoded in those unit tests so that they fail when the bug is fixed.

Integration tests need to be the default and people need to learn when to use one or the other.

Are these bugs in the sense that the code's plain wrong or that the developer misunderstood the spec? Good unit tests (when written alongside the code they're testing) should go a long way towards preventing the former.

The latter. Also misunderstandings about APIs. IME these combined are usually something like 70% of all bugs so the ROI on unit tests is not particularly high.

I find integration tests paired with BDD do a good job of catching "code plain wrong" bugs as well as misused APIs and misunderstood specs. They're harder to build and run slower but the ROI is still higher.

Because that seems like they're doing things backwards...

You should know what result you want before you even start writing the code.

>You should know what result you want before you even start writing the code.

In theory, but IME about 50% of bugs end up being specification bugs at the high level anyway.

Once you drill down and start writing unit tests on lower levels to imitate higher level APIs that % only goes up.

Unit tests are extremely boring for people who don't particularly like programming but do it anyway as means to an end. It's just another piece of code you are writing instead of releasing the dopamine of the finished product. It's anticlimactic.

If you first write the test and then try to satisfy them(test driven development) can work to an extent as it can divide the large task into small pieces where you get a prize each time but it also alienates the developer from the larger picture, diminishing their value to the project since they no longer can put their intellectual output into the project.

In my experience, any complex and widely used library is dead in the water without unit tests - if you make a change, you WILL break some use case, ending up in an endless cycle of fixing your changes based on user feedback, and souring your users on your library, who will refuse to update.

If you don't believe this, just write some (good) unit tests, then make the change - it's very very likely they will catch some weird corner case you've forgotten even existed.

I don't doubt the usefulness of unit tests.

I find it makes programming so much more enjoyable both in an immediate gratification and long term way.

More green checks = more dopamine Higher quality code = long term satisfaction.

But there's nothing long term about unit tests, they only verify a particular version of the code and must be tweaked whenever that code is changed to return different results. The "happy green checks" that really help in the long term are comprehensive type checks, reflecting a reasonably stable underlying design.

I don't understand.

Code does things, tests verify that it does the things I thought it did. Getting a bit of validation makes me feel good while working.

If it now needs to do completely different things then it needs different tests.

Also, if you are making huge changes to tests whenever you change a single detail, that's an issue with your code, not testing in general.

Type checks are good too. Using a typed language is even better.

This reads to me as means/ends confusion. If you write your tests afterwards, to reify the implementation of the system, this is true. They may then require radical changes when the implementation changes. On the other hand, if you write your tests first to reflect business rules and requirements, then those tests will change only when the business rules and requirements change. (Which they will, and that's OK.)

I'm not an all-TDD-all-the-time type by any means. You kind of have to at least have something to start writing tests against or it'll just be compiler errors all day long. But for the projects where I've most consistently found success in building correct systems, writing tests as soon as the basic interfaces are in place has resulted in more flexible, more reliable tests that last over the long term.

As for type checks...isn't that what a compiler is for?

Don't you think that it also alienates you from the projects? Green checks are green checks anyway, you can no longer put your input into the project and the best strategy for you becomes the minimal viable effort to earn a green check.

> minimal viable effort to earn a green check

That's literally the thesis of the Kent Beck book.

I'm not some idiot copy and pasting `expect(true).toEqual(true)`

I add functionality one test at a time. I get satisfaction of knowing it works as specified in the form of an increasing count of green checks.

I don't understand why it would alienate me.

Personally, I hate any system designed to give me lots of small bits of validation. I don't want to see green check marks, just show me any errors that I need to think about.

That's literally the same thing. I write a failing test, I fix it. Those are the errors to think about and the green check marks when I don't need to think about them.

It's the opposite. One is motivation from being completionist all-green checkmarks. Where checkmarks are good. I only want to see/think about errors. I don't care if it passed 100 tests or a 1000 tests successfully.

Edit: I get that logically they are the same, but that's true of a half-empty/half-full glass. Psychologically they are different.

Most test frameworks only really show you the failing tests in any detail as the top level results.

As you say, you don't care that there are 20k passing tests. You care just care that there is 1 failure.