The biggest benefit of unit test isn't catching bugs, but making aggressive refactorings possible.

My experience is that more often than not, unit tests get thrown away in aggressive refactorings, and system level tests survive. If you have a system that depends on units A, B and C, and refactor it to use a totally different structure divided into units D, E and F, then the unit tests for A B and C get thrown away, as they don't really have an equivalent in D, E and F. But the system level test stays, as the system as a whole does the same thing. So, the system level test is what guarantees the safety of the refactor.

That is not always the case, but that happens in most of the large refactorings i've been involved in.

This is true of any code you throw out. If you throw out the code, you should definitely throw out the corresponding tests.

It doesn't invalidate the fact that unit tests increase your confidence in your own code.

The point is, as your tests get more and more fine grained, and the units themselves get more and more fine grained (in my experience these go hand-in-hand with a TDD approach), the changes that the units themselves maintain intact approaches zero, because otherwise the only refactorings possible are completely trivial.

One could also say that bad unit tests make aggressive refactoring impossible. Both sides of the coin don't necessarily make for a persuasive argument.

Huh. Here was me thinking that was what static type systems were about. ;-)

Static typing does not allow you to refactor code with confidence that you didn't break the application logic. Only automated tests can do that, regardless of type system.

Sure. But an awful lot of failures in refactoring in dynamically typed languages would be avoided by a static type system.

As it happens, I am a Rust fanboy with all that entails, so I (like everyone else in such discussions) am clearly biased.

My point was in jest, as the “;-)” (because HN simply discarded a U+1F609) indicated.

static type systems != unit tests

That's what strong type systems are about. A strong explicit dynamic type system with appropriate tests can track down bugs more easily than a statically typed weak type system.

Unit tests can also prevent aggressive refactoring by being heavily coupled to the implementation due to the nature of mocking essentially private details of the code under tests. You then have to aggressively refactor your tests along with your code and all the safety that the test would give is gone.

units are likely to be aggressively refactored, no?

You can achieve this with non unit tests as well.

Sure. I should have just said "test".

System level integration tests also tend to be more flaky. Both unit and integration tests are useful.

Unit tests are also pretty quick to write once you have the mocking setup.

That's one of the dirty little secrets with end to end tests that almost no one talks about. You will probably spend more time running after ghosts in the machine than finding actual bugs.

I've had that experience too. But also the opposite.

I've worked both on code where writing the tests was more effort than the code, and on code where writing the tests was easy, quick and helpful. The latter makes sense, after all a good test is straightline code, zero ifs, zero loops. But the former?

I think the key is that mocking should be used sparingly, but without hesitation.

Plenty of people talk about it:

http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-m...

The dirty little secret that nobody talks about is the cause of the ghosts: poorly engineered tests.

>System level integration tests also tend to be more flaky.

That's usually a sign that they've been engineered poorly or you have bugs in your code.

System level integration tests need appropriate environmental isolation and solid asynchronous & multithreaded code. Nobody can be bothered to write these properly for tests, hence the flakiness ("ooh let's just insert a sleep here" / "eh, does it really matter which version of postgres we run?").

They usually tend to be quick to run too; integration tests are rarely fast enough to run during development.