The biggest problem I see with people advocating for tests and employing TDD is that they do change how they write code to accommodate tests. This leads to inclusion of lots of unnecessary abstraction and boilerplate patterns that make code less readable and more bug-prone. OO world has spawned numerous non-solutions to turn your code inside-out so that it's easier to mock things, at the expense of code quality itself.

That said, if you go for functional style in OOP, i.e. shoving as much as you can into static helper functions and most of the rest into dumb private stateless functions, you suddenly gain both a clean architecture and lots of test points to use in unit tests. So you can have testable code, but you have to chill out with the OOP thing a bit.

In our work we use C# and it is very hard, even next to impossible to make a static class pass a code review - given it's not for extension methods (which I hate... why not be explicit about the first parameter and stop acting as a part of the class </rant>). They just tell us to use IoC and move to the next point. I honestly don't know why. Our IoC library can treat a dependency as static or singleton, but those are also discouraged. Once I had a static class named GraphRequestHelpers* and the reviewer got really negative, FSM knows why. She told me that we need IoC to make everything testable and "Helper" in the name is a code-smell. Sounds cargo-culting to me but I have only 6 years of experience so who I am to know.

* Now we have RequestExtensions and everything is apparently perfect.

Helper is a code smell as it's a sign of "we don't know what the responsibility of this is or what to call it so we'll just chuck a load of shit in this file and call it a helper". The methods in should belong to something and live on that class, not in an external class.

RequestExtensions is more shit than the original solution. Extension methods are even worse! Shoot the reviewer.

For example would you argue against string formatting helpers? Or would they need to be written to an interface and added to myriad DI bucket lists?

String formats are data so they would be stored as constants so that they are interned. They can be stored in a const class which is a static class with no methods i.e.:

   sealed class StringFormats {
       public const DateFormatX = @"...";
   }

The Haskell community has its dogmas too. And it's fair share of "let's do this simpler" blog posts.

As an aside Haskell has many equivalents of dependency injection and sugars to help and you can "inject all the things" there too.

My point is to think of the problem you are trying to solve, rather than ticking off the SOLID / Martin Fowler etc. tick boxes.

Understanding OO patterns, SOLID etc is a good thing but being prepared to "break the rules" is good IMO too.

Every exception I have shot myself.

Duly noted! Although I'll try talking to her first, I'm sure there's more behind the decision :)

One of the methods that was inside takes a request, extracts the body and returns the parsed graph from the body. It's used by many controllers from many projects. I don't know where to put such a thing, hence the request extension.

Usually that's a single responsibility class:

   interface IGraphParser {
       Graph Parse(Request request);
   }

Best solution would have been a pipe operator if you ask me.

If you start using a custom delegate solution, then your code is not consistent with everything else that uses DI, making it harder to understand. I can understand interfaces are annoying when navigating code, but the IDE still helps with that even if it is a few more button clicks, and the pros outweigh the cons.

I don't see how you have any less control over it than any other code you wrote. If you don't want it to write log statements, then don't do that. Most static methods are small and pure so don't need to write log statements anyway.

> Now your testing will dirty whatever database you're using, as well as run 10x slower.

I've never used a logging framework that didn't allow you to configure where log statements were written, or give you control over the logging threshold for individual classes. However if your method is writing logs then presumably there is a reason, which is just as useful in the tests. If you mock it out then you're testing against different code to the one you will actually run against.

> If you start using a custom delegate solution, then your code is not consistent with everything else that uses DI.

Passing functions as arguments directly is 'DI', just without the need to configure that through an external container. Reducing the amount of interfaces (often with a single implementation) and external configuration makes navigating the code easier.

> Most static methods are small and pure

This is very assuming, tests are a way of being specific about your intent.

If your code is structured using small static functions, you don't have any dependencies in the first place, just arguments you are passed and transform. You will probably create interfaces for external services you depend on, but you can avoid needing to mock them if you express the transform directly.

> This is very assuming

I'm not assuming anything, since I wrote the static method and I also decided to call it, presumably for the result it calculates. Your argument appears to be that static methods could contain bad code but that applies to all code you depend on.

> what if at some point later another developer adds a database call into the static method to do some logging?

Then you have a developer that does not grasp the idea of functions, and how they can help you improve your code. That's a call for education, not for changing your tests.

You don't need interfaces for everything in order to do DI. Interfaces should be used only for having multiple implementations or to break dependency loops.

Other than that I'm in agreement, static methods generally aren't a good idea. They can all too easily balloon into big chunks of imperative code with inner dependencies (static or not) at 10 indentation levels deep. Non static methods can too, but not as easily, and you have more options for fixes/workarounds in those cases anyway. The only place they really make sense is as part of a set of pure primitive data transforms, and ought to be small.

It sounds like a problem is a few layers higher. Why is there a live database in your unit testing environment? Why are working credentials configured? If they're unit test, not integration tests, all db operations should be DId / mocked / whatever. Any call that isn't should fail, not take longer time. Db interaction is for the integration tests.

    static_function(db, other, arguments) { ... }
    test { static_function(fake_db, 1, 2) }

    void TestCreateUser() {
        var repo = new UsersRepository();
        var mockUser = new User("John", "Smith");
        repo.AddUser(mockUser); // db call
        var addedUser = repo.GetUsers().Single(); // db call
        Assert.StructureIsEqual(mockUser, addedUser);
    }

IAdder { double Add(double a, double b) }

Test:

car mockAdder = ... // Mock 1+1=2 etc.

There is no functional difference between a class with static methods and a class without, of which one instance is available to other classes.

Other than the fact that it isolates state, allows mocking and substitution and testing.

Static methods usually don't rely on any hidden state at all. The example originally given was for a graph operation which could just take the input graph as an argument and return the result. When your code is composed of small independent functions you don't need mocking and substitution at all. In my experience most uses of mocks come from functions that do too much in the first place.

Static methods with no side effect are wonderful, but static state is really bad and static methods which perform IO are horrible because they cannot be mocked in a unittest.

But some people miss this distinction and just say static methods are bad for testing.

I practise TDD for peace of mind - if I add new functionality to existing code I can be 99.9% sure I haven't made any regressions. When a client's system goes down on a friday, I can 99.9% guarantee it wasn't my code that is at fault. If I have to work at the weekend to update a production server, I'm 99.9% sure it'll go smoothly as my tests say it will.

I can actually write entire features with appropriate test coverage from the ground up and they work first time and have close to zero defects in production.

It's amazing when you spend 5-6 days writing code that does nothing and at the last moment, everything slots together with a few integration tests and wham, feature done. Not talking trivial stuff here either; big integrations across several different providers/abstractions, bits of UI, the lot.

You see a lot of people arguing against this but I'm going to be honest, they churn out a lot of stuff that doesn't actually work.

My anecdata cancels out your anecdata. The TDD practitioners that I've met have, without exception, written code that worked fine for only the one case that they've tested. Example: They'd test a method for sending a message with the string "hello". Turns out the method didn't URL-encode the message before POST-ing it, and sending anything with a space was broken. They were confident and pushed the change.

Not saying you're wrong, just that TDD doesn't seem to work for everybody, and can even be a distraction.

If you only test the expected outcome you are a dumbass.

I use HttpMock myself in test cases which fires up an http server for personal projects. We use Wiremock commercially.

1) Write a test that runs the service and saves output to a file. 2) Mock out the call to just return the data from the file and validate results. 3) If you need variations on this data just modify the file/data (often as part of the test)

I usually leave number 1 in the code but disabled since it often relies on remote data that may not be stable. Having the test run more than once is not very beneficial but being able to run it later and see what exactly has changed is great.

That leads to you either not writing tests or having to refactor bits of it. It's easier to do this much earlier on.

To liken it to something you may be familiar with - when commenting your code, do you think it's better to add comments in as you write the code? Or add in the comments at a later date after the code is all written? I'm sure you immediately know which approach results in better quality commenting, and it's the same with TDD.

Not to take the analogy too far, but usually when writing a chunk of code I can keep it's behaviour in my head for a good amount of time and find it's best to add comments at the "let's clean this up for production" phase when you can take a step back and see what needs commented. If you comment as you go, you'll have to update your comments as the code changes and sometimes throw comments out which is a waste of time.

Likewise with tests, I'm not saying write them far into the future, but I think having to strictly stick to red/green/refactor is going to waste time. What's wrong with writing a small chunk of code then several tests when you're mostly happy with it? Or writing several tests at once then the code?

Doesn't that get caught in code review anyway though? I find being forced to write tests first can be clunky and inefficient. Also, I've worked with people who insist on the "write the minimum thing that makes the test pass" mantra which I find really unnatural like you're programming with blinkers on. TDD takes the fun out of coding for me sometimes.

Generally I'd rather sketch out a chunk of the code to understand the problem space better, figure out the best abstractions, clean it up then write tests that target the parts that are most likely to have bugs or bugs that would have the biggest impact.

I find when you're writing tests first, you're being forced to write code without understanding the problem space yet and you don't have enough code yet to see the better abstractions. When you want to refactor, you've now got to refactor your tests as well which creates extra work which discourages you from refactoring. When the behaviour of the current chunk of code you're working on can still be kept in your head, I find the tests aren't helping all that much anyway so writing tests first can get in the way.

You find that you need to do less mental work, as your tests make the required abstractions apparent for you. 'the minimum thing that makes the test pass' ends up being the complete solution, with full test coverage. Any refactoring done is safe from regressions, because of your comprehensive test suite. And when other colleagues inevitably break your code, you already have a test lying in wait to catch them in the act.

As much as I like the idea of TDD, I have a problem with this part. When some refactoring is needed, or the approach changes, it seems like you have two choices. One is to write the new version from scratch using TDD. This wastes extra time. The other is to refactor which breaks all the guarantees you got before. Since both the code and the tests are changing, you may lose the old coverage and gain extra functionality/bugs.

And unfortunately in my experience, the first version of the code rarely survives until the deployment.

Also some refactorings are easier with tests, some are harder.

The kind @viraptor mentiones is the kind that spans more than one compoment. For example when you decide that a certain piece of logic was in the wrong place.

The kind of refactoring that becomes easier is when you don't need to change the (public) API of a component.

Take for example the bowling kata. If you want to support spares and strikes and you need extra bookkeeping, that's the easy kind of refactor where your tests will help you.

But if so far you have written your tests to support a single player and now you want to support two players who play frame by frame... Now you can throw away all the tests that affect more than the very first frame. (yes in the case of the bowling kata, you can design with multiple players in mind, but that's a lot harder in the real world when those requirements are not known yet)

I've been forced to follow TDD for several years and also been given the same kind of comments to downplay any reasoned arguments against it which I find frustrating to be honest. I don't see why the benefits wouldn't be immediately apparent.

> You find that you need to do less mental work, as your tests make the required abstractions apparent for you. 'the minimum thing that makes the test pass' ends up being the complete solution, with full test coverage. Any refactoring done is safe from regressions, because of your comprehensive test suite. And when other colleagues inevitably break your code, you already have a test lying in wait to catch them in the act.

You can do all of the above by writing tests at the end and checking code coverage as well.

With the right tests this works great. I have also seen the opposite where a test suite was extensive and tested the last details of the code. Then the refactor needed more time to figure out what the tests are doing than the actual refactoring. As often, moderation is the key to success.

Only when I understand the problem, the SDK and the expected result I could start writing some tests.

TDD seems to be for business applications where you're using tried and tested technology to implement new problems.

Scientologists make the same argument

If you don't know how to solve a problem you actually need to do some research and possibly try a bunch of different approaches. Over encumbering yourself with specific production focused methodologies hurts. If you're doing something genuinely new this can be months of effort.

After the fact you should go back and rewrite the solution in a TDD manner if you think it benefits your specific context.

Why? Why not add tests to your existing code?

When I was a UX prototyper, none of my code ever made it into production.

That's why it's better to start with the highest level tests first and then move down an abstraction level once you have a clearer understanding of what abstractions you will need.

I find TDD proponents don't take into account that writing tests can actually be really time consuming and challenging, and when you've got a lot of code that is tests, refactoring your tests becomes very tedious.

Define "all written". If we are talking about a new function - obviously you write you comment for it after the function ready to be commented on. And obviously you won't be commenting every string you put there, right?

Now, if we are talking about the whole new feature, that can consist of many functions and whatever - yeah, you usually comment your code in the process of writting the feature, rather than doing it at a later time, which will never come.

Trying to write unit tests afterwards lands me with something that appears as more of an afterthought or add on. It doesn't have to be this way I suppose, but it is more prone to.

This might be because I am more used to the red, green, refactor method though.

Of course, types alone can't represent every error cases out there (especially the one related to number or string), so I still write Unit Test for those cases. But the number of Unit Tests needed is much lower.

It completely fails on concrete types. That's the reason haskellers create a DSL for everything.

Because I am always keen to understand how to TDD GUI code and I don't mean the data model behind the pixels.

I usually find I build a general page object first ('this text is somewhere on the page'), then write the UI, then make the test more specific if I can after (but it's an art, as too specific and you risk creating too many false negatives when you make UI changes).

(Also as you are interacting with the UI, these would be known as integration tests.)

[1] https://semaphoreci.com/community/tutorials/using-page-objec...

Which is why it is my main question on TDD talks, that always use some library or CLI app as example.

Edit: Slides from the talk I saw: http://slides.com/bahmutov/snapshot-testing

That said, if you go for functional style in OOP, i.e. shoving as much as you can into stateless static helper functions and most of the rest into dumb private stateless functions, (...)

Of course I do not mean you should abandon objects where there is a strong connection between a set of data items and operations that work on them, or where polymorphism is a right abstraction. But from my experience, quite a lot of code is made through transformations applied on simple data, and when you write that kind of code in a functional style (whether as static methods grouped in helper classes, or private methods within an implementation of your class), both quality and testability rises in lockstep. And my point is that quite a lot of code can be written this way even in an OOP project.

Wow, this is exactely totally opposite of how one can achieve testability in OOP! For more details I recommend excellent Misko Hevery's article "Static Methods are Death to Testability" [1]. Also, I'd argue that "functional style in OOP" is an oxymoron - you're either OO or something else (functional, imperative...)

[1] http://misko.hevery.com/2008/12/15/static-methods-are-death-...

> The basic issue with static methods is they are procedural code.

So is any object-oriented code. OOP is a subparadigm of procedural programming.

> Unit-testing needs seams, seams is where we prevent the execution of normal code path and is how we achieve isolation of the class under test. seams work through polymorphism, we override/implement class/interface and than wire the class under test differently in order to take control of the execution flow. With static methods there is nothing to override.

Why did it not occur to him that the function boundary is the "seam" he's trying to find?

I mean, `method(a, b)` is equivalent (as in: equally expressive, and usually implemented in the same way) as `a.method(b)`. Therefore, any problems with one case equally apply to the other case. If his problem is that `method(a, b)` may call other, non-mockable functions, then that criticism equally applies to `a.method(b)`.

(As I'm writing this, it occurs to me that the author may be suffering from the "OOP = Java" delusion.)

Go with what the ecosystem supports, and you'll find your tooling helps you a lot more than if you fight against it by trying to force non-idiomatic structures. Your colleagues will appreciate it, too.

For me, IoC seems invented almost entirely to make up for how difficult testing can be in particular languages. Which, sure, making up for shortcomings as good, but the necessity to use IoC for it feels bad.

Nowadays the benefits are clear to me: more modular, testable code, and also lifecycle management.

Give it a try, you may well change your views.

I agree that it's one of the sanest options when it's required, I just think that language design should incorporate testing ergonomics from the start.

That means you didn't actually change the code. It means you added unnecessary abstractions around your code in order not to change it.

Unit tests guide you towards simplicity. In my experience, the only times they haven't done that is when I have made some assumptions about what the code should be and not allowed the tests to drive me towards that simplicity.

http://blog.metaobject.com/2014/05/why-i-don-mock.html

    That said, if you go for functional style in OOP, i.e. 
    shoving as much as you can into static helper functions and
    most of the rest into dumb private stateless functions,

method(a,b) is equivalent to a.method(b) and exactly as much typing. You do save manually typing the extra part of the definition but 'eh'. A few languages treat these interchangeably.

Have you considered the possibility that those people are right? That's a reasonable conclusion to make if you are seeing lots of otherwise smart people that share an opinion that disagrees with yours.

There are lots of valid reasons to change the style in which you write code. In my mind, fitting somebody's fad testing scheme is not one of them.

Here's a second opinion from a guy who also likes tests, but doesn't think it's a good idea to structure your whole codebase just to accommodate them:

http://david.heinemeierhansson.com/2014/test-induced-design-...

    Have you considered the possibility that those people are right?

Anything that is this hard to get right shouldn’t be the equal responsibility of every team member. For every other problem of this magnitude we have people who specialize and everyone else just has to be aware of the issues and consult when in doubt.

So it’s a struggle for me to try to get people do adhere to the strategy we’ve accepted without believing it’s the end all be all of software robustness. Because I’m not convinced. Nothing I’ve ever mastered in software has taken me half as long as testing, and that just ain’t right.

That said, I still like the structure about 80% of my tested code has. It usually does exactly what it says and nothing else. Local reasoning is a big deal to me.

Obsession of "one size fits all" or "silver bullet". I believe the authors of agile manifesto wrote this disclaimer.

If it doesn't make sense to make unit tests to MVC controllers, then don't.

In my experience, management looking at code coverage not being 100% is one reason (although bad) that this "unit test everything" happened. I tried shouting this out, but team lead didn't have the ability to learn from a junior and use Sonar's configuration.

[1] Obviously some things do make sense to put behind interfaces, but I find that most Java developers go interface crazy and the code ends up being an unreadable mess.

If you mock everything, when you refactor, the test will break because the dependency structure will change, and the mocks are no longer relevant to the new implementation. You have to rewrite the tests. You did twice the testing work and more importantly you get absolutely no proctection against regressions because the tests for the 2 versions are not the same.

If you build integration tests, they can remain the same. Less work and actual protection for your refactor.

Although, if you offshore development here in the third-world where internet gets slower everyday. Running an integration test that queries to Amazon RDB can take forever.

I hope this issue gets a spotlight and be noted that integration tests in third world countries is very very slow. And this high cost should be included in the estimates.

To give you an idea, here it takes AT LEAST 5 seconds to load a page from the amazon console. Lol, even the software companies owned by the ISPs/Telco here complains that their access to AWS is super slow. They said that bad routing is the main issue and for some reason the ISP isn't doing something about it.

Why wouldn't you run your test on aws if it needs to integrate with rdb anyway?

Or are your code changes so massive that "git push" is slow?

All this while you are expected to fix 10 tickets for the whole day plus anything that goes wrong in production.

If the code changed due to a big behavioral shift then your integration and E2E tests aren’t safe. It’s more than twice the work at the higher layers because people get trapped by the Sunk Cost Fallacy. They try and try to save the old tests before they finally rewrite.

That is the observation that convinced me to stick to unit tests. People aren’t emotionally attached to individual unit tests.

I generally avoid the term altogether and recommend testing stable API's (which are often public) and avoiding testing internal API's that are more likely to change. This assumes you have a stable API, but that's true of most libraries.

SubcutaneousTest

https://martinfowler.com/bliki/SubcutaneousTest.html

I've seen what happens when a developer tries to abstract away a database in a database driven app so it can be "better unit tested". It's a goddamn mess.

If your app relies heavily on using a database, your app naturally integrates with a database then it makes no sense to test without it. You are intentionally avoiding testing in a way that will pick up bugs.

>Unit tests run faster, are written faster

Unit tests test less realistically. That means they don't catch bugs integration tests do.

They also often take longer to write and are more tightly coupled.

How coding this way came to be seen as a best practice is beyond me. Tight coupling and premature optimization is usually seen as bad practice in other areas.

Also, with Docker it's now actually feasible to automatically test against a real database at a reasonable speed. A Postgres container spins up in a couple of seconds, a SQL Server one in a little over four.

But you can of course achieve the same by running a script before your tests start. There are also some frameworks for doing this sort of thing too, such as Fixie for .NET

There are also other scenarios that become really simple with disposable DB instances. Want to test a remote data transfer feature? Just spin up two databases.

Namely a context like travis-ci where you get a new clean environment each time.

Personally, I've develop few libraries/web UIs that relies on external software like a DNS server or an ldap directory.

And for those, I've a quick and dirty shell script that deploys bind/OpenLDAP.

It's far easier, faster and more accurate than mocking.

For example, what comes to my mind is the testing I do for all the SSL/TLS modes I support (SSL, StartTLS, certificate check disable or not, etc) in my ldap web application.

Travis also has available services in their build envs (stuff like redis, mongodb, mysql, cassandra...).

The reason? They don’t “believe” in unit-tests. They don’t think unit-testing “works in the real world”.

They absolutely fail to accept that they need to write their code differently for automated testing to work well.

How do you change such a mindset?

I wrote a component at work. Sure, there are some unit tests (the SIP parser working? Check. But for that component, unit tests only go so far as I need to query another program that actually implements the business logic (it goes SIP -> my program -> custom protocol [1] -> business logic program and back again). To mock the business logic program (I need to make sure to return the proper information per the SIP request) is to reimplement the business logic program, so when testing my component, we also use the business logic unit. The business logic also requires two more programs to run. At this point, there is no difference between a unit test and an integration test as I'm running five programs ... no, ... six, I do have to mock a cell phone (basically, respond to a custom protocol and make a web request), to test the "unit" that is my program.

Oh, and to make it even nicer, this is legacy C and C++ (C with classes) code.

[1] Legacy code. It works. At the rate of production deployments we (our team) gets, it would be around two years [2] to remove the custom protocol. So it stays.

[2] Have I mentioned the very scary SLAs?

Next bug you find that needs a unit/functional test (e.g. it is caused by a simple error in transformation in one function), write the test first as a table of inputs vs outputs, find it fails, fix the function, and leave the test in. Gradually, the code base will contain unit tests which are useful, people will see they are useful, and other people might start using them too where appropriate.

You are unlikely to persuade them without actually doing what you say is beneficial and exposing others to its benefits.

A test accompanying a bug fix holds value in many ways.

Firstly, it demonstrates to those reviewing the change that the fix is suitable.

Secondly, the presence of a test encourages reviewers to consider what a test does and doesn't cover, sometimes resulting in comments regarding improvements that had not otherwise been considered.

Thirdly, and of most importance in the long term, a test for a bug fix serves to document oddities previously discovered that were for a time not known about.

Tests that go along with bug fixes are some of the highest value tests, but they must be previously failing tests.

I can’t tell you how many times I’ve reviewed fixes with “tests” that are basically affirming the consequent; they assert something that was already true and it turns out they’re not actually fixing the reported bug.

If it's for 100% test coverage: forget about it.

If you test private methods: you're doing something wrong.

What people usually see from the "unit test evangelists" are codebase for which you have tests for every method in the code. Then you do some refactoring and you have to rewrite tons of tests. And as those tests are just made to get 100% coverage you end-up with logic bugs because most unit tests have been written to go through the code, not check limits and edge-case. When you stumble upon this kind of test harness you can only think this as only cons (more to write upfront, less willingness to refactor) and no pro (the code is still brittle). Then your integration tests feel like your real harness: you can change anything in your code it'll tell you what has been broken when used.

Now if you consider your unit tests as a kind of integration tests for the API your classes present then you get the benefits of unit tests. But this mean testing only public methods. And mutation testing resilience is a better metric than test coverage.

Also: those tests do not replace a real documentation which can be a lot faster to read and understand than code.

As for the difficulty or refactoring, if you refactor the private unit, you ensure that its tests continue to pass, since its consumers depend on that behavior: you ignore the failing tests of the consumers so long as the subordinate unit's tests are failing. If you eliminate the private unit, you eliminate its tests. Modifying the behavior of the private unit may be equivalent to eliminating the unit or refactoring it. The number of tests you will have to modify is equal to the number of units you modified the behavior of: the branch count of the private unit, or that plus its consumers. If each private consumer was responsible for testing all of the edge cases of the private unit, then you will have to change its branch count multiplied by the number of consumers worth of tests instead.

The distinction between private and public is wholly synthetic. It is a binary layering mechanism that does not map well onto most architectures which have many layers. From the perspective of many full architectures, everything down in the data layer is private: no customer will have direct access to the data layer. Yet you will still test the data layer.

The internals of a library are not special simply because the layering is thinner and binary.

A side-benefit is that the tests for the original class can be a lot simpler too, as I can just mock the responses from what used to be the class internals. Another benefit for libraries, is that it allows consumers to swap out your implementation. I've lost track of the times I've wanted to change something deep inside a library but it's implemented in a private method so I can't even override it.

This does lead to more, smaller, class files. But unless taken to extremes I've not found it to make things less comprehensible, and it definitely makes things more composable.

Maybe a silly question, but why?

If I refactor a class to pull a common piece of functionality into a private method, why would I not want a test for that?

One of the principle benefits of tests I see is allowing me to change the implementation without worrying about the behaviour, and I'm not sure why that wouldn't apply to private methods?

Testing private functions also seems to be a smell that the overall setup of testing the class or function is too difficult. This can be because the class has too many branches in it, the argument list is too large, or too many other systems must be in place for it to function correctly. This, to me, indicates a public interface that is hard to use and will pass much of these issues on to the caller.

Lastly, if you are testing private functions to gain coverage then arguably the behavior in the private method isn’t actually useful to the public interface. The reason I say this is that testing the behavior of the class should end up touching all branch conditions inside the class or the public interface isn’t fully tested. By only testing the public interface it then also becomes easier to locate dead/unreachable code.

Hope that answers the why.

As for the smell, sometimes things are irreducibly complex. Some things in this world do require tedious book keeping. All the refactoring in the world cannot change the degrees of freedom of some problems.

Tests on consumers should not test branches of subordinate units. If you did this then the number of tests would explode exponentially with the number of branch conditions to handle all the corner cases. If a private unit produces a list of objects, but has special cases for some values of its argument, test those branches to verify it always produces the correct list. Then just make sure each caller does the correct thing with the list of objects. That is the purpose of separation of concerns: the consumer does not need to know that some values were special.

Then wouldn't you want to write something that was able to iterate through those edge-case interactions and ensure they are correct?

There should be no situation where the same exact call in the public interface could take multiple different paths in the private method. The only thing I can think of that could make that happen would be some dependancy, which should be mocked at the top level to control these cases.

But people who use your class don't care. They input something in your public methods and expect something in return. The details of what happen inside should not matter. Adding tests there only help to slow you down and make the dev team resist needed changes. And when you add tests you increase the chances you make them useless or wrong.

In Python I am more loosely goosey about my unit tests and unit tests there are more for helping me write/think about tricky code.

This is how you write brittle tests which fail easily and cause high maintenance costs and reduced confidence in the unit-tests as a safety net.

Definitely an anti-pattern.

Im a realist and don’t see any point or value in chasing that goal for a 20+ year old company code-base.

But I expect new modules to be fundamentally testable.

> if you test private methods: you're doing something wrong.

Agreed.

> What people usually see from the "unit test evangelists" are codebase for which you have tests for every method in the code.

Is that really so? It’s easy to be opposed to extremists of any form.

I only “evangelize” that business-logic should be tested and thus needs a code structure to isolate the business-logic from its dependencies (databases, services, factories, etc).

I find that perfectly reasonable.

One of the core principles of TDD is that you write tests to facilitate refactoring---to make it _easy_ to refactor and have confidence in the system after doing so. I've been practicing TDD for ~8y and this is rarely a problem. TDD encourages good abstractions and architecture that lends itself well to composition and separation of concerns.

But if you change an implementation, of course a test is going to fail---you broke the code. It works exactly as designed. What you want to do is change the test first to reflect the new, desired implementation. Any tests that fail that you didn't expect to fail may represent a bug in your implementation.

Of course, I haven't seen the code, so I can't comment on it, and I won't try to do so.

That's why I agree that the focus should mainly be on integration tests. Or at least functional tests. Ideally what you want is to have a system where all the state and logic is in a model (that model can include an external db). The gui should be as much as possible a function of the model i.e. model-view. Then you write the majority of your tests as integration tests against the model and include as many scenarios as you can think of. These tests should reflect the requirements/interface for the system and not the implementation. You should write some gui tests but these should be much less. They just need to verify that the ui reflects the model accurately. You shouldn't be testing scenarios as part of the gui tests.

I have come across too many code bases where the unit tests test that the code is what it is, rather than the code does what it should. Where 'what it should' == 'requirements/interface' == 'something close to integration tests'

That is, as far as I know, the only tangible evidence that unittests are good unless you need to get something out the door quickly (which sadly is most of it).

I'd argue that is not the main benefit of unittesting however. That is the way code is structured, and especially how dependencies are explicit, e.g. injected with constructor arguments.

IME you can't. If they even recognize that what they are doing is not unit testing then your doing well.

> They absolutely fail to accept that they need to write their code differently for automated testing to work well.

I've been thinking lately that having to code differently might be a fault of the tooling that's built up over the years and that they might be right. I've been getting back into c lately and had a look at the mock/stub options there which were very complicated and not very compelling compared to what I've been used to in the .net world. In the end I found the pre-processor was the best (for my project) option:

#ifdef test #define some_method mock_some_method #endif

The advantage has been that the code is written exactly (more or less) as it would have been if there were no tests. There are no interfaces to add, functions to virtualize or un-static and no dependencies to add, this all translates to no performance hit in the production code and the project being simpler all around.

Such behavior is not science. In science, you don't believe, you understand not faith. You check evidence, you use reason.

A mindset that relies on "believe" but not on reason. It maybe because of decades of advertising. Can be a side-effect of participating in believe movements.

How do you change such a mindset? For me, it was reading lots of philosophy and atheist vs theist debates. For Socrates, he died for it.

The definition of legacy code for the author (which I like) is: untested code. So the book is more about getting code in a testable state than random refactoring to get to Clean Code level.

It has helped me refine what I consider good code and good effort wrt to testing.

Because they can find bugs and errors which unit tests cannot.

Nobody disputes that if they came for free then full unit test coverage would be a good thing. The area open to reasonable debate is whether they give the best bang per buck in terms of testing (as opposed to the role of tests in TDD - which is a different kettle of fish: http://www.drdobbs.com/tdd-is-about-design-not-testing/22921... )

A system test catches the backend API issue. A unit test can demonstrate that my component degrades gracefully if the backend API is not available (because I used a mock to provoke a timeout response).

That said, the big issue with unit tests is they don't test the glue which is where a lot of issues happen. In languages with strong type systems, this is less of an issue.

Unit tests are great when you actually have complex logic with relevant corner cases, but when you're building web apps, 95%+ of your code is just boilerplate data munging

In my experience, the vast majority of test failures (after I've made a code change) end up being issues with the tests themselves, and not with the code change. If you're testing obviously-correct code, that's just more that can spuriously break later and require time and effort to fix and maintain.

You probably meant "Because they can find bugs and errors that unit tests cannot." to remove the ambiguity.

tl;dr, I am illiterate; but a little change to the original comment could make it easier.

If you are writing a library write unit test, if your app mostly binds two libraries together unit tests are meaningless, write integration tests.

For algorithms, I love how I can refactor the implementation and still have 100% confidence in the result if it tests sample input against expected output.

Because they test that you are actually using some available network ports, have the correct database model in mind, didn't mistake the version of your libraries, got the deployment script right, and isn't just restarting everything in an infinite loop?

Or maybe because E2E tests actually test stuff your company cares about, instead of some made-up rules that you've got from nowhere?

Really, if you have unities, you should unit-test them. But your E2E tests should be the ones you really care about, and they should certainly not break randomly with unrelated development. If yours are lasting for 5 quarters, you may be doing something wrong.

I’ve been doing automated testing for a while. It’s hard to learn, there aren’t many people to emulate. Well, there are people to emulate but the winning strategies are conterintuitive, so your gut fights you the entire time. It took me 8 years to feel confident in tests and my own test code routinely makes me sad because I find antipatterns and I should know better. Also other people copy all of my mistakes :/

I’ve seen a number independent groups two or more years into their testing adventure and the failure modes are not that different. Everyone pretty much makes the same mistakes I do, and it’s frustrating watching everyone go through the pain before they accept that something has to change and it’s probably them.

The best strat I know of for testing is to use inductive reasoning and sampling to verify. If you don’t like the plumbing analogy then this is the Logic version of the same thing. If A -> B and B -> C then A -> C. Only a couple of your tests should verify A -> C and the bulk should check every kind of A [edit] and every kind of B.

If you want to do things like this without making your code not ‘say’ anything (a huge pet peeve of mine, so I can empathize with your concerns) then there are a couple of things to do there. One is an old trick from Bertrand Meyer: split code that makes decisions from code that acts upon them. Beth is split leaves the code more legible, not less.

Most of the boundary conditions are in the decisions. And this code is side effect free you can test the hell out of it with no mocks. Getting every permutation is straightforward and you can count your tests and your conditional branches to figure out if you are done.

Once your code looks like this, adding and removing new rules to the system later is a snap. Even much later.

Sorry, but I am still unconvinced people got that reason correctly.

Let's say you have that A -> B; B -> C pipeline. How many tests you should have on each step (and on the origin) depends completely on how much freedom that steps grants you. It is not something one can say generalities about.

For example, if you are writing an enterprise CRUD application, almost your entire freedom resides on the data mapping. That means that your tests should be equally divided between data validation and data storage/retrieval. And the second can only be done at the integration or E2E levels.

If you are writing a multi-client statefull server (like a threaded web server), the freedom concentrated on launching and reloading it is so large that you can't even reasonably test for it. You'd better design your software around proving this is correct and let testing for less problematic stuff.

My biggest issue with the unity test pushing isn't even that it forces a bad structure into the code (what it does), or that it's pushes for fragile and valueless code (what it also dies). It is that it's wrong at the larger level, oversimplifying stuff and letting people get out of the hook without thinking for themselves.

They can only test parts of its behavior, not everything, and are too brittle to any simple UI/UX change.

You suggest that instead of writing integration tests, this problem can be avoided by "writing better code". But how exactly would you rewrite the code to avoid the above problem? Declare that A and B should not interact at all, and move all their interactions into class-C? Now you've just given a new name to the same problem: "How do we adequately test class-C?" And once again, the correct answer is to ease up on the mocks and just write some integration tests.

You might be speaking about those cases where people write large god classes, pervasively side-effectful code, zero API design, and a general lack of pure abstractions - then their tests would equally be bad. Tests are code, so one's ability to design programs would reflect on their tests and vice versa.

But a reasonable programmer who cares enough about what they do can still end up with a brittle test suite because of other factors.

Unit tests written in a dynamic language is a major drag on refactoring. It is not so much that we test the internals of the system or get tied up with the shape of code internal to an object. Even if you follow Sandi Metz's wonderful guidelines around testing: i) do not test internals, ii) test incoming queries by asserting on their return value, iii) test incoming commands by asserting on their side effects, iv) mock outgoing commands, and v) stub outgoing queries, you end up with a brittle test suite that is hard to refactor thanks to connascence.

Whenever you refactor your names, or shuffle the hierarchy of your domain elements, you are now left with a thankless chore of hunting and pecking your unit tests and making them reflect the new reality of your code. Here integration tests help you know that your system still works end to end, and unit tests simply remain that one thing that refuses to budge until you pay it its respects.

Unit testing complex views is still a hard problem. There are no well-defined stable "units" to speak of in an ever changing HTML UI. We have snapshot tests, we try to extract simple components on whom we can assert presence/absence of data, and we have integration tests that blindly railroads over everything and make sure the damn thing worked.

But in a different context unit testing is the one true answer. If your system is statically typed (say in Haskell or OCaml), and your functions are pure and compositional, you don't so much worry about mocking and stubbing. You can make simple assertions on pure functions and as the granularity of your functions increase, they end up covering more parts of your system and get closer to an integration test. Static types form another sort of guarantee, the most basic one being that it takes a class of bugs away in form of undefined data types, the system going into invalid states, and of course the clerical mistake of named connascence. We often abuse unit tests in dynamic languages to cover these scenarios, leading to huge test suites with very brittle tests.

I think it is important to call out that the value of unit tests are still contextual - "it depends" like everything in the world, and despite our best efforts, they can become hard to refactor. There is a case to be made for writing integration tests because they deliver more business value at a cheaper price than a pervasive set of unit tests when dealing with a highly effectful dynamic system. This lets us also think about other forms of testing like generative testing and snapshot testing that come to the same problem from different angles.

When working with dynamic languages I always end up writing a bunch of unit tests and a bunch of integration tests. I've experimented some with type hinting and static analysis in some dynamic languages but it's not the same as having the compiler make guarantees.

Honest question: Are most of your unit test failures due to bugs or due to refactoring (e.g. changing APIs)?

Most people I know who do unit testing have mostly the latter (easily over 80% of the time). At that point one feels like they are merely babysitting unit tests.

If unit tests have such high false positives, how useful are they?

Note I'm not saying that it's not possible to write unit tests that are relatively immune from refactors. But it is more challenging and, in my experience, quite rare to find a project that writes them this way.

What I know is that when the dev team can’t adapt to shifting business requirements is a world of pain for everyone. I try to test a lot of business logic as low in the test tree as I can and when they change Always to Sometimes or Never to Twice, I roll with it. Because I know that Never and Always mean ‘ask me again the next time we court a big payday.’

What I remark on are the tests that get deleted when the business says they’re wrong. And those happen but the toll is easy with unit tests. You just ask and answer a different question and it’s fine.

I've run into this a couple times and noticed at some point that writing unit tests while developing (not necessarily TDD) helps a lot in clarifying boundaries early on, and generally improves code quality.

I want a statically typed language with reasonable affordances for FP, for the 20% of the code that is dead ugly when forced into object structure.