I think TDD (or rather, unit tests):

1) ... over-emphasise the importance of reduced coupling, and can actually increase the chances of integration failures. Why? Because components are tested in isolation rather than together, but are still considered "tested" - especially when using stubs and mocks. I've seldom seen mocking used well, it almost always over-specifies implementations.

2) ... increase the tested "surface area" of the code, making it much more costly to modify in a way that changes the "surface area". When two bits that normally only ever face one another are tested on both sides, you now have 4 places to update when you move functionality between them instead of 2. And that's just for a small refactoring.

3) ... encourage a spurious modularity that increases the overall conceptual complexity of the solution. When every dependency needs to be replaced for testing, they become parameters, one way or another; and thus the code becomes over-generalized and over-abstracted, more removed from the work it's doing and more concerned with the bureaucracy of communication and coordination. It's this spurious modularity that's the cause of Enterprise Java FactoryFactories and friends.

That sounds quite negative on unit tests, and actually I'm not negative at all. I think code can be modified in more than one way to accommodate tests; poorly, by introducing single-implementation interfaces everywhere; or better, by reducing the number of types of interfaces, converting control flow into data flow, and generally adopting more functional and higher-order compositional architectural patterns.

It's hard to get more specific without talking about examples, though, so I'll stop here.

I worked at a place recently where all of these factors, but especially (3) turned out to be a huge cost for an ostensibly simple service that it severely hampered any further development. Since it was a hard TDD shop, any discussion of cutting out all the intermediate layering was completely off the table. Internal structure was easily 75% of the size of the project.

After all, a lot of effort had gone into building all of the internal mocks, and while many of the early design decisions enshrined in the tests were quite dubious, to revisit those assumptions would have been backsliding. Even though there were no users to speak of yet, the existing tests had to pass.

And of course because of (1), the product didn't actually work because there were no integration tests at all. Because all of the unit tests passed, the ongoing assumption was that the platform was basically sound.

The most charitable thing I can say about TDD given that context is that it might be valuable, but isn't sufficient.

“Any technique, however worthy and desirable, becomes a disease, when the mind is obsessed with it.” —Bruce Lee

I've always thought that the bulk of the value of tests is for the unfortunate person who has to refactor or extend your code years down the track.

By that logic, if you write testable, but untested code, then you're still making it difficult for people to refactor later. This applies even if the code is well designed and uncoupled.

> the value of tests is for the unfortunate person who has to refactor or extend your code

This is probably one of the easiest things to quantify. I just finished working on a project that used BDD and TDD. The project was a rewrite of an older system that required manual testing. It took about 6 months to get a change into production. After the new system went live, they found a simple way to improve it. The code change was very small, but because all of the TDD and BDD efforts could be leveraged as regression tests, the change was deployed in a matter of weeks. That single change was able to capture an additional $5 millions in value that would have been lost if things had of gone through the previous 6-month cycle for testing the entire application manually.

You don't even have to think that far ahead - consider some code you wrote in week 1, that was then modified by a colleague in week 2, now you return to the code in week 3 - what does the code do? Any mental model you had at week 1 is now irrelevant as the code has changed in unknown ways. What is the current intent of the code? Do you feel safe modifying it, despite the fact you no longer know how to manually test all the end points? You only have to miss one to have clients and managers breathing down your neck when the blame game is played as to who broke the app in production.

> Any mental model you had at week 1 is now irrelevant as the code has changed in unknown ways.

Isn't that what good git commit messages are for though? (Or any other source code versioning tool.)

I'm not saying tests aren't useful, but often times I find reading a file diff history is way more useful to understanding it than its tests.

IMO, tests should be there only to provide some level of feeling safe modifying files, as you can easily know if you messed up something.

It'll definitely show you the changes, but how far back do you start from? and how many files do you do this for? In addition, it doesn't show you how the dataflow changed; you need to model that in your head still.

Tests would also do this for you, but without the mental burden of brain compiling. It's nice to be able to set some breakpoints in code, then start a quick debugging session with the relevant test and see the data flow through the code - you can understand any function usually within 5 minutes.

What do I do? Review the changes in git. Am I the only one that doesn't have much faith in unit tests preventing a regression even on a well covered project?

Helping with regressions is a big benefit (and not just for years down the road). But that's more about unit testing in general than TDD. Really, we only disagree about which is bigger. I think for the first few years of a developer's career, learning about solid (pun intended) design is hugely valuable.

Many times that unfortunate person is you just a month or two later.

How do you test your test if you don't write the test first?

If you write the code, and then you write the test, then you haven't tested your test. You have no proof that your test will detect broken code. The only way to prove that your test can detect broken code is to run it against broken code. So you can write the code, then write the test, and then break the code and run the test. Or you could just write the test first, and run it. It should fail. Surprisingly, sometimes it doesn't: either because the test was bad, or the existing code didn't work like you thought, or sometimes the language doesn't behave like you think.

I've dealt with hundreds of thousands of lines of tests where I could go into the code and just start deleting functionality en masse, and no tests failed. If you don't test your tests, they're just bloat. They're the kind of pointless bloat people are complaining about in these unit testing threads.

Although on the other hand, writing your test first with no code and having it fail isn't really telling you much about the test. It just shows that the test handles the least interesting case (the case where there exists no code). You still don't know that your test is correct when you get it passing. You just know that it is correct when no code exists.

For example,

    TestSort(ISorter sorter)
    {
        var input = { 1, 2, 3, 4 };
        var output = sorter( input );
        Assert.IsSorted( output );
    }

This test will fail if you run it before the ISorter is implemented. But then it will pass if you implement an empty ISorter because the test still has problems (ie it starts with sorted input).

Cue the ever familiar: You still have to use your brain when using TDD.

If TDD needs brain usage to function, maybe the real thing that is useful is the brain usage and TDD is vestigial.

If someone gave me that test, I'd implement the code as "return input;"

Forget that you, a human, thinks you know what an ISorter should do. What do the tests demand that you do?

Doing TDD, the goal, when coding, is to write the minimal amount of broken code that makes the test pass. If you passed me {4,3,2,1}, then my code would be "return {1,2,3,4};". If you wrote a second test that passed me {6,5,4,3}, my code would be "return (input[0]==4)?{1,2,3,4}:{3,4,5,6};". You've got to write a test that makes me actually code what you want. I use this adversarial approach even when I'm writing both the test and the code.

Usually the first test I write if I'm starting with a blank slate, is to pass null and check that I get a NullPointerException. That gets the class created. After that, its got to be randomly generated data.

And that works fine ... as long as you know all of the cases that will cause the minimal amount of broken code to fail in a test case.

What happens if you're implementing something like a unification algorithm, but you don't know about the occurs check. You'll progressively add test cases that break your code and eventually stop. Until an end user creates a query that contains itself and your algorithm fails to terminate.

The solution to get TDD to work is to know what all of the weird edge cases of your algorithm are. However, I assert that the thing that makes TDD ever work is that the practitioners who are "doing it right" already know all of the weird edge cases of their algorithms. And I assert that if you know the weird edge cases you can drop the test first part of TDD and still get working algorithms.

Random data is interesting because it's not something I've seen anyone mention when they talk about how they do TDD before now. I suppose if you knew nothing about your problem and had some sort of tool progressively feed you intelligently generated random data, that might work for algorithms that have few edge cases or relatively simple edge cases. However, it seems like you would want something more like quick check for cases where you have really esoteric edge cases. And I don't see how you avoid implementing bubble sort (or its equivalent in your domain).

I don't think TDD can't work, but so far nobody has described TDD to me where I have any reason to assume that TDD is actually doing anything worthwhile. It's always sounded like TDD was irrelevant and the real key to success was thinking about the problem carefully. Additionally, it sounds like there are problem domains where anything that TDD might bring to the table would be nullified (ie certain problems that exhibit a certain level of complexity).

I can only suggest that you commit to it for some considerable time, such as six months. I, too, thought it was, at first, a fucking stupid idea, and then, when people I respected continued to advocate it, grudgingly tried it, and thought it was a pain in the ass - this was in 2006. However, the number of times where I've written some code that I thought worked, and then commented it out and rewrote it using TDD, and found bugs, is too high. Its rare, but its too high. So either I'm not that great a programmer, or TDD is useful. Another way to think of TDD is rubber ducking, perhaps.

So you're in a local maximum, and to get to the next, higher, maximum, you're going to have to go through a trough.

I'm not discounting or discrediting your experience. If it worked for you, then great. Unfortunately, my personal experience involves a few decades of "just try it my way and you'll see" ending in disappointment at best and catastrophe at worst. Maybe it's just the way I'm wired.

That being said, if anyone wants to fund me for six months (plus either some compensation to my employer for having me gone for six months OR a significant bonus for my own financial security) to try TDD, then I'll gladly do it.

That being said, I plan on putting it through one heck of a trail. At the end I'll either have a pretty good idea of exactly why TDD works OR I'll have a compelling argument for why it's gains are illusory.

When I write a test after altering/adding code, I will generally stash the altered application code, and run the test I wrote on unaltered app code to confirm that it fails.

As long as one maintains decent version control discipline, you can be pretty confident that the test is valid with this technique. (It does get a bit more difficult when there are schema changes, but then it also encourages good discipline and vigilance when you're making schema changes ;)

That's good. I suggest that you're doing unnecessary work (having to stash and unstash) when you had "broken" code at first, but at least you are testing your tests, so well done.

In my experience coaching teams on TDD, its quite easy, when writing code first, to add more code than is needed to solve the problem. And then only the problem is tested for. Meaning that there is code that is not tested. Maybe an if statement where a branch is never taken. Strict TDD means you don't write any code without a failing test, and you write the minimal code needed to make it pass.

Now quite often, a programmer has no idea how to tackle a problem, so I advise that they just write something that works. Just have a go and get something working. Most programmers (if they test at all) will then just go and write a test for it, usually for happy path. And that usually results in code with bugs. I know this because many times the programmer has done as I suggested and commented out their code, and reimplemented from scratch using TDD, and doing TDD finds a bug in the original that nobody saw.

Oh absolutely. I've done my fair share of "pure" TDD, but my style has evolved over the years to not always favor a pure approach (usually in the interest of time).

It really comes down to the nature of the problem - if I foresee a chance that the final architecture is not immediately obvious, I'll start breaking down the problem into small pieces and TDD the units, then the integrations.

I guess I'm of the opinion that not everything needs TDD, but everything does need diligence - my code after doing years of TDD has improved drastically, whether or not it was strictly test driven. Merely considering whether it will be easy to reliably test the code improves it, IME.

you could test your tests using mutation testing [0]. it's not bulletproof but quite close.

[0] https://en.wikipedia.org/wiki/Mutation_testing

For years I have written too many tests. Recently I have been trying something more like this:

- One test to prove I can cleanly load the component or object in a module. This is about loose coupling and ensuring that each module contains only one object.

- One test to prove the component or object does what I what it says it does. This is primarily about the Single Responsibility Principle. If I need six tests to prove an object works, then the object might be doing too much and might need to be refactored.

Other tests can be added if bugs appear, but just two tests per object can help us keep things light. Of course, if I have a module full of utility functions, then that is a different matter. But I think the limiting the amount of code in my tests and the overall number of tests is valuable.

"having test play some part in how the code takes shape"

That is exactly the point why I don't like TDD.

I mean, there are enough constraints that shape the code, why should something artificial like tests shape it too?

With mocking and everything you end up writing code for tests and not code for your problems.

If you use mocks extensively in your tests then you probably aren't getting much of the benefit of TDD. Mocks indicate high coupling so using them is just powering though testing your bad design rather then driving good design via testing.

How do you handle dependencies of the class under test then?

This is the biggest challenge. What I find works is separating your orchestrations from your data processing.

For example, instead of creating one class that does the network call, and processes the JSON response, break these out.

One class does the network call (keep it extremely thin). Dependency inject stuff in if you want. But view this purely as an orchestration.

if (connectionValid) json = makeNetworkCall anotherObject.processJson(json) <- test separately end

Then, in the other plain old data process class, do you heavy off-by-one edge case testing there.

If the first case, the orchestration, I can write tests like "One make the network call if the connection is valid". But that's all that test would do.

I am at the point now where I actually don't even test that. I assume that is going to work, because it is so simple. That and I generally hate mocking. Way too much coupling. Way too hard to refactor.

Where I do test heavy is on the plain old object side, that has no outside ports or connectors. These are just plain old objects.

I agree with some other comments I have ready here. Mocking is a bitch and I generally don't like it. But it's there if I needed it, so I do use it occasionally.

But it has so many downsides I try not to. Instead I separate the orchestrations from the processing. Unit test the processing (in an integration test sort of way), and try to stick to testing the public APIs of my class, so I am free to change the internals without things breaking.

My 2c. Hope that helps.

Would love an answer to this question, as well. I've only come up with DI as a long-term mostly win for this problem and it's something that I grapple with in every code base at work, since we have some microservices and other HTTP API layers that we need to interface with and I want to write testable code.

Well, you write code for design flexibility. Testing just forces the issue and helps you explore possible design needs. For example, what if the database query you're using isn't suitable any more? Instead of designing around a User table, you start passing around User records. Then your design is more future proof in case you start getting Users from a service or in-memory cache instead of a database.

> ...not code for your problems...

Right. It's not your problem now. It may be your (or your successors') problem in the future.

Is it always worth the effort? Nope. But experience, communication, and good teamwork will help with balancing short-term and long-term goals.

This is exactly the problem. In general you should not be writing code for design flexibility. Your code should contain the minimum number of abstractions to satisfy its requirements. If flexibility is a requirement right now then that's ok. Otherwise refactor in the flexibility later. But don't make the code flexible for the sake of tests. It vastly over-complicates it for no benefit. Instead write tests as close to the requirements level as possible (ideally most of your tests should be just below the ui)

> Otherwise refactor in the flexibility later.

I'm saying, with experience, teamwork, communication (including through tests), you'll know when this proposition makes sense or not. It's not universally true that "we'll worry about it later" make sense.

I don't consider taking steps to make sure the code actually as you expect it to as "artificial" feels pretty essential, to me.

If you're writing lots of mocks, odds are you're writing a lot of unit tests. In my experience these are rarely useful. Most often they are either really testing a compiler's output (indirectly), supplying an ad-hoc and incomplete type checking system to a dynamic language, or more appropriately covered by asserts that will scream when integration tests fail.

Because tests tend to make your code do what you originally intended, and keep your attention on the quality of the result.

Tests are not artificial constraints. They are the guarantees which (should) matter most to your users and stakeholders.

I am quite convinced stakeholders couldn't really care less about the guarantees being made by unit tests. Higher level integration tests, maybe.

I'm not hearing anything negative here. Why is any of this innately bad?

Tests shaping your code is bad, because? Because you already have "enough constraints"? Sorry, I don't get it.

> code which is hard to test is almost always going to be hard to maintain

This is a pointless equivalence to make. You're trying to suggest that code which is hard to test could be written differently, and better. This is not always true, and therefore useless to say.

STUXNET was very difficult to test. Reports are that it required an entire Israeli nuclear facility as a test environment. Would you suggest that STUXNET was written to a substandard level of professionalism? I doubt it.

You're missing the word "almost". It's important. Attacking parent comment with an absurd statement no one would agree with isn't a great argument.

He literally admits it's not always true in the sentence

Maybe it's only difficult to test end-to-end? I would assume there's code in there that's algorithm-like. Give it these inputs, it should return these outputs. If so, it should be possible to isolate that code and test it.

But, I dunno, I haven't looked at the source code. It might be very difficult to maintain.