* What's my confidence that the code will work correctly under different situations.
* How maintainable the code is and related how easy is it to make changes.
* How much time/effort does it take to create.
It's funny (or sad) how the XP guys like Kent are always talking about doing the right thing and not being religious. This stops working when the people using it don't know what the right thing is any more. They write terrible tests, they make bad changes to their code to support those tests, it ends up a big un-refactor-able ball of mud that takes longer to create and is more difficult to maintain.
If you're writing a shortest path algorithm or a JPEG decoder then you absolutely should code up some tests to check your code. The tests are relatively simple. They don't need to know anything about the internals of your code. They help you check quickly whether your code is correct and don't interfere with any refactoring. You can rewrite your JPEG decoder from scratch without needing to touch the tests at all.
in tdd environments the test harness is used as some kind of crutch to avoid having to have difficult discussions about what the product should* be doing - what makes sense. the test acts as some kind of oracle in the regard, even though it was written by an intern last spring who didn't really have any idea what was supposed to happen either.
i'm basically afraid to say 'the test is wrong' anymore. i always get back a blank look of incomprehension or even outright fear.
I like to think of production code as being inherently a liability, not an asset, and the same goes for test code. That's not to say that code, or tests, cannot be valuable; it's just that every line of code carries a cost to the business just by adding complexity to the development process and by taking up disk space and CPU cycles. Not all code offsets this cost, and a lot of tests certainly don't.
We should stop viewing tests like holy scriptures. Some add long term value, some add value during initial development, some should never have been written in the first place.
And on a side note, our tests should test the business requirements, i.e. they should be tracable to specs. There are a lot of tests out there which amount to little more than to "test" some arbitrary decisions which we made when someone initially fleshed out a class diagram.
All successful projects in my career had full functional regression test coverage.
For those successful projects, I made following conclusions:
* My personal TDD process are:
* Script enable - always run every night, every weekends.
Output to html files to see overall green/red in < 1 second.
* When any existing test broke, fix/debug the tests are the highest priority before any more features development.
* The test reports from all the nightly and weekend test should always be green.
* Any red(s) will stop all features dev.
* There are 5 minutes function test scripts that require to run before every commit.
* Normally, I prefer commit in the morning after overnight test runs were all green.
* These were small project with team only 1-2 devs.
* Much easier to enforce the process. (Just need to convince I, me and myself.)
* I worked on big team projects with 100+ devs.
* I can test only my own part of code. For any big team, I can take the horses to the water but can't make anyone to drink it. The overall system SW quality was still bad.
* I Only do full functional tests - I don't do any small unit tests with mocks.
* For CDRW drive, there were 100+ test cases for
close, read, open, repeat,
close, read, write, loop, open, repeat,
for different CD format, disk, drive, etc
* For DNA sequencer machines, there were tests for
Motor controls, 3 axis robot,
CCD camera, laser control, script subsystem, comm sub sys, etc.
(This is old project, 20+ yrs ago)
Deployed in the field for 15+ years with version 1.0 Firmware.
Good for company bottom line - not sure for the SW guy's career. :-)
(Multi billions $$$ of revenue for the company, $500+M first year)
* For video decoder systems with 19FPGA + 10G network: (10+ years ago)
Streaming control, snmp, IGMP, all each has their own test scripts and
combine system level scripts for complex network, HA setup.
Deployed in Comcast for long time without any issue.
Only one issue reported was system has problem after 255 days of
non-stop running.
I did have test case for 10 ms counters (for SW bug happened for 25, 49 days)
I didn't create test case 100ms counters (for SW bug happened after 255 days).
Again, very good for company bottom line.
(Tens millions $$ of revenue for the startup)
But the company don't need me anymore after they shipped that product.a) you are disciplined in your commitment to quality
b) you offload as much of the tedious, repetitive work to the computer as possible.
Regarding unit tests and mocking, I have a similar opinion. I tend to test at the larger component/package level rather than the smaller class/module level. Furthermore, I avoid mocking as much as possible. Object-Oriented decoupling through dependency injection makes mocking a lot easier (and removes the need for mocking frameworks) and Functional decoupling[0] makes mocking largely unnecessary.
In addition to mocking frameworks, I've also encountered home-grown solutions that were complete time-sinks to use. At some point, you're investing huge chunks of time, not just testing, but mocking in particular.
It can literally layer an entirely different problem domain onto your dev effort, apart from the actual "business problem" the code is intended to solve.
At a certain point, it's like, "wait, what are we doing here again? There has to be another way." Always wondered how people became so dogmatic and determined that they fail to see that they are veering from their original purpose. There is a balance to everything and something about the dev community causes us to lurch in different directions, well beyond that balance.
BTW, there is a post [0] that reminds me of this on the blog you shared. Good blog, BTW, even if it is no longer active.
I'm confused by this. In C++, I found that one of the only ways to effectively use mocking was via dependency injection. For us, dependency injection was introduced to enable TDD. That it had other good qualities was a bonus.
Another approach is to model business logic using pure functions. Instead of business logic calling the integration logic (directly or indirectly) you have the integration logic calling the business logic.
The consequence of this is that when you are unit testing your business logic you do not need to mock anything. It's simply a matter of checking input/output mappings.
See this series of articles for a deeper explanation: http://blog.ploeh.dk/2017/01/27/from-dependency-injection-to...
Yes, this is similar to what I did. I ensured business logic dealt only with data structures and algorithms, and knew nothing about the rest of the world. If it needed to report an error, it would call an interface class's "ReportError" function, and pass a string. It did not need to know if the error went to the console, a file, an email, text message, etc.
However, in our case, the communication is still two way - just separated via interfaces. The integration logic does call the business logic, but the business logic may need to do things like report the results, log interesting information, etc. So we still had to mock the interface class (i.e. mock the ReportError function).
But in the bigger picture - yes. Not allowing your business logic to know anything about the details of the integration was really helpful. In principle, our customers could have written the business logic for us.
Sadly, my team did not accept it so we never adopted it.
I think he's doing it right in spite of what other religious zealots may say. In my experience, unit test driven development causes three massive problems:
* Unit tests are intrinsically tight coupling.
* Unit tests require you to build a model for most things that you are testing against (e.g. your database). Building these elaborate models (i.e. using mocks) is prohibitively expensive when you have tightly coupled code but they're often still expensive to build (in terms of developer time) even when you have loosely coupled code.
* Unit tests are intrinsically unrealistic because they are built against models rather than the real thing.
In my experience unit tests work adequately for domains that are heavy on complex logic and light on integration (e.g. writing a parser). In domains which are light on complex logic and heavy on integration they fail badly (e.g. most business apps).
Optimizing for the overall speed of your test suite by using a more unrealistic test that catches fewer bugs is a massive false economy. CPU time is dirt cheap. QA and dev time is very expensive.
There's a testing methodology we used at my last company called "Change Risk Analysis" or something. Essentially, the point was that you separate business logic and integration code, and you write unit tests for the business logic. There was a metric that combined the the cyclomatic complexity with the test coverage to produce a "risk" score for methods and classes, and all code (or at least all new code) had to be under a certain risk threshold.
I would also add that domains that are heavy on complex logic and light on integration are also highly amenable to more sophisticated verification methods, such as generative testing and model checking.
> All successful projects in my career had full functional regression test coverage.
Regression testing is not the definition of TDD, however.
TDD is this crazy religion of writing tests for everything, including the lowest-level helper functions in a module that don't correspond to anything that would be tested in a regression test suite.
You write these little tests first (which must be shown to fail at least once and then shown to pass). Then you keep them around when they pass and keep running them all the time even though nothing has changed.
For instance, if you develop a Lisp, you must forever have a test which checks that, yup, CAR on a CONS cell retrieves the CAR field, CAR on NIL returns NIL, and throws on everything else. The image is 10X larger than necessary and it takes 27 minutes to get a REPL prompt on a fresh restart. :)
By the name, nothing says you need 100% coverage (a stupid measure abyway, cause you can get those 100% but still neglecting thousands of corner and limit cases for onput data, so actually you need another measure) or that it has to be unit tests...
Not true in my experience. You should only test code you wrote that can break. You do not test core functionality, the OS underneath etc. You do not test the test themselves.
A common source of errors are mismatches between what your dependencies say they'll do and what they'll actually do. There's been huge, costly failures from developers expecting the dependency would honor its documentation or just take a standard approach to error handling. So, either testing them or at least good catch-all error handling + logging should be done.
Anything else is not TDD.
Robert Martin is likely the biggest evangelist for TDD, and he does not advocate what you say.
Why do people keep claiming you have to test every line of code in TDD?
No, it doesn't. TDD means tests are written to test the code, and the code being written must pass all tests. It's ok if at any point some parts of the code aren't covered by tests, because TDD is a cyclic process where at each cycle new tests are written to accompany changes in the code.
It seems that the only people complaining about TDD are those who pull the strawman card and portray the process as being unreasonably convoluted and picky, to a level which is almost autistic.
And like all crazy religions, the followers ignore the prophets.
Robert Martin is likely the biggest evangelist for TDD, and he does not advocate what you say.
Writing tests after the code has been written - to lock down behavior, quickly identify breakage, and support refactoring is a valuable process. But it's not the original vision of what constitutes test-driven development.
This is the single biggest problem everywhere I have looked. Outside the development team itself no one is willing to wait for a new feature just because a test failed. And for many large programs there will be substantial subsets of users who will not be affected by the bug so they will argue that the delivering the feature is more important, and contributes more to the bottom line, than fixing the bug.
For instance, when he was working on an unstructured log parser, there was no way TDD would work because the logging stream didn't have a pre-defined structure. Instead, he developed a process where he could quickly inject his parser into the logging stream, visually inspect the results, tweak, and then test again.
Others (Such as Martin Fowler and Uncle Bob) have also emphasized the value of automated unit testing integrated with a CI pipeline as a way of ensuring that you didn't break existing functionality.
As always, what's important is not the specific practice, but rather the engineering principles behind it. The key questions being:
* Am I on the right track?
* Did I break existing functionality?
* How quickly and frequently can I get answers to the above two questions?