I talk a lot about web application security. The following line from flesh and blood developers on multiple occasions:
"Shouldn't your tests catch security vulnerabilities?"
To which I always respond:
"Yeah sure, they'll catch the vulnerabilities that you already thought of"
I've personally seen codebases with test suites that cost (even with the most conservative estimates of time spent and day rate) upwards of 300,000 USD with 100% coverage that allowed an anonymous client to drop the users table.
If a lack of security bugs (and therefore bugs in general) is any indicator of software quality, TDD has yet to impress me in the wild.
My development process has always been to prototype out three or four implementations, evaluate them against the known use cases and then settle on a final API design. Then I'm ready to do TDD. Glad to know I'm not the only one who works roughly this way.
It also bugs me a little that many otherwise intelligent developers use TDD as their sole software design tool much of the time. The exact term they use for this is using tests to "drive out" a design. While I understand this may work in textbook examples, in real life I've seen the concept being used to justify designs that clearly aren't fit for purpose.
100% code coverage is not 100% coverage of every different possible code path. In fact, for most software systems it is not even remotely feasible to achieve that level of code path coverage. The "100%" figure can easily lull less experienced devs into thinking they have sufficient test coverage when nothing could be further from the truth.
That's part of the idea of the "unit" in "unit test." You decompose the program into units for which correctness is relatively simple – and testable.
How to do this properly is not obvious, and for teams on which TDD is imposed without this understanding, it will all become a bizarre and useless ideology.
When it is understood, the resulting program will have properties that resemble those of purely functional programs, especially composability and correctness.
It is not that hard to write pure functional code that is not very compassable or correct. Keeping it simple stupid is more important.
After I've got a strong enough grasp as to the overall requirements, then I throw away this spiked code to start fresh, working along similar lines to 'programming by wishful thinking' whilst keeping the original spiked out structure in mind.
By following a strong TDD philosophy, it's easier for me to throw away my initial 'spiked out' code base for a couple of reasons:
1. Personal experience of how much better a test-driven code base can be than ones I've later tried to wrap tests around/dismantled in the process. 2. Having the understanding from the get-go that this code is not 'production-ready', and should not be considered reliable, but as a very fast learning experience. 3. Knowing that I've spent a small time prototyping as preparation, and having this laid out as an overall part of the process means I'm not considering this time 'wasted'.
For example, I'm building a statistical model. I need to load a bunch of data from files, transform it in straightforward ways, then run my model against it. TDD doesn't help me come up with the right model, but it does help me not screw up while loading the files. When my model fails to be predictive, I know the model (rather than the loader) is at fault.
So... if you are trying to take 14 different approaches... TDD isn't there to solve it for you. That's what I said.
But then that's planning. And when you've found your approach, you write it with TDD.
So, those Sudoku testing article fails because the author is breaking rules of TDD. I mean, the first test: test_cells, does so much more than it needs to do to pass the test. He's designing first, and then testing. That's not how TDD works.
If you can't see why that first test is a bad test, you aren't qualified to judge the merits of TDD.
> Of course somebody is going to chip in here and say … you should be testing at the point where you can create a stable API that remains unchanged
That's not true. At least, not the way you phrase it. You are, of course, forgetting that TDD involves 3 steps. Writing a test, Passing that test, and then refactoring. That refactoring includes refactoring the test. API's can, and should change to evolve. That's part of refactoring.
You pretty much repeat the same arguments against TDD that I see time and time again. Arguments born out of misunderstandings. TDD isn't easy. It's challenging, it requires practice and effort. It does take time.
> TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all.
Yes, but nothing about TDD prevents you from easily making changes as needed. Again, Refactoring is apart of TDD. You say something like this multiple times, making it seem like refactoring or changing code is some painful process.
It's not. It's trivial. If it's not, your process has a serious problem.
if you are looking at Ron Jeffries Doyen of the Agile set, and saying "well of course he did not get the results, he was doing it wrong" the we are well into id-protecting, religious territory.
if one of the leading exponents gets the process wrong there is something wrong with applying the process to the real world
TDD really seems to me to be two parts:
- Design up front, disguised as writing an API
- Ensuring you write repeatable tests before commuting
I'm more interested in extracting what works from a concept than slavishly following a cult leader. IME, being a "leading exponent [sic]" of anything doesn't make you infallible, even in that domain.
> Design up front, disguised as writing an API
The TDD literature I am familiar with doesn't advocate writing an API up front, it recommends writing essentially a minimal fact statement up front about the API, writing tests to verify that fact statement, writing code to pass that test if it doesn't already pass with the existing code base, and iterating like that. You could design more than the single fact that you make the test and code for at one time, but that's not a demand of TDD or even relevant to it, and is not, in most cases, consistent with the kind of decide-as-late-as-possible rule that the Agile and Lean approaches in which TDD is usually embedded prefer. I suppose it would make some sense if you were doing waterfall-adapted-to-incorporate-TDD, but why on earth would you do that?
Wouldn't worry about it.
I found it extremely dubious. He was doing some kind of accounting/banking app, and came up with a new approach. Well, he spent his career on this domain. I find it far less likely that TDD caused the new idea than it was just that he'd hoed this row so many times before. If you have done it 10 times before you can do bottom up, top down, inside out, outside in, or completely random, and get a reasonably structured, working code base. I conclude nothing by him creating accounting app #11 with a scheme that is perhaps well suited to how his brain works, but is perhaps terrible for mine.
I'm not against testing, quite the contrary, but I find some of the claims rather exuberant.
No, you are misunderstanding the parent. He meant some people assume you write _all_ tests first. If you read his entire comment, you'll see how he describes exactly the process described by Kent Beck. A small test, a small implementation, followed by refactoring both test and implementation (the 3rd part being the part commonly forgotten).
Bingo! Exactly that. Right on the money, especially the part about refactoring both tests and implementation.
> You write a tiny bit of test, then a tiny bit of implementation to satisfy the test. If you notice something you'd missed previously, you refactor BOTH tests and implementation.
That's TDD.
He was alluding to the fact that people might assume that they have to write all the tests first.
Are you really going to sit their and say that he couldn't have been wrong? That he's perfect? That he can't have made a mistake.
The first test in that blog post was a bad test to start with. It's as simple as that. I'm sorry, but appealing to authority isn't going to work here.
He was wrong.
Agile has the concept of Spikes (http://www.scaledagileframework.com/spikes/) for researching new problems. The key point that code is thrown away after completion -- this gives you the freedom to ignore normal methodologies and not have to worry about technical debt.
Presumably, you then go back with this new knowledge and only now concern yourself with writing supportable code.
I hate writing tests that require me to do any kind of boilerplate code. Tests should (theoretically) only set up and exercise whatever is needed for that test. I think if you are able to do that (I've had success with AutoFixture for .NET as an example) then you'll feel a lot less bad about throwing away tests and they will also be much much faster to write. When writing tests is tedious or brittle, then I feel they are a chore and am less likely to write them to begin with.
Of course, as you delve deeper into the problem, you often learn something that changes your understanding of the problem. That's where the third step of TDD comes in - refactor. TDD doesn't make people reluctant to refactor their code; it forces them to do it. Refactoring is a vital part of TDD.
It is sad that in this day and age waterfall is still taught as a serious methodology.
http://pragprog.com/screencasts/v-kbtdd/test-driven-developm...
This is a giant strawman. Who is saying to write tests for your prototypes? You seem to be confusing a bunch of different methodologies here. You're talking about rapid prototyping to understand a domain before building a long term solution. That isn't TDD, and it would be foolish to write tests for an exploratory exercise that you're planning to throw away.
>TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all
How do you figure? If you've done a rapid prototype like you're advocating, you should have a better understanding of where boundaries lie and how the API should work. Further, you seem to be working under the impression having tests make it hard to change things. If anything, I've noticed the opposite - tests make it much easier to change code, especially when you test behavior, not implementation.
In my experience once everyone is happy with the way something works then it is sensible to move to lock-down — tests, documentation, performance testing, refactoring.
Doing any of those four things is a waste of time and energy until the design is decided upon. In agile development the design is not decided upon before the first code is written. It is always jointly discovered through iteration and discussion between designers, user-testers and developers.
If you write your documentation first then I suspect you can get away with writing all your tests first. But that's not a process I would call 'agile' and I doubt it ever results in good software.
The guy I was interviewing with didn't like the fact that I thought about how the problem might be shaped (and therefore made assumptions as to what I might want to implement and refine via tests) prior to “writing” that first failing test. It was absolute nonsense and I was rather shocked that the company managed to ship anything, to be honest.
I'm a big fan of tests; I'm a big fan of pairing. They are tools; nothing more.
I honestly don't work like that. So I'm kinda glad I failed that interview.
It always begins a discussion of the problem and what we want to solve, plus a bit of how—so we can figure out what it is that we're going to be testing for and how to build from that point.
Which is not to say that both TDD and iteration don't have their uses. But they work together poorly. And the problems best solved by TDD seem boring to some of us.
In fact it only gets in your way. Solving problems that are new (to you) involves writing code to find out if a certain approach works, then throwing it away and trying something else. TDD makes that process a lot slower, and makes people more reluctant to throw away incorrect models of the problem that they have codified into tests without even realising it.
Of course somebody is going to chip in here and say that if the above is happening then you are writing your tests at the wrong level of abstraction and that you should be testing at the point where you can create a stable API that remains unchanged while you throw away the speculative implementations underneath. But if you are tacking a new problem you have no idea where those boundaries lie. It's hard enough to determine that for problems that you already understand and have working solutions for, never mind ones you don't. TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all. Either that or you just write vast vast numbers of trivial tests for every single function that you write (which does seem to be what some TDD tutorials advocate), but then you are creating a huge maintenance burden for the future where any change to the code is going to necessitate rewriting dozens of tests.
I would argue that TDD only works if you are solving problems that you are very familiar with. By the time you are writing your 5th CRUD web app you probably have decent mental model of the parts you need before you start. I feel that a lot of the love for TDD comes from people who spend most of their time working on problems they already solved dozens of times before, but they learned how to solve them before they discovered TDD.
For people first learning how to code it is absolute death. It discourages people from doing the most important thing they need to learn: throwing code away.