In my experience once everyone is happy with the way something works then it is sensible to move to lock-down — tests, documentation, performance testing, refactoring.
Doing any of those four things is a waste of time and energy until the design is decided upon. In agile development the design is not decided upon before the first code is written. It is always jointly discovered through iteration and discussion between designers, user-testers and developers.
If you write your documentation first then I suspect you can get away with writing all your tests first. But that's not a process I would call 'agile' and I doubt it ever results in good software.
The guy I was interviewing with didn't like the fact that I thought about how the problem might be shaped (and therefore made assumptions as to what I might want to implement and refine via tests) prior to “writing” that first failing test. It was absolute nonsense and I was rather shocked that the company managed to ship anything, to be honest.
I'm a big fan of tests; I'm a big fan of pairing. They are tools; nothing more.
I honestly don't work like that. So I'm kinda glad I failed that interview.
It always begins a discussion of the problem and what we want to solve, plus a bit of how—so we can figure out what it is that we're going to be testing for and how to build from that point.
Which is not to say that both TDD and iteration don't have their uses. But they work together poorly. And the problems best solved by TDD seem boring to some of us.
This is a giant strawman. Who is saying to write tests for your prototypes? You seem to be confusing a bunch of different methodologies here. You're talking about rapid prototyping to understand a domain before building a long term solution. That isn't TDD, and it would be foolish to write tests for an exploratory exercise that you're planning to throw away.
>TDD forces you to decide about one of the hardest parts of development (API design / where to place abstraction boundaries) up front before you have worked out how to solve the problem at all
How do you figure? If you've done a rapid prototype like you're advocating, you should have a better understanding of where boundaries lie and how the API should work. Further, you seem to be working under the impression having tests make it hard to change things. If anything, I've noticed the opposite - tests make it much easier to change code, especially when you test behavior, not implementation.