People who think TDD creates a good API probably have good instincts and a problem where an acceptable API is easy to create. When you have a complex problem that will have hundreds of users (thus it demands not just an okay API but the best) you need to think about the API first. Otherwise you can design yourself into a corner where to fix the API you have to change all your tests.
Once you have the API design in place TDD is a great tool for refining it. You need real working code using an API to sake out all the details and TDD is a great way to figure what you missed. You need a direction in mind first.
Well, what is good? You suggest such an API has to solve a complex problem with hundreds of users. I should think that I've written a few modules with just such an API using TDD methods. So what are we really saying here with this definition?
Your elaboration goes back to the myth I would like to see dispelled as my experience, and that of many others, has demonstrated that it's a bit of a red herring.
When I start a module I don't know what the API should be or what invariant is important or anything. All that test driven development asks is that I first make an assertion about my problem and try to prove that it's true (or not as the case may be).
This leads me to good API designs because the tests invariably contain the assertions about my invariant, they document the API at a high level, and they demonstrate that my implementation is correct as to those assertions I made. If I am thoughtful in this process I know that some assertions contain a quantification over a universe of inputs so I use property based tests to drive my specification. Other times a handful of examples is good enough to demonstrate correctness and so I simply test the units with them. And the API is entirely under my control this whole time. The tests are guiding me to a good API that does exactly what I say it will do by way of executable specifications that cannot be wrong about my implementation (or else the test would fail).
There's nothing in the above that says I cannot take a top-down approach and design my module's interfaces first. Nothing.
Personally I prefer the bottom-up approach for many of the reasons I explained in my OP. I like to define things in small units that I can compose over with abstractions that let me achieve more interesting results. Testing keeps me honest as to the true requirements so that I don't go off into the woods.
What you are missing is that TDD gets you to AN answer, but it gives you no information that you got the best answer. TDD does make some bad design choices hard or obvious; but there are other times where that there was a better choice isn't obvious.
Bottom up and top down are very different design considerations. You can get bad APIs in either one. You can do TDD with either.
I'm a fan of TDD and use it. However I'm not blind to the limits.
When working on protocol specifications and proving the correctness of a certain invariant or property of the system I find TDD to be lacking and tend to turn to formal methods for results.
However at the module/interface level of an individual software library or component TDD is a very good tool that helps guide me towards a good design. As do other tools like a sound type system. When they work in concert the system almost writes itself -- I just have to come up with the queries, invariants, and assertions.
The greatest and most poorly communicated benefit of TDD right there.