> I start with the lowest-level, simple operations. I then build in layers on top of the prior layers.

I was discussing this with a coleague this week: starting by the lower level details vs starting from the more abstract/whole api. I prefer to start by writing a final API candidate and its integration tests and only write/derive specific lower level components/unit tests as they become required to advance on the integration test. My criticism on starting with the bottom-up is that you may end up leaking implementation details in your tests and API because you already defined how the low level components work. I have even seen cases where the developer end up making the public API more complex than necessary due to the nature of the low level components he has written. Food for thought!

This is the approach suggested in the book "Growing Object-Oriented Software Guided By Tests". You start with an end-to-end integration test and then write code to make it pass, testing each component with unit tests along the way. I find it useful myself, although I don't always adhere to its recommendation to start off with a true end-to-end test (sometimes I can identify two or more layers that can/should be worked on separately).

Your point of view resonates with me.

Over the years, I've learned to test "what a thing is supposed to do, not how it does it". Usually this would mean to write high level tests or at least to draft up what using the API might end up looking like (be it a REST API or just a library).

This approach comes with the benefit that you can focus on designing a nice to use and modular API without worrying on how to solve it from the start. And it tends to produce designs with less leaky abstractions.

Of course YMMV.

I just updated our “best practices” documentation to include the recommendation that tests be written against a public API/class methods and the various expected outcomes rather than testing each individual method.

I think the latter gives an inflated sense of coverage (“But we’re 95% covered!”) but makes the tests far more brittle. What if you update a method and the tests pass but now a chunk of the API that references that method is broken, but you only happened to run a test for the method you changed?

I like to think I’m taking a more holistic view but I could also be deluding myself. =)

Why doesn't your best practices document include a requirement to run the whole test suite?

Ah, I guess that wasn’t clear. Typically, yes, all tests would be run, though as someone else mentioned sometimes during development a smaller subset may be run for the sake of time (but all tests would still be run prior to QA and final release).

In some contexts there may be valuable tests that take long enough to run they should be run out-of-band. That said, I don't see where the parent says they don't run the whole test suite.

> now a chunk of the API that references that method is broken, but you only happened to run a test for the method you changed

Ah, yeah, not sure how I missed that. Maybe they meant the test for the broken method isn't run because it doesn't exist, but I very much agree that that's not the most natural interpretation.

Sorry, this wasn’t totally clear and I thought “run all tests” was implied. What I was trying to get at was the difference between “we have a suite that includes individual tests for every single separate method so we have great coverage” vs “we have a suite of tests that run against public APIs that still manage to touch the methods involved”. The former may test “everything” but not in the right way, if that makes sense.

I should have said “you’ve only written a test” rather than only having run a test.

I agree with this approach. It seems that one of the consequences of "Agile" is a general reluctance to invest a fair bit of time upfront thinking about interfaces. It's as if the interface design process somehow became associated with documentation, regardless if any non-code documentation is ever created.

I'm not convinced this is a consequence of agile in as much as it is more a consequence of bad pragmatism or expectation management. In terms of setting expectations, you are not doing anybody any favours by rejecting some up-front thinking in favour of rolling with the punches. And in the same way, you are not being pragmatic by cutting every corner you possibly can - pragmatism is just as often about deciding when to spend time up front to save time later on, and not just sacrificing everything from the short term (which I think is a common and valid criticism laid against poorly implemented agile workflows).

That said, I don't think TDD is a good way to figure out your API. It has its value for sure, but not if you take a purist or dogmatic attitude towards it. In any case, it seems to assume that more tests are always better (hence you start every new piece of code with a corresponding test), when I'd argue that you want a smaller suite of focussed and precise tests.

> That said, I don't think TDD is a good way to figure out your API.

There are plenty of ways to do it and one way that has worked well for me is in practicing TDD. So it is indeed one good way of doing it.

There are also people who are inexperienced with TDD or who misunderstand it and implement it poorly. There are people who are just not ready to design APIs yet who are forced to learn on the job. None of those things invalidate what I said.

If TDD doesn't work for you then you'll have to find what does and I hope that you test your software in a sufficient manner before deploying it.

> In any case, it seems to assume that more tests are always better

I think you're projecting your own opinions into the matter. There's nothing about TDD that prescribes what kind of tests one should write or how many. You could write acceptance tests and work from there if that suits you better. It's test driven development. The operative word is what we should focus on: testing is a form of specification and we should be clear about what we're building and verify that it works as intended.

It comes from an opposition to other forms of verification that used to be popular: black box/white box testing where the specifications were written long before the software development phase began and when the testing happened long after the software was developed.

That's where it's most useful: as a light-weight, programmer-oriented, executable specification language.

> There are plenty of ways to do it and one way that has worked well for me is in practicing TDD

Unless you're talking about the default state of failing tests when no implementation exists, I'm a bit confused as to how TDD helps with interface design.

Best of both worlds is to design top-down and build bottom-up. By design I don't mean something abstract, it should be translated into code sooner rather than later.

> Writing the test first has nothing to do with knowing the final API. When I write tests first I am looking for an API.

The greatest and most poorly communicated benefit of TDD right there.

No, it is an anti-pattern. TDD will get an API, and it won't be the worst possible API. However TDD does not do anything to ensure the API is good.

People who think TDD creates a good API probably have good instincts and a problem where an acceptable API is easy to create. When you have a complex problem that will have hundreds of users (thus it demands not just an okay API but the best) you need to think about the API first. Otherwise you can design yourself into a corner where to fix the API you have to change all your tests.

Once you have the API design in place TDD is a great tool for refining it. You need real working code using an API to sake out all the details and TDD is a great way to figure what you missed. You need a direction in mind first.

TDD does not do anything to ensure the API is good.

Well, what is good? You suggest such an API has to solve a complex problem with hundreds of users. I should think that I've written a few modules with just such an API using TDD methods. So what are we really saying here with this definition?

Your elaboration goes back to the myth I would like to see dispelled as my experience, and that of many others, has demonstrated that it's a bit of a red herring.

When I start a module I don't know what the API should be or what invariant is important or anything. All that test driven development asks is that I first make an assertion about my problem and try to prove that it's true (or not as the case may be).

This leads me to good API designs because the tests invariably contain the assertions about my invariant, they document the API at a high level, and they demonstrate that my implementation is correct as to those assertions I made. If I am thoughtful in this process I know that some assertions contain a quantification over a universe of inputs so I use property based tests to drive my specification. Other times a handful of examples is good enough to demonstrate correctness and so I simply test the units with them. And the API is entirely under my control this whole time. The tests are guiding me to a good API that does exactly what I say it will do by way of executable specifications that cannot be wrong about my implementation (or else the test would fail).

There's nothing in the above that says I cannot take a top-down approach and design my module's interfaces first. Nothing.

Personally I prefer the bottom-up approach for many of the reasons I explained in my OP. I like to define things in small units that I can compose over with abstractions that let me achieve more interesting results. Testing keeps me honest as to the true requirements so that I don't go off into the woods.

Good is in the eye of the beholder. And I was intentionally implying hundreds of users. If - this is the majority of APIs - there are only a couple users it isn't worth the effort to make the API good. When there are hundreds or thousands of users it becomes worth the time to start at a whiteboard and design the API, a few hours up front now can save minutes in the future and those minutes add up.

What you are missing is that TDD gets you to AN answer, but it gives you no information that you got the best answer. TDD does make some bad design choices hard or obvious; but there are other times where that there was a better choice isn't obvious.

Bottom up and top down are very different design considerations. You can get bad APIs in either one. You can do TDD with either.

I'm a fan of TDD and use it. However I'm not blind to the limits.

If what you're saying is that TDD can guide you to a good design but it doesn't guarantee it then I think we're in agreement there.

When working on protocol specifications and proving the correctness of a certain invariant or property of the system I find TDD to be lacking and tend to turn to formal methods for results.

However at the module/interface level of an individual software library or component TDD is a very good tool that helps guide me towards a good design. As do other tools like a sound type system. When they work in concert the system almost writes itself -- I just have to come up with the queries, invariants, and assertions.

There's methodologies in TDD that let you go explore to find the ideal API by working with it, then going back and removing that code (like you would a prototype) and TDDing it afterwards with the destination in mind. I personally think much more easily with this approach. I have to "play with it" and know where I'm going before I can test towards that goal. I think they call it spiking, but I might be misremembering the term.

Crucial part of rapid development is to get a prototype.

The problem is that in many cases management tries to push prototypes into production as quickly as possible and you might get stuck with the mistakes. Forever in the worst case.

Cheers for bringing me back memories of “Spike solution” from the early 2000s :)

http://wiki.c2.com/?SpikeSolution

>Writing the test first has nothing to do with knowing the final API. When I write tests first I am looking for an API. The tests help guide me towards a nice API.

Yeah, it means for people who don't have the time to rewrite the code and the tests multiple times.

> > Writing tests before the implementation code implies that you are certain about your final API, which may or may not be the case.

> How does this myth continue to persist?

Yes indeed. TDD helps you find a nice API.

TDD encourages you to implement an overly complex API with too many possible configurations. It encourages delivery of a box of parts, rather than sub-assemblies.

IMO TDD has almost exactly zero to do with the design of your API.

You can TDD top down by writing an end to end test, and recursing TDD down each level until you get to a single unit, repeat until all inception layers of the tests pass.

You can TDD bottom up, by starting with components you'll know you'll need, working out some APIs at their level, then doing the same to piece together larger components.

Ultimately you're in control of how your API looks, and this has nothing to do with TDD itself - if you get caught up exposing a bunch of inner workings and configurations, then you're simply not designing well. It has nothing to do with TDD.

Likewise, if you get caught short with an API that's too simple, well, that has little to do with TDD either.

To my respondents, who may have read more into my comment than I intended,

I merely wished to support agentultra's claim that

> When I write tests first I am looking for an API. The tests help guide me towards a nice API.

and to counter the article author's claim that

> Writing tests before the implementation code implies that you are certain about your final API, which may or may not be the case.

I did not mean my comment to be interpreted as "using TDD will guarantee you find an nice API". If you prefer, please interpret my comment as "TDD (under certain circumstances and when used in an appropriate way) can help you find a nice API".

I'm the exact opposite. I start with the highest level integration tests to test the API out. I iterate on this process quite a bit until the API is more or less settled down, and then I start implementing the API with appropriate unit tests.

As much as I'd love a very cleanly implemented API, in the end many more devs will be using the API, so the use of it has priority over an API that is easier to implement.

Your comment resonates a lot with me. The bottom-up approach is my favorite, but I think most people are more top-down. When a project has unit tests, its architecture is more pleasant for me. There is also less coupling between classes, this makes the code more reusable. TDD has clearly an influence on the style of the code. I think it forces the code to be better with better API.

I find TDD results in tighter coupling. The problem is we found dependency injection and a mock framework early and never looked back. As a result we inject everything everywhere, and all my classes know about all its collaborators and are tightly coupled. Sure we have a mock and so technically we are just coupled to the interface, but the result is the same: change one change the other.