Yes, this is an obvious downside to the type of TDD that you sometimes see advocated, where they write tests for every little thing.

I try to find a compromise solution. I pretty much only write one unit test for a given module to start with, the nominal case. This is important, because it forces you to think a little bit about how you actually want your code to work, which I find to be a rather healthy thing to do.

Once that is done, a little later I will no doubt find a bug. When I'm trying to debug, and I start wondering whether a block of code is doing what I think it is doing, I write a test to verify. This does one of two things - it shows up the bug, which I can then correct, and verify that I have corrected, or it allows me to remove the tested code as a possible cause for the bug, allowing me to concentrate on other hypotheses for the cause of the bug, speeding up the debug process.

By the time I get to the end of the project, I generally have a pretty comprehensive test suite, without ever once feeling like it was a chore.