I agree. Unit tests are what make refactoring safe.

Adding known bugs to the unit test suite is a good way to prevent regressions.

"preventing regressions" is a subset of "finding bugs", and I've found the other methods I listed to be more effective (from an ROI perspective) at doing that too.

Apparently you work at Justin.TV. I don't know how big your individual projects are, but personally, I didn't get unit-testing religion until I started working on really big stuff, close to the limits of what I'm capable of. I worked for a year on a 60,000 line program, and it started getting painful. I worked for another year on a 100,000 line program, and that's when I finally had to relent and start writing unit tests.

Here's a common scenario that made me want to write tests. I want to implement feature A, but to make it work, I'm first going to have to write library B and C. So I write library B, but there is no way to test it in the program yet, because feature A doesn't exist. Right after I've finished the code, I have a pretty good idea of where things might go wrong, and what corner cases I want to exercise. But by the time I finish library C and then get around to implementing feature A, I've forgotten that stuff. Pain points go unaddressed.

In summary, unit tests really start to show their value when you're working on a program that's too big to fit into your head all at once.

Couldn't you have tried to break that 100,000 line program up into ten 10,000 line programs? You might have found the additional benefit that each of the ten programs could be useful on its own. That's how I would usually attack this kind of problem, rather than throwing unit tests at it.

I think it would be preferable to do both. Have a few really solid and well contained modules that are well tested.

Testing is only a piece.

It seems like you're still using some form of automation to find bugs, regardless of whether these would be classified as unit tests, correct?

Sure, some of the monitoring systems I've written do some fairly detailed testing. For example, I wrote the Justin.TV chat system. There's a group of bots that periodically log in to the production chat server, send messages to each other, and email me if anything doesn't work.

That's certainly not unit testing though.

I speak only for myself, but I would say the following: The real key is fully automated testing, no matter how it is done. The finer the grain, the better, generally. Matching an actual business case is near ideal.

Fully automated is necessary so you can run them frequently, automatically, near-continuously.

But beyond that, I think you get into religious territory, and I think people getting dogmatic about the definition of "unit test" tends to mistake definition for virtue (a common failing). If you've got automated testing, great! You win. Doesn't matter how it works.

Or, rather, it does matter, but only within your context, which nobody else is really competent to judge you on.

Yeah, the only thing that matters to me is whether the pro-unit testing position has become more religious or whether the anti-unit testing position has become more religious. I'll take whatever seems serious. I liked what Joel Spolsky had to say yesterday but I disagree with the linked blog post today.

do they ever say anything interesting to eachother?

heh, no they're awfully dull - they just say things like "testing 123" and "received 'testing 123' from bot456".

Also, TDD/BDD are most useful as design practices - not as bug-finding practices. They help you think more carefully about your design and make it more modular and clear.

I agree entirely. Whenever I leave out writing unit tests, I end up with a web of messy interdependencies which ends up far too hard to extend, and will always need refactored as soon as I need to add the smallest feature.

Writing unit tests up front forces me to write the code in a way where inputs & outputs can easily be stubbed out, which in turn makes writing messy interdependencies simply too hard!

Basically, if I write with unit tests, it becomes easier to write maintainable code, and harder to write unmaintainable code.

>> "If not for my unit tests, I'd be scared to change much of anything, for fear of stirring up a whole bunch of trouble that I'll have to fix later on."

For me, if I had unit tests everywhere, I'd be scared to change much of anything, knowing that after the refactoring, and usual testing/checking things work, I'd then have to rewrite/fix unit tests.

That's why I stopped for a while. Then I discovered BDD style testing (or context/specification, or...).

Whatever it's called, it's not quite as fine grained as a unit test and it's not as large in scope as integration testing. It's at the level of your business specs. So the only time you'll break your tests is if you've jacked up a spec.

Honestly, if my changes cause new behavior in the system but the people paying me won't care, then I don't care either.

Unit testing can be this way, but it doesn't have to be. At a really low level, unit testing can be "testing the implementation", which may be useful while you're writing code, but isn't useful during a refactoring.

But slightly higher level tests can be written to almost never have this problem. Let's say you've written code to ensure that a CC transaction doesn't go through with an invalid number. No matter what the implementation looks like - no matter how much you refactor - this test should always work. The same is true for a wide range of useful tests. (These aren't strictly "unit" tests, but they fall into a similar category.)

My personal experience is that a good unit test suite is 99% useful when refactoring, and 1% painful.