Personally, I've given up on categorization entirely. It's just "automated tests". People seem to be very prone in getting caught up in the question of whether it's a "unit" or "acceptance" or "integration" test when that's a completely uninteresting question.
Even "Test what, not how" is only a rule of thumb; if I've got a complicated "what", I'll sometimes at least probe the implementation to be sure I'm on the right track during development. Of course, if I change the "what" I just nuke them. I say this having done this entire cycle over the last week. My "what" was too slow and I need a new "what", but unit testing the "what" helped me get it to the point where I could benchmark it much faster than if I had tried to "not test the what". (Sometimes, you can also nuke the "what" tests before putting them into long-term storage.)
If you can avoid getting caught up in attaching goodness and badness to the terms, unit/acceptance/integration can still be useful communication terms, but I resist using them for anything else, and I don't personally use them at all.
(Also, to be clear, I consider this amplification or reflection upon your post, not disagreement.)
"My "what" was too slow and I need a new "what", but unit testing the "what" helped..."
If only there were higher levels of abstractions to say what you mean.
My explicit point in this case was about how the labels are actually bad, get in your way, and shouldn't be used because they hurt more than they help.
I have also had times when I moved away from unit testing in favor of higher-level tests, but ultimately found that the quality of the code got worse and the build time became unacceptable. I'm not trying to generalize here, I'm only talking about my code on a particular project at a particular point in time.
I've had good luck attacking this problem directly. If your integration tests are sufficiently integration-y, often this slowness directly reflects the fact that your system is slow to the users as well!
Or, failing that, that you've got too many dependencies hardwired into your system, or some other real problem.
Following up on my other proximal post here, the question is ultimate not whether it's high or low level, it's whether it's fast or slow, period. I've written slow low-level tests because they fully tested a 10x10x10x10x10x10 state space accidentally, I've written (after some work on the underlying system) some very fast tests that literally simulate user input to the system via the web server at the highest possible level.
The real key is that you can't really follow a testing recipe, you always have to be looking out for the cost-benefits. I suspect the other key is that you must come to internalize the idea that automated testing is really important and if that means spending a man-week to improve your testing environment is what you need to do, then do it. It's really OK.
If you know ahead of time exactly what the inputs and outputs should be then sure, writing unit tests first is a good practice.
The example he works with is writing a driver for Tokyo Tyrant in java. He begins by writing a high level test of what he eventually wants to be able to do. Instead of implementing it though he comments it out and then starts to play around with potential implementation code inside the body of the test. Once he's finished playing he then gradually refactors code from the test body into the implementation until he's got everything he needs and uncomments the test.
It may well be the case that by the time you've finished exploring that you've realised the test you've written isn't very relevant. This is fine, the purpose of the writing the initial test is to guide your exploration in a productive direction. Once you have some implementation code you can always go back and reframe the test according to your new understanding.
All my actual programming work here consists of adding printfs at strategic points to tease out exactly where I should be looking for the point data. Unit tests are completely useless for this process.
Come to think of it, even once I've got it figured out, it's very hard to see how to usefully use unit tests here...
http://paulgraham.com/arc0.html
I don't use Arc (or any kind of Lisp) but I understand what Paul Graham was trying to achieve.
The first problem is data consistency. Both require consistent data in whichever environment they are running. Not an insurmountable goal, but one that often leads to a lot of false failures. These lead to a lack of trust in the test feedback.
The second problem, as others have already pointed out, is that integration tests and functional tests don't tell you what is failing or why. Can I create a user? Nope. Why? 500 error. Time to fire up the debugger...
The third problem is that functional tests do not scale. Each functional test will take orders of magnitude longer to run than integration tests, which take orders of magnitude longer than unit tests. You very quickly end up in the situation where you have more hardware than people or you no longer get fast feedback.
Personally, feedback is the primary driver behind test automation. It is nice to know what I wrote will probably work and it certainly helps simplify the code. However, the biggest pay off comes from the fast and repeatable verification that what I changed has not impacted the code base negatively.
Most of the time, however, we are creating web apps and things that run at a much higher level of abstraction than the underlying algorithms. We should probably write tests at the same level of abstraction our solution is.
Although, I guess, when you're writing medical software or something for a bank, you should be a bit more strict than when you are making the next best social sharing app.
A guy I know once solved a bug by having a unit test for the md5 function from the standard library of the language at hand ... turns out a specific version was buggy.
Either way, I prefer using integration/functional test. What I care about is that the right output flies in for a certain input. What I don't care about is how. So I don't test for that.
Also, Unit tests must be created before the code they test.
yes they should. It's very helpful to think about how your code will be used before you write it, helps you think it through as well.