I would add that as soon as anything in programming is well understood or becomes measureable, it is almost immediately automated as either a supporting tool or software library. The humans are necessarily just doing the stuff that hasn't been turned into code yet.
Not everything that counts can be counted and not everything that can be counted counts. - Albert Einstein
I don't think programming is necessarily unique in this regard. As with any other profession, there are things that can be measured, and things that you just have to trust your gut on.
This does not sound like something Einstein would have said. For one thing, it's a play on words in English. Let's see what the google has to say...
Edit: yep. It comes from Informal Sociology (1963) by William Bruce Cameron. Never heard of him? Me either.
By the way, this the first time I've seen quoteinvestigator.com come up in the search results. That's welcome; it's a great site. If you like this kind of thing, check it out. http://quoteinvestigator.com/2010/05/26/everything-counts-ei...
p.s. fully agree about the barriers to empirical study of programming.
What is holding programming back from the degree of empiricism found in a field like mechanical engineering? Is it that it's hard to fund or even structure meaningful experiments? Should someone be developing non-ferromagnetic monitor/keyboard sets so that we can study programmers of different levels of demonstrated ability with their heads stuck inside MRI machines? (Such research has been done with piano virtuosos playing on electronic keyboards and rappers.)
http://www.ted.com/talks/charles_limb_your_brain_on_improv.h...
Another thing I'd note, is that a lot of debate about programming is word salad full of reasons why you can't say X or Y. There seems to be a lot of the Epistemic Viciousness that exists in martial arts schools.
http://lesswrong.com/lw/2i/epistemic_viciousness/