I didn't mean to say that Joe the webdev would be choosing the PRF, that's insane. But whoever providing the (library) implementation would have their life facilitated by having modular primitives, instead of having to code another construct from the ground up.
Edit: True, the AES competition was a much higher-profile event. There is an important difference (this is not a rebuttal, but a remark): AES is a cipher, PBKDF is a construction. AES has no proof of security, nor hope of one: it's a purely heuristic security argument. Through models like the random oracle, however, we could show that the PBKDF construction is secure, if H is secure. In such a case, there is not as much need for a competition, unless you're competing for performance or the like. That said, I would love proofs of security (or show the lack thereof) for PBKDF2.
Event hough hash functions are used in almost every system, we know far less about hash functions than we do about block ciphers. This is one of the failures of the cryptographic community. Compared to block ciphers, very little research has been done on hash functions, and there are not many practical proposals to choose from.†
It is also the reason why we are sponsoring contests to replace SHA2, because the research horizon for the current generation of cryptographic hash functions is... ominous.
You're making a noncontroversial statement (ciphers are better studied than hashes) sound like a controversy. It's not really a controversy. And please note: I didn't bring "conservatism" into this discussion; the blog post we're responding to. If you put it to me directly, I'll say bcrypt is more conservative than PBKDF2/SHA2 (which is what every current PBKDF2 system is going to end up using). But I didn't write a blog post that says "don't use PBKDF2".
† I say this only to make the point that it's not an argument pulled from thin air or from Moxie comments; something that Schneier commits to writing is, I mean to say, very likely to represent conventional wisdom.
I tried to be careful with the wording, precisely to avoid making a common-sense statement into a controversy. Yes, block ciphers are more well-studied (2012-70s > 2012-80s). My point was that the gap is much smaller now than (say) when AES was standardized (I wonder, was that passage also in the "Practical Cryptography" 2003 book?).
As a personal note, I doubt 5-10 years from now SHA-2 will have been compromised for password hashing (or HMAC), though. It has been remarked several times during the process that SHA-2 would have made a great SHA-3 candidate, its only major flaw being length extension attacks. The main fear was that SHA-2 would succumb to the same techniques as SHA-1 and MD5; so far, that has not been the case (perhaps because everyone is fighting it off with the SHA-3 candidates).
Of course, I didn't write a blog post either with anything. I don't blog. Given the choice, I'd give priority to PBKDF2; that's about it. Of course, I'm not a customer-facing developer, so my worries are different.
Trusting blockciphers over hashes based on this high-level argument is suspect, at least in principle. (To be clear, I'm not arguing with your practical recommendations as they seem pragmatically justified.) It could well be the case that the ways we build password hashes from blockciphers appear to be more secure simply because we haven't sufficiently cryptanalyzed blockciphers for the necessary collision-resistance-type properties. Perhaps collision resistance is just too much to hope from any highly-efficient function, and it is only a matter of investing the effort to find collisions in the functions derived from blockciphers. Adding to the suspicion is the fact that blockciphers give us good hashes by coincidence, not design (And formal evidence doesn't help explain the situation - although I see hand-wavy claims about related-key attack resistance being sufficient, in my work I've only found reverse connections).
So are you aware of any deeper reasoning behind the blockciphers-over-hashes argument? Could trusting blockciphers for collision-resistance just be a good usage of security-through-obscurity, because a tremendous amount of effort has been invested in finding collisions in the hash functions but not in the blockciphers?
The good news is that SHA-2 and HMAC remain in pretty decent shape (known limitations notwithstanding).
Also, for lay developers, choosing a crypto construct for its modularity is like choosing a smoke detector because it allows you to use different radiological bits in it. You're not supposed to be messing with those bits. The whole point of the package is not to have random developers changing them.
Also, I want you to note something:
You pick AES not because it's a standard but because it's the product of a contest in which many of the world's best cryptographers competed to design the replacement to DES. That's not what PKCS standards are. A PKCS standard is simply something that survived a standards group discussion.