Agreed. The problem is that RDD works extremely well if you want to get ahead.

you laugh, but a guy who knows Hadoop is worth more than a guy who doesn't all else being equal

One must disagree with this. It doesn't matter what the technology is, if one has the appropriate mindset and enough years under the belt, it will be picked up.

What matters in terms of better value is whether or not one can understand the problems space and find an appropriate solution for it. All sorts of solution space solutions can be used, some of which are better than others. The point is to try and keep it simple enough for the user, irrespective of how difficult it might be for the IT guy.

The knowledge of Hadoop in and of itself is not as valuable as one would suggest it is.

[Edit fix wording]

Unfortunately that's not how the job market works in most cases.

That's only true if the management level can't see that they are screwing themselves over, either willingly or ignorantly.

Hiring a hadoop guy for what used to work in excel is a bad decision in most ceases...