One basic problem is that science (and related disciplines) is really a slow incremental process where no one knows what is going on, and many people make small contributions, but this doesn't sell well. So every little thing ends up being promoted as if it is. It's the classic cargo cult problem: no one is actually a supergenius -- if there ever was such a thing really -- so we'll just act as everyone is because that's the corner we've painted ourselves into.

Another problem is that academics is full of dead ends, and it has to be if it's being done right. But no one likes that either. So rather than working on changing expectations, we do the opposite and bean count.

Yet another problem is the classic fan fiction / fan community issue: if you get into enough expertise, the experts have to become the audience, because no one else understands it, but then that leads to increasingly narrow fields of view because of increasingly narrow interests. You could broaden the audience, which is increasingly a demand, but that has downsides too, because often what's really the way forward is incomprehensible and boring to the average person. People love their iphones; not so much all the incremental computer science, physics, and engineering that went into every little part.

Something I think this essay maybe misses is that there are a lot of senior academics who would say publishing doesn't matter, that a paper is a dime a dozen, and what does matter are grants, and lots of them. This is how this dynamic has shifted in a lot of places, from the science -> publishing -> money. You might argue that this is better but it has its own set of incentive problems.