Knuth's original statement on premature optimization presupposed that the cost of optimizing later isn't really that much higher than optimizing now. I mean, obviously it's going to be somewhat harder - code is harder to read than to write, and fixing a performance failure is going to take longer than getting it right the first time. But still, that's not a humongous cost and there are so very many opportunities for optimization that chasing them without performance data is a waste of time.
But when you have an actual barrier to optimizing? Then you've got a real problem. If you're committed to a technology that has a hard performance ceiling and no easy way to break the slow parts out into another technology? That's a different problem.
It seems evident that Knuth's advice about premature optimization can be indiscriminately applied when you're talking about code for an individual component... but should be considered carefully against the potential costs when thinking about large architecture.
Building software is hard, and it's not at all surprising (or reprehensible) that people will seek out shortcuts to winnow their decision space.
Most people I've known have used it as shorthand for "decisions based on optimization concerns require some justification that the optimization is important to the specific application and warrants the cost that come with it"; it's not an appeal to authority (indeed, the authority behind it is almost never referenced, just the bare quote), and its usually, IME, a defense against a preconceived, ideological decision (it works poorly to justify a preconceived, ideological decision, because it doesn't have carry much weight if the person proposing the optimization has any basis for making the claim that the proposed optimization isn't premature; as it shouldn't.)
People who 5 years ago were sermonising to me on premature optimisation are now telling me they're rewriting something in a faster language or how many millisecs they've saved using some new profiling snake oil they've just bought.
(whilst my background is in assembler/C, I now prefer to use prematurely optimised Ruby where possible just for the maintainability and the magic)