If people raise objections, that suggests that they are having issues with understanding. I know I've been confused by it before, and I usually feel compelled to use modifiers to distinguish between 'literally millions' and 'actually, no-exaggeration, literally millions' when I want to be understood.

That doesn't necessarily mean we're moving in the wrong direction. Language evolves to satisfy speakers' criteria, so it will gradually improve median utility. But any change is going to have cost, and confusion could be the price we pay for progress.

Alternately, you can imagine a situation where 1% of the population derives a huge amount of utility from a word, while 99% only get a small benefit from another definition. Improvements to the median in that scenario can have negative consequences in aggregate. Or you can see that this particular usage has been confusing people for a hundred years, and the cost of change may outweigh any potential benefit.

I think the position that natural language will just take care of itself is just a touch hand-wavey. Human interests and linguistic-evolutionary movement are often coaligned, but sometimes diverge- the same as with human interests and economic growth.