A good 80% of the people I see riding 3k+ road bikes (in an area that likes to bike, there are lots of them) are carrying far more extra pounds on their bodies than they could ever shave off the bike. It's probably safe to say that the vast majority of people riding bikes have far better bang-for-buck performance improvements available to them than a new bike or components, but don't choose them....
I'm not saying you have to spend a lot of money. I'm saying people mostly don't do the obvious performance changes anyway, and for most riders it doesn't really matter (as they aren't doing anything competitive enough to need it).
Of course it made me run headlong into the fact that the bike was geared to go up hills and I was hitting the top gear of it even on relatively easy rides.