This is because of the toxic effect of economics and marketing. Everything has to be centered on making the platform more money. Even in cases where it makes no sense to do so!

I've seen open source projects embrace market driven design methodologies, because they mistakenly think it matters.

There are so many designers and developers who have only worked at startups, and dark patterns have become baked into our development culture.

Mostly this attitude that you need to manipulate the experience of the user, and disregard any feedback from them, because you are the expert, and they don't know what they really want. That's what Steve Jobs said, and he was a success, right? So you should do that also.