I don't know, do I? How am I supposed to know? How am I supposed to explain to my mom when to click yes and when not to? The average person shouldn't ever have to think about this.
Imagine if any website could ask to access any other website, for an innocent reason, and then scrape whatever account information they wanted? "Do you want to let this website access google.com?" Great, now your whole digital life belongs to that page. It's a privacy nightmare.
> it destroyed to ability to build so many things on the internet
It only destroyed the ability for any website to access another website as the current user. What it destroyed is the ability for a web page to impersonate users.
Also, one thing I can speculate that phishing would become even easier if such things were allowed
Fundamentally this all boils down to you, the person building the site, being cheap. You don't want to pay the handful of dollars to make your own HTTP requests.
I think it's much more about open web and letting the user decide than about being cheap.
You'd be constantly flooded with permission popups, and attackers would just host part of their code on one ot more of the popular domains that everyone had gotten used to clicking "allow" for.
The whole CORS thing is so off and it destroyed to ability to build so many things on the internet. I often think it protects websites more than it protects users. We could have at least allowed making cookie-less requests.