Its only been a viable option to self publish for about 30 years, and more like 10 if you didn't want to invest large amounts of personal resources.

Open access review and channelization is still an open problem. The for profit journals actually do provide a service of worth and note.

There's also sort of an issue from a bunch of open access publishers acting like con-artists - charging researchers made up fees to publish in their made up journals. A certain librarian made a rather useful list of "predatory" open access publishers: https://beallslist.weebly.com/

Well for one, open access requires the Internet. The closest thing to "open access" prior to the Internet was the willingness of universities to allow unaffiliated members of the public to walk into libraries and read the journals stored there. It took the Internet for there to even be something to figure out.

Beyond that, it is just the general time needed for society to adjust to a new way of thinking; people, scientists included, do not start questioning the status quo overnight.

> Perhaps a blunt question, but what took scientists so long to figure this out?

The other science disciplines (mostly the life sciences), saw the AI/ML field greatly advance with their open research (e.g. arxiv, Github) and now want to copy it.

And yes, it was specifically AI/ML. Look at IEEE -- they're holding back the entire EE field. The ACM holds back the other areas of CS.

CS may be the faster growing field in the arXiv over the last decade but it was relatively late to the party. https://arxiv.org/help/stats/2017_by_area/index

From an outside perspective it appeared to me that the leaders in this were actually the physics and mathematics community, rather than ML.

Physics and other disciplines were broadly using arxiv before ML did it. When I switched from physics to ML (CS) I was surprised that arxiv isn't used more.