Re:The State of Web Scraping in 2021
by mdaniel on Monday October 11, 2021 @12:48PM
(#28830280)
I would expect it's roughly the same answers, just varying in the specifics:
* those which don't offer a _reasonable_ API, or (I would guess a larger subset) those which don't expose all the same information over their API
* those things which one wishes to preserve (yes, I'm aware that submitting them to the Internet Archive might achieve that goal)
* and then the subset of projects where it's just a fun challenge or the ubiquitous $other
As an example answer to your question, some sites are even offering bounties for scraped data, so one could scratch a technical itch and help data science at the same time:
* those which don't offer a _reasonable_ API, or (I would guess a larger subset) those which don't expose all the same information over their API
* those things which one wishes to preserve (yes, I'm aware that submitting them to the Internet Archive might achieve that goal)
* and then the subset of projects where it's just a fun challenge or the ubiquitous $other
As an example answer to your question, some sites are even offering bounties for scraped data, so one could scratch a technical itch and help data science at the same time:
https://www.dolthub.com/repositories/pdap/datasets/bounties