News for hackers. Stuff that matters.

A comment on Web Scraping for Me, but Not for Thee. Parent comment

Re:Web Scraping for Me, but Not for Thee by mrkeen on Friday August 25, 2023 @04:51PM (#37266881)
> Scrappers can be dicks about it too

It's not always about individual bad actors. You can have lots of small players causing problems. I wonder how many python developers there are right now trying to make their own offline copy of stackoverflow.com.

Wikipedia has a great defence against this. They ask you not to scrape, and at the same time, provide torrents of the data (https://meta.wikimedia.org/wiki/Data_dump_torrents)

[ Parent | on HN ]
Re:Web Scraping for Me, but Not for Thee by rzzzt on Friday August 25, 2023 @05:33PM (#37267305)
Stack Exchange also provides one: https://archive.org/details/stackexchange

There was a hiccup around June but that seems resolved now: https://meta.stackexchange.com/questions/389922/june-2023-da...

[ Parent | on HN ]
Re:Web Scraping for Me, but Not for Thee by genewitch on Saturday August 26, 2023 @06:59PM (#37277689)
yeah i thought this was the case. There's a couple of other sites that - i hesitate to state this completely factual phrase - leave dumps for the public to pick up.
[ Parent | on HN ]