Re:Ask HN: What’s the legality of web scraping?
by reefoctopus on Sunday June 23, 2019 @01:12PM
(#20257004)
Teach your students to ensure there’s a delay between requests so they aren’t hammering anyone’s server, and follow the rules in the robots.txt. I’ve scraped more than a billion pages without any issues.
[ Parent
| on HN ]
Re:Ask HN: What’s the legality of web scraping?
by avip on Sunday June 23, 2019 @02:21PM
(#20257621)
He asked about legality, not technical difficulty.
Re:Ask HN: What’s the legality of web scraping?
by malshe on Sunday June 23, 2019 @05:36PM
(#20258826)
Actually many of the students are technically competent to do the scrpaing mostly using Python and I am pretty sure they learned not to overwhelm web servers.
Re:Ask HN: What’s the legality of web scraping?
by gshdg on Sunday June 23, 2019 @01:13PM
(#20257016)
Just because it’s technically feasible does not mean it’s legal or ethical.
Re:Ask HN: What’s the legality of web scraping?
by dheera on Sunday June 23, 2019 @01:21PM
(#20257083)
I'd say the act of web scraping alone, is almost never unethical if you are careful to not cause undue load to servers. From an ethics, not legal perspective, I don't see a whole lot of difference between your computer's silicon eyes and your organic eyes just looking at something that's already in plain view.
It might be illegal in some jurisdiction; IANAL but I think you can just get out of that jurisdiction and scrape away if that is the case. It might violate some ToS but ToS isn't law; the consequences of violating a ToS are usually on the order of getting your IP banned.
What you do with the stuff you scraped can be ethical or unethical.
Re:Ask HN: What’s the legality of web scraping?
by reefoctopus on Sunday June 23, 2019 @01:15PM
(#20257029)
What makes it unethical?
Why should I be treated differently than search engine spiders?
If somebody doesn’t want their site scraped then they can let people know with robots.txt. Get off your high horse.
Re:Ask HN: What’s the legality of web scraping?
by patrickmcnamara on Sunday June 23, 2019 @06:52PM
(#20259244)
They never said it was unethical.