News for hackers. Stuff that matters.
Re:Ask HN: What’s the legality of web scraping? by reefoctopus on Sunday June 23, 2019 @01:12PM (#20257004)
Teach your students to ensure there’s a delay between requests so they aren’t hammering anyone’s server, and follow the rules in the robots.txt. I’ve scraped more than a billion pages without any issues.
Re:Ask HN: What’s the legality of web scraping? by avip on Sunday June 23, 2019 @02:21PM (#20257621)
He asked about legality, not technical difficulty.
Re:Ask HN: What’s the legality of web scraping? by malshe on Sunday June 23, 2019 @05:36PM (#20258826)
Actually many of the students are technically competent to do the scrpaing mostly using Python and I am pretty sure they learned not to overwhelm web servers.
Re:Ask HN: What’s the legality of web scraping? by gshdg on Sunday June 23, 2019 @01:13PM (#20257016)
Just because it’s technically feasible does not mean it’s legal or ethical.
Re:Ask HN: What’s the legality of web scraping? by dheera on Sunday June 23, 2019 @01:21PM (#20257083)
I'd say the act of web scraping alone, is almost never unethical if you are careful to not cause undue load to servers. From an ethics, not legal perspective, I don't see a whole lot of difference between your computer's silicon eyes and your organic eyes just looking at something that's already in plain view.

It might be illegal in some jurisdiction; IANAL but I think you can just get out of that jurisdiction and scrape away if that is the case. It might violate some ToS but ToS isn't law; the consequences of violating a ToS are usually on the order of getting your IP banned.

What you do with the stuff you scraped can be ethical or unethical.

Re:Ask HN: What’s the legality of web scraping? by reefoctopus on Sunday June 23, 2019 @01:15PM (#20257029)
What makes it unethical?

Why should I be treated differently than search engine spiders?

If somebody doesn’t want their site scraped then they can let people know with robots.txt. Get off your high horse.

Re:Ask HN: What’s the legality of web scraping? by patrickmcnamara on Sunday June 23, 2019 @06:52PM (#20259244)
They never said it was unethical.
Re:Ask HN: What’s the legality of web scraping? by matz1 on Sunday June 23, 2019 @02:22PM (#20257627)
Likewise, just because its not legal or in some perspective its unethical, doesn't mean one should not do it.