Re:Ask HN: Best practices for ethical web scraping?
by chatmasta on Saturday April 04, 2020 @01:27PM
(#22779803)
I disagree. The risks are similar to those of disclosing a security vulnerability to a company without a bug bounty. You cannot know how litigious or technically illiterate the company will be. What if they decide you're "hacking" them and call the FBI with the helpful information you included in your user agent? Crazier things have happened.
Anonymity is part of the right to privacy; IMO, such a right should extend to bots as well. There should be no shame in anonymously accessing a website, whether via automated means or otherwise.
Re:Ask HN: Best practices for ethical web scraping?
by a1369209993 on Saturday April 04, 2020 @04:00PM
(#22781030)
> such a right should extend to bots as well
No, it very much shouldn't, but (as you probably meant) it should extend to the person (not, eg, company) using a bot, which amounts to the same thing in this case.
Oh, and for 3.: if you can, apply some heuristics to your reading of the robots.txt. If it's just "deny everything", then ignore it, but you really don't want to be responsible for crawling all of the GET /delete/:id pages of a badly-designed site… (those should definitely be POST, and authenticated, by the way).