News for hackers. Stuff that matters.
Re:Ask HN: Best practices for ethical web scraping? by wizzwizz4 on Saturday April 04, 2020 @12:38PM (#22779365)
1. is the only one I don't like. I think you should use your real user agent first on any given site, as a courtesy; whether you give up or change to a more "normal" user agent if you get banned is up to you.

Oh, and for 3.: if you can, apply some heuristics to your reading of the robots.txt. If it's just "deny everything", then ignore it, but you really don't want to be responsible for crawling all of the GET /delete/:id pages of a badly-designed site… (those should definitely be POST, and authenticated, by the way).

Re:Ask HN: Best practices for ethical web scraping? by chatmasta on Saturday April 04, 2020 @01:27PM (#22779803)
I disagree. The risks are similar to those of disclosing a security vulnerability to a company without a bug bounty. You cannot know how litigious or technically illiterate the company will be. What if they decide you're "hacking" them and call the FBI with the helpful information you included in your user agent? Crazier things have happened.

Anonymity is part of the right to privacy; IMO, such a right should extend to bots as well. There should be no shame in anonymously accessing a website, whether via automated means or otherwise.

Re:Ask HN: Best practices for ethical web scraping? by a1369209993 on Saturday April 04, 2020 @04:00PM (#22781030)
> such a right should extend to bots as well

No, it very much shouldn't, but (as you probably meant) it should extend to the person (not, eg, company) using a bot, which amounts to the same thing in this case.

Re:Ask HN: Best practices for ethical web scraping? by gilad on Saturday April 04, 2020 @01:56PM (#22780051)
As for delete, use authenticated DELETE, not POST, it’s why its there in the first place
Re:Ask HN: Best practices for ethical web scraping? by mpclark on Saturday April 04, 2020 @01:22PM (#22779745)
Also, if a target site is behind Cloudflare then you probably won’t be able to masquerade as any of the popular bots - they block fake google/yandex bots.