News for hackers. Stuff that matters.
Re:Ask HN: Best practices for ethical web scraping? by snidane on Saturday April 04, 2020 @12:28PM (#22779282)
When scraping just behave as to not piss off the site owner - whatever that means. Eg. not cause excessive load or making sure you don't leak out sensitive data.

Next put yourself in their shoes and realize they don't usually monitor their traffic that much or simply don't care as long as you don't slow down their site. It's usually only certain big sites with heavy bot traffic such as linkedin or sneaker shoe sites which implement bot protections. Most others don't care.

Some websites are created almost as if they want to be scraped. The json api used by frontend is ridiculously clean and accessible. Perhaps they benefit when people see their results and invest in their stock. You never fully know if the site wants to be scraped or not.

The reality of scraping industry related to your question is this

1. scraping companies generally don't use real user agent such as 'my friendly data science bot' but they hide behind a set of fake ones and/or route the traffic through a proxy network. You don't want to get banned so stupidly easily by revealing user agent when you know your competitors don't reveal theirs.

2. This one is obvious. The general rule is to scrape over long time period continuously and add large delays between requests of at least 1 second. If you go below 1 second be careful.

3. robots.txt is controversial and doesn't serve its original purpose. It should be renamed to google_instructions.txt because site owners use it to guide googlebot to navigate their site. It is generally ignored by the industry again because you know your competitors ignore it.

Just remember the rule of 'not to piss off the site owner' and then just go ahead and scrape. Also keep in mind that you are in a free country and we don't discriminate here whether it is of racial or gender reasons or whether you are a biological or mechanical website visitor.

I simply described the reality of data science industry around scraping after several years of being in it. Note that this will probably not be liked by HN audience as they are mostly website devs and site owners.

Re:Ask HN: Best practices for ethical web scraping? by codingdave on Saturday April 04, 2020 @03:28PM (#22780785)
You are correct that I don't like this advice... not because I find it to be wrong, but because you are approaching it solely from a competitive perspective -- "Your competitors don't have ethics, so you shouldn't either." That doesn't help someone who is engaging in research and trying to hold themselves to a higher standard.
Re:Ask HN: Best practices for ethical web scraping? by jdc on Sunday April 05, 2020 @12:16PM (#22786437)
It seems to me that you are conflating ethics with compliance.
Re:Ask HN: Best practices for ethical web scraping? by codingdave on Monday April 06, 2020 @07:32AM (#22792761)
When the compliance we are talking about is whether or not to communicate clearly to others who you are, and whether or not to respect their directions, are you so sure they are not correlated?
Re:Ask HN: Best practices for ethical web scraping? by lordgrenville on Saturday April 04, 2020 @04:04PM (#22781057)
I'm neither a web dev nor a site owner, but OP literally asked for tips on ethical web scraping, not "what's the most I can get away with".
Re:Ask HN: Best practices for ethical web scraping? by trickstra on Saturday April 04, 2020 @11:23PM (#22783508)
That's because OP is operating under the wrong assumption that sites won't ban an ethical scraper. The reality is that they will, and much faster than an unethical one. They don't care about your science project, they want ad revenue, conversions...
Re:Ask HN: Best practices for ethical web scraping? by wizzwizz4 on Saturday April 04, 2020 @12:38PM (#22779365)
1. is the only one I don't like. I think you should use your real user agent first on any given site, as a courtesy; whether you give up or change to a more "normal" user agent if you get banned is up to you.

Oh, and for 3.: if you can, apply some heuristics to your reading of the robots.txt. If it's just "deny everything", then ignore it, but you really don't want to be responsible for crawling all of the GET /delete/:id pages of a badly-designed site… (those should definitely be POST, and authenticated, by the way).

Re:Ask HN: Best practices for ethical web scraping? by chatmasta on Saturday April 04, 2020 @01:27PM (#22779803)
I disagree. The risks are similar to those of disclosing a security vulnerability to a company without a bug bounty. You cannot know how litigious or technically illiterate the company will be. What if they decide you're "hacking" them and call the FBI with the helpful information you included in your user agent? Crazier things have happened.

Anonymity is part of the right to privacy; IMO, such a right should extend to bots as well. There should be no shame in anonymously accessing a website, whether via automated means or otherwise.

Re:Ask HN: Best practices for ethical web scraping? by a1369209993 on Saturday April 04, 2020 @04:00PM (#22781030)
> such a right should extend to bots as well

No, it very much shouldn't, but (as you probably meant) it should extend to the person (not, eg, company) using a bot, which amounts to the same thing in this case.

Re:Ask HN: Best practices for ethical web scraping? by gilad on Saturday April 04, 2020 @01:56PM (#22780051)
As for delete, use authenticated DELETE, not POST, it’s why its there in the first place
Re:Ask HN: Best practices for ethical web scraping? by mpclark on Saturday April 04, 2020 @01:22PM (#22779745)
Also, if a target site is behind Cloudflare then you probably won’t be able to masquerade as any of the popular bots - they block fake google/yandex bots.
Re:Ask HN: Best practices for ethical web scraping? by hutzlibu on Saturday April 04, 2020 @03:58PM (#22781012)
"or making sure you don't leak out sensitive data"

If sensitive data can be scraped, it is not really stored sensitive. So I would not care too much about it and just notify the owner if I notice it.

Re:Ask HN: Best practices for ethical web scraping? by HenryBemis on Saturday April 04, 2020 @06:39PM (#22782118)
Keep in mind that if you end up with data that are protected under GDPR, merely having them puts you in a damning position. The intended owner will be fried for not protecting it adequately, but you violate GDPR since "I never agreed to you collecting, processing, etc" the data. And imagine the world of pain if you are caught with children's data.
Re:Ask HN: Best practices for ethical web scraping? by wolco on Saturday April 04, 2020 @07:49PM (#22782476)
The site might be if located in Europe but if it's public you are not bound by those terms as long as you don't share with Europeans.
Re:Ask HN: Best practices for ethical web scraping? by fiddlerwoaroof on Saturday April 04, 2020 @10:01PM (#22783143)
In the US I’d be more concerned about the CCPA.
Re:Ask HN: Best practices for ethical web scraping? by aspyct on Saturday April 04, 2020 @12:54PM (#22779503)
Well, having a few websites of my own, I really do think that point 1 is the worst. I can't filter bots that disguise as users from my access logs, and they actually hurt my work (i.e. figuring out what people read).

Totally agree with the rest though. Maybe adapt the "large delay" of 1 second to the kind of website I'm scraping though.

Thanks for your feedback!

Re:Ask HN: Best practices for ethical web scraping? by the8472 on Saturday April 04, 2020 @01:51PM (#22780018)
> I can't filter bots that disguise as users from my access logs, and they actually hurt my work (i.e. figuring out what people read).

If the bots aren't querying from residential IPs you could match their IPs to ASNs and then filter based on that to separate domestic and data center origins.

Re:Ask HN: Best practices for ethical web scraping? by aspyct on Saturday April 04, 2020 @01:56PM (#22780056)
Ha, that's a good idea! Is there a list somewhere of the cidr blocks that are assigned to residential vs server farms? I mean, how can I tell an IP is residential?
Re:Ask HN: Best practices for ethical web scraping? by the8472 on Saturday April 04, 2020 @02:52PM (#22780533)
The other way around may be easier, i.e. excluding known datacenter ranges. There are some commercial databases for that, i'm not sure if there are any free ones. But you can also do this manually by running a whois on an IP and then extracting the ranges from the whois response and caching then. Then you can look at the orgname or something like that. You can also download the whois databases from the RIRs, but they don't contain the information what kind of entities they are.

    $ dig +short reddit.com
    151.101.1.140


    $ whois 151.101.1.140

    NetRange:       151.101.0.0 - 151.101.255.255
    CIDR:           151.101.0.0/16
    OrgName:        Fastly
    [...]
So if you see a known hoster here then you can exclude it from your statistics.
Re:Ask HN: Best practices for ethical web scraping? by capableweb on Saturday April 04, 2020 @04:04PM (#22781059)
What I've done in the past is to pull down all the IPs of request I see, filter by unique, do whois for each one of them (you're gonna need to have a backoff/rate limit here as whois services are usually rate limited) and save the organization name, ASN and CIDR blocks, again filter by uniqueness, then create a new list with the organizations of interest and match with the CIDR blocks. Now you have an allow/blocklist you can use.
Re:Ask HN: Best practices for ethical web scraping? by kitteh on Saturday April 04, 2020 @09:10PM (#22782916)
There are some geoip databases that will denote if it's end user networks and if it's fixed (DSL/cable) or mobile.
Re:Ask HN: Best practices for ethical web scraping? by amelius on Saturday April 04, 2020 @09:11PM (#22782924)
You forgot to mention to use different IP addresses.
Re:Ask HN: Best practices for ethical web scraping? by erdos4d on Saturday April 04, 2020 @03:10PM (#22780664)
Perhaps I am behind the curve here, but why would sneaker shoe sites get scraped hard?
Re:Ask HN: Best practices for ethical web scraping? by abannin on Saturday April 04, 2020 @03:15PM (#22780701)
There is a very active secondary market for sneakers. If you can buy before supply is exhausted, you can make some decent money.