Ask HN: Should you block web scrapers?

Re:Ask HN: Should you block web scrapers? by softwaredev__ on Monday February 01, 2016 @01:37PM (#11014074)

You should go after the low hanging fruits first. Check if the request is coming from a trusted browser (Chrome, Firefox, Safari, etc). If not, block them. I'm sure that'll cut down scrapes by a lot. And then, you can go after some behavioral stuff. You can do things like measure the milliseconds between requests. If it's too fast for a human to have clicked that link after the initial page load, then you know it's a bot.

[ Parent | on HN ]

Re:Ask HN: Should you block web scrapers? by ldjb on Monday February 01, 2016 @01:46PM (#11014181)

Please don't do that. Some of us use rather obscure web browsers, and unless you keep the whitelist updated, users of newer browsers will be locked out of the website.

Besides, if your website blocks me, I'm not going to switch browser — I'll just go elsewhere.

[ Parent | on HN ]

Re:Ask HN: Should you block web scrapers? by softwaredev__ on Monday February 01, 2016 @03:59PM (#11015381)

Just curious, what obscure browsers do you use?

[ Parent | on HN ]

Re:Ask HN: Should you block web scrapers? by ldjb on Monday February 01, 2016 @04:41PM (#11015708)

Some examples are Lynx, Iceweasel, Chromium, Steam's in-game browser and the Nintendo 3DS browser. Okay, none of them are that obscure but they probably aren't the first things that spring to mind when you think of browsers.

Depending on your method of blocking browsers, it might automatically lump Iceweasel together with Firefox and Chromium together with Chrome. But if this is something you really want to do, you have to be very careful about these things.

[ Parent | on HN ]

Re:Ask HN: Should you block web scrapers? by jorgecurio on Monday February 01, 2016 @05:46PM (#11016243)

so it seems like the only sure-fire way of detecting bad bots is via an IP address and not relying on Browser Agent.

Is it any beneficial if there's a way to detect if a given IP address is bad or not right from the get go? Possibly by searching through millions of IP addresses that are blacklisted through a bloom filter and REST API? Do you think this would help-> a long roster of abusive IP address databases?

[ Parent | on HN ]

Re:Ask HN: Should you block web scrapers? by ldjb on Monday February 01, 2016 @06:25PM (#11016501)

IP address blacklists can be effective, and certainly make a lot more sense than blocking anything that doesn't appear on a whitelist.

Depending on how you are presenting the data, you might also find it helpful to impose a rate limit (accept only a certain number of requests from a particular IP address within a given timeframe).

[ Parent | on HN ]

Re:Ask HN: Should you block web scrapers? by jorgecurio on Monday February 01, 2016 @10:43PM (#11017532)

if there was a public API that you could cross reference IP from your website that you suspect as a bot, would you use it? Would you pay for that kind of API?

Of course I would start off by ironically scraping proxies from public domain as those are often used. Not sure what else determined scrapers use to get data, but it's safe to say that 98% of scrapers are trying to pay as little money as possible and often complain when a scraper gets blocked...fuck these people seriously

[ Parent | on HN ]

Re:Ask HN: Should you block web scrapers? by ldjb on Tuesday February 02, 2016 @09:31AM (#11019580)

I personally don't have any use for such an API, but I'm sure others do. This might be something that already exists, I'm not too sure.

[ Parent | on HN ]