News for hackers. Stuff that matters.
Re:Ask HN: What do people use to prevent crawlers? by bad_user on Sunday July 23, 2017 @07:22AM (#14831751)
The other day I've made a Chrome extension for scrapping a protected website. It worked wonderfully, as it simulated a normal user session, bypassing the JavaScript protections the website has. You can also run such scripts with a headless browser for full automation, PhantomJS being an obvious choice.

You really can't protect against this unless you start making the experience of regular visitors much worse.

Re:Ask HN: What do people use to prevent crawlers? by winkv on Monday July 24, 2017 @09:02AM (#14838047)
Interesting, i am facing problem with this approach.There is a site which pops up alert box for username and password and i can't seem to switch to it with chrome web extension.With selenium it is as simple as driver.switchTo().alert(); have you ever faced this problem? by any chance you know how to fix it? I am trying to accomplish the same thing as described in this question in SO https://sqa.stackexchange.com/questions/20710/not-able-to-us...
Re:Ask HN: What do people use to prevent crawlers? by inglor on Sunday July 23, 2017 @08:00AM (#14831862)
Sure you can protect against this - there are several companies that use machine learning to spot small differences between selenium and real users (mouse delays etc).

For example, it might detect that a mouse click is dispatched at exact intervals (and block it). To which you'd think "I'll just add Math.random() * 2000" which it'll easily detect as well.

It's _definitely_ doable, but it's not as trivial as recording a selenium macro. (Not to mention these tools look for selenium presence and extensions anyway).

Re:Ask HN: What do people use to prevent crawlers? by bad_user on Monday July 24, 2017 @05:20AM (#14836969)
The issue here isn't that it can't be done, the problem being one of false positives.

By doing this you must accept that a certain percentage of legitimitate users, which can be quite significant, will be blocked. In case you're wondering, yes, this does happen with solutions such as Cloudflare.

And at that point, either your website isn't popular, in which case you can't afford to lose users, or it is very popular, in which case you'll piss of enough users as to receive bad reviews.

Basically you can't afford to do this, unless you're Facebook or Google, and you have to then wonder why Facebook or Google do not deploy such protections

So going back to my main point, of course it's possible, but the experience for users gets significantly worse such that you won't want to do it.

Re:Ask HN: What do people use to prevent crawlers? by inglor on Monday July 24, 2017 @07:04AM (#14837404)
I've found Cloudflare's solution to be very inadequate compared to Distil or Incapsula.

Both Distil and Incapsula _did_ have a small number of false positives (showing a captcha to users). We did have to write some code to overcome that.

Re:Ask HN: What do people use to prevent crawlers? by laughfactory on Monday July 24, 2017 @01:03PM (#14840262)
Yeah, but if you can use ML to pick out the differences between real users and bots then you can also use ML to mimic real user behavior. Both sides spend a lot of time and money trying to overcome each other, and end state is the same it was at the beginning: near total access to the data, at least for those sophisticated enough.
Re:Ask HN: What do people use to prevent crawlers? by pbhjpbhj on Sunday July 23, 2017 @08:10AM (#14831893)
IF you throttle the connection then wouldn't download time and network latency, etc., give you a better random?
Re:Ask HN: What do people use to prevent crawlers? by StavrosK on Sunday July 23, 2017 @08:31AM (#14831962)
You can generally tell because of the distribution, which tends to show up after lots of data points.
Re:Ask HN: What do people use to prevent crawlers? by amelius on Sunday July 23, 2017 @07:28AM (#14831763)
But your crawler must be very slow then.
Re:Ask HN: What do people use to prevent crawlers? by pmx on Sunday July 23, 2017 @07:30AM (#14831768)
They generally are fairly slow but it doesn't matter because you just set it and leave it to it until it has all the data you want, leaving you to get on with other things.
Re:Ask HN: What do people use to prevent crawlers? by alexcnwy on Sunday July 23, 2017 @07:33AM (#14831781)
Plus very easy to parallelize
Re:Ask HN: What do people use to prevent crawlers? by amelius on Sunday July 23, 2017 @11:27AM (#14832672)
Unless the websites work together to protect themselves against crawlers that work that way (e.g., rejecting requests to different hosts from the same origin within a time interval).