News for hackers. Stuff that matters.
Re:Detecting PhantomJS-based visitors by andrenotgiant on Friday January 16, 2015 @03:32PM (#8901491)
The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google

In my experience, these two use detection algorithms that could best be described as "pattern-based."

Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or forced to fill out a CAPTCHA.

For example, you can hit Google Search once an hour via a painfully obvious scraper no problem. But, even if you take control of a Chrome browser via selenium, and write your scraper to do everything exactly like humans down to typing in the searches and moving the mouse around and clicking results, Google is ridiculously good at identifying bot vs human traffic patterns.

I think the algorithm builds a "normal usage profile" for a combination of IP,cookie/user,device and sets a threshold, activity above that threshold gets flagged.

Re:Detecting PhantomJS-based visitors by hwh on Friday January 16, 2015 @05:59PM (#8902331)
I can trigger Google's Captcha simply by using Chrome's Incognito Mode plus a SSH-based SOCKS proxy. No automation at all. Which is probably against the "normal usage profile" build for data Google gets from me despite Incognito Mode. Well, for starters, dropping Cookies all the time is probably an indicator. Not sure if they can somehow see my SOCKS tunnel - I hope they can't, but am not basing my security on that.
Re:Detecting PhantomJS-based visitors by codezero on Friday January 16, 2015 @07:03PM (#8902604)
I'm assuming that your SOCKS tunnel endpoint is on a VPS, or some hosted solution, if that's the case, this probably explains it since those IPs are not the source of typical "user" web traffic. I get the same when I use a VPN.
Re:Detecting PhantomJS-based visitors by snowwrestler on Friday January 16, 2015 @05:39PM (#8902208)
This type of protection is also aligned with what I think of as the most legitimate concern, which is resource exhaustion. I don't understand why some site operators object to scrapers even when they are well-behaved, and collecting public information. I think if the load is low, let scrapers scrape.
Re:Detecting PhantomJS-based visitors by weinzierl on Friday January 16, 2015 @03:49PM (#8901609)
My guess is that the pattern is simply "too many requests from a single IP" plus a manually maintained white list.

When working from within the network of a large company that uses a proxy I saw the CAPTCHA regularly for some time.

Of course the CAPTCHA gets answered mostly correctly in this case, which could trigger manual inspection and finally addition to the white list.

Re:Detecting PhantomJS-based visitors by aselzer on Friday January 16, 2015 @04:18PM (#8901766)
I also think it doesn't have to be that complex.

I once scraped Google Search a few thousand times, in a few seconds, a few times, just for fun, and in a school (1000+ people). It was quite funny when everyone suddenly saw a captcha.

Then a month ago or so I scraped Google+ for some statistics, maybe 1-3k times in 2-3 minutes and nothing happened.

The point is that my node.js scraper was far from perfect. Just a fake Chrome user agent, nothing else.

If a large organization with a static IP has a lot of users, Google is clever enough to figure out that it is large and has a high average number of requests by e.g. seeing that many people use their distinct Google Accounts.

Re:Detecting PhantomJS-based visitors by curiously on Friday January 16, 2015 @04:56PM (#8901991)
I think that is overestimating their efforts. To build such profile and to monitor each and every activity would be very costly.

I think what happens is that when they detect too many requests from an ip address in a short interval amount of time they throw a captcha.

Re:Detecting PhantomJS-based visitors by andrenotgiant on Friday January 16, 2015 @06:03PM (#8902362)
There has to be some calibration or customization beyond IP. Traffic from University IP vs my home looks very different.