News for hackers. Stuff that matters.
Re:Detecting PhantomJS-based visitors by weinzierl on Friday January 16, 2015 @03:49PM (#8901609)
My guess is that the pattern is simply "too many requests from a single IP" plus a manually maintained white list.

When working from within the network of a large company that uses a proxy I saw the CAPTCHA regularly for some time.

Of course the CAPTCHA gets answered mostly correctly in this case, which could trigger manual inspection and finally addition to the white list.

Re:Detecting PhantomJS-based visitors by aselzer on Friday January 16, 2015 @04:18PM (#8901766)
I also think it doesn't have to be that complex.

I once scraped Google Search a few thousand times, in a few seconds, a few times, just for fun, and in a school (1000+ people). It was quite funny when everyone suddenly saw a captcha.

Then a month ago or so I scraped Google+ for some statistics, maybe 1-3k times in 2-3 minutes and nothing happened.

The point is that my node.js scraper was far from perfect. Just a fake Chrome user agent, nothing else.

If a large organization with a static IP has a lot of users, Google is clever enough to figure out that it is large and has a high average number of requests by e.g. seeing that many people use their distinct Google Accounts.