Re:Bypassing website anti-scraping protections
by dx034 on Friday June 01, 2018 @08:52AM
(#17204793)
The rule set must be more complex. I often use VPN which results in captchas on many pages but I never get one on Google. I guess the 300 queries/IP only count if other parameters indicate crawling.
[ Parent
| on HN ]
Re:Bypassing website anti-scraping protections
by cm2187 on Friday June 01, 2018 @10:02AM
(#17205388)
But it's a bit clunky. I was running searches through an embedded webbrowser in a c# application, which is really an embedded internet explorer and was very quickly presented with a captcha. It was a human viewing the results, but a script constructing the query string, but that was enough the be labelled as a crawler.