News for hackers. Stuff that matters.
Re:Bypassing website anti-scraping protections by dx034 on Friday June 01, 2018 @08:52AM (#17204793)
The rule set must be more complex. I often use VPN which results in captchas on many pages but I never get one on Google. I guess the 300 queries/IP only count if other parameters indicate crawling.
Re:Bypassing website anti-scraping protections by cm2187 on Friday June 01, 2018 @10:02AM (#17205388)
But it's a bit clunky. I was running searches through an embedded webbrowser in a c# application, which is really an embedded internet explorer and was very quickly presented with a captcha. It was a human viewing the results, but a script constructing the query string, but that was enough the be labelled as a crawler.
Re:Bypassing website anti-scraping protections by jocoda on Friday June 01, 2018 @12:56PM (#17207356)
often all it takes is to be using webbrowser. I would love to use webbrowser for small search utilities because it's so easy to use, but it seems to be a magnet for problems.
Re:Bypassing website anti-scraping protections by jardah on Friday June 01, 2018 @09:00AM (#17204846)
Yea, it was a very general example, since there is at least one rule that is based on rate limiting too, and this 300/IP limit is what have seen on average.