News for hackers. Stuff that matters.

A comment on Avoiding bot detection: How to scrape the web without getting blocked?. Parent comment

Re:Avoiding bot detection: How to scrape the web without get... by leppr on Monday November 01, 2021 @07:36AM (#29065411)
With many of these big anti-bot services like Google ReCaptcha, it's not even specialized anonymity tools that can cause shadow banning, just unusual user-agents.

All of these have independently caused me to get into endless ReCaptcha loops: firefox on android, smartphone with unusual screen resolution, clean browser profile with VPN.

It's so common that I now default to using duckduckgo, which never blocks me. I doubt DDG has a lower DDoS/Resources ratio than Google. Some companies are just lazier and less principled than others.

[ Parent | on HN ]
Re:Avoiding bot detection: How to scrape the web without get... by RNCTX on Monday November 01, 2021 @01:59PM (#29070405)
> it's not even specialized anonymity tools that can cause shadow banning, just unusual user-agents.

"Unusual" = not Chrome and doesn't allow tracking scripts.

Switch to Safari with an ad blocker for a week, see how many more ReCaptcha prompts you get.

[ Parent | on HN ]
Re:Avoiding bot detection: How to scrape the web without get... by charcircuit on Monday November 01, 2021 @08:42AM (#29065986)
As a person who also uses mobile Firefox, I don't feel like I personally have issues with recaptcha.
[ Parent | on HN ]