Avoiding bot detection: How to scrape the web without getting blocked?

Re:Avoiding bot detection: How to scrape the web without get... by sparkling on Sunday October 31, 2021 @08:05PM (#29061723)

There are services that detect residential IPs being used for scraping nowadays. Plus there are other ways of detecting scraping: browser fingerprinting, aggressive rate-limiting and CAPTCHAs etc.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by walrus01 on Sunday October 31, 2021 @08:10PM (#29061755)

Captcha solving services are a thing, can be as crude as something that takes a screenshot, sends an image to a click farm worker getting paid $300 a month sitting in a cubicle in Bangladesh.

There's various captcha solving services where you pay in bulk per captcha and submit data via an api.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by sparkling on Sunday October 31, 2021 @08:13PM (#29061770)

Yup, such click farms exist. But driving up the costs and/or technical implementation efforts for bots/scrapers can be a part of your anti-bot strategy.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by herbst on Monday November 01, 2021 @05:03AM (#29064391)

1000 captchas usually cost around a dollar or two. Honestly captchas never stopped me from anything.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by herbst on Monday November 01, 2021 @05:02AM (#29064382)

It's always just a question of detecting these things and code them in.

It's like writing a game bot with Java robots and pixel detection. It may is inefficient, may takes longer to make than a network solution. But I have yet to be detected anywhere

[ Parent | on HN ]