Avoiding bot detection: How to scrape the web without getting blocked?

Re:Avoiding bot detection: How to scrape the web without get... by walrus01 on Sunday October 31, 2021 @08:00PM (#29061692)

One of the reasons for this is that the vast majority of the time, mobile LTE data users are behind cgnat for ipv4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point.

As a scraper operator on a mobile data connection all you need is a new useragent and browser fingerprint, there's no easy way for a scraper-blocker-operator to tell that you're not a totally new person.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by cute_boi on Monday November 01, 2021 @02:18AM (#29063546)

This is reason why most of services uses App instead of Browser. When there is App it can use lot of thing like phone fingerprint derived from various sources.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by dzhiurgis on Monday November 01, 2021 @03:32AM (#29063920)

Apple provides a framework or some sort of unique id

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by varenc on Monday November 01, 2021 @08:43PM (#29075412)

Note that Apple explicitly tries to prevent apps from generating any sort of overall device fingerprint besides the ad tracking identifier one which now requires user consent in iOS 14. You can still generate an app-scoped device ID though. (Not sure if these persist across re-installs of the same app or not)

[ Parent | on HN ]