News for hackers. Stuff that matters.
Re:Avoiding bot detection: How to scrape the web without get... by TedDoesntTalk on Sunday October 31, 2021 @07:31PM (#29061487)
Not anymore. Now it’s mobile IP addresses.
Re:Avoiding bot detection: How to scrape the web without get... by varenc on Sunday October 31, 2021 @07:52PM (#29061632)
This! Mobile IPs are far more lucrative. Many services will drop captchas and other anti-bot stuff for consumer mobile IPs. I recall Plaid at some point would run their bank scraping through mobile IPs.

This sketchy company lets mobile app developers monetize user base by letting other people pay $$ to route requests through random people’s mobile IPs: https://brightdata.com/

Re:Avoiding bot detection: How to scrape the web without get... by dzhiurgis on Monday November 01, 2021 @02:57AM (#29063761)
Brightdata is formerly known as as Luminati who is owned by same company as Hola VPN.

Similarly NordVPN owns Oxylabs (who mostly hack routers and cameras and sell those as residential IP’s).

Re:Avoiding bot detection: How to scrape the web without get... by varenc on Monday November 01, 2021 @04:27AM (#29064216)
But they have a white paper on how ethical their proxies are! https://oxylabs.io/Oxylabs_Residential_Proxy_Acquisition_Han...
Re:Avoiding bot detection: How to scrape the web without get... by herbst on Monday November 01, 2021 @05:00AM (#29064373)
I never checked their business practice. But I use Luminati rotating residential proxies for a lot of my scraping work.

Mostly to avoid hitting 'ddos protections' or other security bullshit that doesn't really make sense on 1 daily request or so.

Re:Avoiding bot detection: How to scrape the web without get... by cute_boi on Monday November 01, 2021 @02:16AM (#29063534)
This is really bad. Imagine if someone plants these proxy inside app how user are even going to know? I think every OS should come with firewall so if app tries to make connection it should prompt with Accept | Accept Forever | Deny | Deny Forever.

I think these companies used to go for extension developer now it seems they have found new idea to implant malware on apps which is not easy to detect.

Re:Avoiding bot detection: How to scrape the web without get... by TedDoesntTalk on Monday November 01, 2021 @02:51AM (#29063734)
This is EXACTLY how mobile proxy companies like Luminati and OxyLabs acquire their ip address pool. They pay devs to embed a lib inside their app.
Re:Avoiding bot detection: How to scrape the web without get... by dzhiurgis on Monday November 01, 2021 @02:58AM (#29063765)
They hack IP cams and routers too…
Re:Avoiding bot detection: How to scrape the web without get... by walrus01 on Sunday October 31, 2021 @08:00PM (#29061692)
One of the reasons for this is that the vast majority of the time, mobile LTE data users are behind cgnat for ipv4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point.

As a scraper operator on a mobile data connection all you need is a new useragent and browser fingerprint, there's no easy way for a scraper-blocker-operator to tell that you're not a totally new person.

Re:Avoiding bot detection: How to scrape the web without get... by cute_boi on Monday November 01, 2021 @02:18AM (#29063546)
This is reason why most of services uses App instead of Browser. When there is App it can use lot of thing like phone fingerprint derived from various sources.
Re:Avoiding bot detection: How to scrape the web without get... by dzhiurgis on Monday November 01, 2021 @03:32AM (#29063920)
Apple provides a framework or some sort of unique id
Re:Avoiding bot detection: How to scrape the web without get... by varenc on Monday November 01, 2021 @08:43PM (#29075412)
Note that Apple explicitly tries to prevent apps from generating any sort of overall device fingerprint besides the ad tracking identifier one which now requires user consent in iOS 14. You can still generate an app-scoped device ID though. (Not sure if these persist across re-installs of the same app or not)