> I can't imagine many of the owners of those IPs know what they're being used for.

They don’t. For instance, Luminati, possibly the best known player in this market, uses HolaVPN users as exit nodes.[1]

[1] https://www.trendmicro.com/vinfo/hk-en/security/news/cybercr...

I was approached by Luminati on Twitter to turn my browser extensions into exit nodes, they are enticing developers to exploit users.

https://i.imgur.com/EbT96an.png

This is why these days I roll my own extensions except for a few extremely popular ones developed in the open. Forget small QoL extensions, if I need it I write my own. Malware businesses routinely reach out to any remotely popular extension for acquisitions or “joint ventures”, it’s just too risky to install anything that’s not trust: ultimate, or known to be vetted by tons of people.

Btw, Firefox seems to have made it impossible to run extensions uncrippled from source without uploading to their server for signing. Utter madness.

I was wondering why my extensions weren't loading up even if after I messed with the settings, thank you!

Auto-update of extensions and apps should be set to off by default. There would be less reason for extensions to be bought by other developers for their installed user base.

Secondly, how come there is no regulation about selling chrome extensions and apps?

If you turn auto-updates off, you're just reinventing early 2000s desktop security and attackers everywhere will thank you. Updates need to be automatic but there needs to be stricter rules about notifications to users about changes in ownership or functionality. A huge fraction of this would go away if corporate liability didn't allow hiding all of this in terms of service.

This is said way too often, but in this case I think it's justified - I really don't understand how this is legal. Doesn't using a non-consenting person's computer and network resources for your own purposes meet the definition of "hacking" in most jurisdictions?

It probably isn't legal. But who is going to pay for the fight to stop it? Mom & pop who have had their home connection used? A class action that no one but the lawyers really win anything from? The fact that most of the projects involved are in jurisdictions that make the legal process more complicated and more expensive, or jurisdictions that simply don't care as long as it is affecting other countries and not them (coughChinacoughRussiacoughcough*)?

What the market will bare doesn't just apply to product pricing.

And someone clicking through a EULA might be an argument for them having invited the use of their resources by a very similar argument to that many used to justify using other people's WiFi back when that was gloriously insecure most of the time ("but the AP broadcast its existence, effectively inviting me to join, and happily gave me an IP address to work with, how could I not believe I was welcome to use it?").

I am an administrator on a forum that was attacked heavily by these "residential proxy services"; most of the time the users have agreed to a hidden EULA somewhere like at installation time of the extension buried deep within.

But almost everyone "a reasonable person" would not recognise or understand the technical words used in the disclaimer, nor even read all the way down where it's buried several links in.

On the other hand, if these things get common enough it's great for plausible deniability!

I've never heard such service exists and uses browser extension user's network. It looks horrible. Possibly random browser extension user may arrested by unknown reason.

Thanks for the link. I figured something like that had to be the case. I can't really think of a legitimate way to get access to that many IPs.

I periodically see ads on Kijiji (Craigslist for Canada) offering you 50 bucks for your internet connection. If that is the going rate, 50 million I enormously expensive.

The boxes have the ability to try to DHCP-release-renew your IP to get a new one after being banned for spam/fraud/DDoS/etc.

However, this is not how these residential proxies are sold- mass market ones with millions of IPs are generally botnet/hacked/malware/extension malicious author sellout/etc./ There are not actually millions of botnet user, they just try to make themselves look big. Like they say AT&T has a /8? So they get 1 botnet hacked PC on AT&T broadband and say "wow we have a /8 worth of proxies!!"). Criminals not known for their truthfulness.

The ones that ask you $20-50/m and to plug something into your network is generally used for fraud, account hijackings and maintenance (getting a stable IP for stuff like a credit card checkout or paypal account to use for more than a few days for fraud, long term social media abuse, similar) instead of mass market ban evasion.

These are split into either unknowing people with their computers abused like Hola, malware running proxies being sold, or otherwise. Some mobile games and chrome extensions expose proxies also as a monetisation method.

Don't support them.

I've been contacted by several of those "SDK companies" that basically turn phones and browsers into botnet drones for $0.01 an install.

Do you know of any reputable ones with a sustainable source of IPs that can be used for scraping?

There are no reputable ones; they are (>99,99%+ probably) all based off of unwanted adware or malware/botnet exploit installation. There are some fake ones that also commit wire fraud, fraudulent letters of authorisation and do BGP hijacking, which is arguably significantly worse.

https://packetstream.io is 100% opt-in and transparent

Do you need residential IPs for this? As far as I've experienced, YouTube doesn't block dedicated server providers, as such you may be able to use a commercial VPN provider, some of them have 1000s of IPs and offer easy or automated ways to switch frequently.

SEO companies are big customers of products like this. Scraping Google at scale requires a huge budget for proxies and IP blocks, and I believe they apply ML to detect people abusing their service.

I actually don't think I would need to make a ridiculous amount of requests. The project is a pinning service where people submit a video URL so they can do things like manage lists of the videos they watch (think goodreads for web videos). I need to scrape the video description and tags for each submitted URL. It would be nice to just do it from the user's browser, but the same origin policy won't allow that. A browser extension is the next obvious route, but that has its own issues. I was thinking to first attempt the request via CloudFlare workers, then fall back to a residential proxy service. No idea how usable workers would end up being for this sort of scraping.

This sounds like your would be better of using Youtube's API https://developers.google.com/youtube/v3/getting-started

Yeah it would probably be the easiest way to get started. Their ToS[0] just seems so convoluted it's hard to know if you're following it or not, so I don't want to tie myself to it with no recourse.

[0]: https://developers.google.com/youtube/terms/api-services-ter...

Yes, I was going to mention this. I worked for a small company that had a YouTube product that retrieved video descriptions, duration, thumbnail, etc, using the API.

We started with 100k daily API calls and that got later increased to 1 million after requesting it.

Looks like the limit is 10k/day now. What did the process look like to get an increase?

automated form for me.

Nice. And that got you from 10k to 1M?

I am at 10m+, keep in mind it isn't 1:1 with requests. Things like GET read only video metadata is one request, but if you are uploading a video, it might count as more out of your quota.

You are expected to be well behaved though. You can also lose your quota or API access for severe abuse, like using API to mass upload spam.

Very nice. I only need metadata. And if I'm not mistaken I can even batch requests to multiple video ids if I ever reach that level of scale.

I can confirm that, scraping Google instantly needs huge effort and money. In our best we can scrape 2500 SERP per IP.

But i must say using proxy services and other things did not helped us much. Because most of them were banned before we use.

Yeah, and then you recycle the IP back into the pool for the next guy to work with. An operation I know of was getting 6+ million SERPs a day, budget for proxies was hundreds of thousands a year.

How does it wotk out for IPv6?

IPv6 is just not widely used, so when you do use it, you stick out like a sore thumb. Think like a bayesian: for Google, it's easy to just block whole /32s of IPv6 space.

At what point would you consider IPv6 "widely used"? It's currently 30% of traffic:

https://www.google.com/intl/en/ipv6/statistics.html

I bet you half of that is scrapers!

Complete /64's get blocked.