IPv6 makes it trivial to come up with a mind boggling number of ip addresses. You don't even have to leave your network

Nobody does rate limiting or other abuse work on a full /128. Typically a /64 is treated as the equivalent of an IPv4 address.

This will have many false positives. Especially if the address is from a mobile network.

Last I read this is how Instagram rate limits IPV6. See https://news.ycombinator.com/item?id=32687694

So apparently not enough false positives for them to care.

Is it common for mobile networks to hand out short (64 or smaller) prefixes to phones? Phones are routers after all (hotspot) so this would seem to break stuff.

Multiple phones will use addresses with same prefix. This makes it hard to apply targetted limits to an IP. This is why mobile IPv6 proxies are valuable.

people don't care about false positives on CGNAT either, so not much difference to the IPv4 situation if you target /64s

First of all, Twitter does not support IPv6.

Secondly, if they would, they could simply rate limit by the first x chars of the address.

So your solution is to just make it a walled garden. What about the scraping bots who simply log-in?

My solution is to make it less of a walled garden. At the moment, they often refuse even reading a single tweet when not logged in.

And for the "simply log-in": It is even more expensive (way more expensive) to get good email adresses as it is to get good IPs. Email providers have very tough checks these days before they give you one.

It has to be behavioral, not IP based for the most part because of the ways these networks operate—primarily on personal computers/routers on US-based residential connections.

I deal with these constantly for our clients. It’s amazing how smart they function to get around limits (thousands of IPs and valid recent user agents) but how they miss easy things like using the same cookie to submit 1000s of different entries with obvious purchased lists of names/emails, many of whom are deceased.

"Thousands of IPs" can not hurt Twitters bottom line in any way. With a "read 10 tweets per IP per day" limit, that is 10k tweets. Delivering 10k tweets does not even cost a penny.

10 per day is a ridiculously low limit. Also, NAT and shared networks (think college campus) mean that hundreds of people often share the same IP, making that limit even more ridiculous.

Reality would be that limit would have to be much higher.

https://brightdata.com/proxy-types/residential-proxies gives you access to 70 million IPs. You'd have to be a little insane to scrape millions of tweets like this though because the bandwidth costs would be very high as even with massive volume you're probably not getting any price lower than $5-7/GB. Fortunately Twitter doesn't rate limit by IP (at least as long as you're not posting), they allow guest access and they have an endpoint that lets you get 100 tweets at a time so it's not very difficult to scrape as I described in https://news.ycombinator.com/item?id=36667225.

I dont know what the hell Twitter is doing, but I simply dont use it anymore. I simply cannot access this page without logging in:

https://twitter.com/SarahKSilverman

I check it ONCE per day, thats it. and for weeks every time it just prompts me to login. no thanks, done with Twitter.

You can use nitter to access Twitter content anonymously at the moment:

https://nitter.net/SarahKSilverman