Re:Why Adaptive Rate Limiting Is a Game-Changer
by costco on Saturday July 22, 2023 @08:47PM
(#36831479)
https://brightdata.com/proxy-types/residential-proxies gives you access to 70 million IPs. You'd have to be a little insane to scrape millions of tweets like this though because the bandwidth costs would be very high as even with massive volume you're probably not getting any price lower than $5-7/GB. Fortunately Twitter doesn't rate limit by IP (at least as long as you're not posting), they allow guest access and they have an endpoint that lets you get 100 tweets at a time so it's not very difficult to scrape as I described in https://news.ycombinator.com/item?id=36667225.
[ Parent
| on HN ]
There are no comments on this comment yet.