Re:Confirmed: Twitter switched temporarily to rate limited mode
by CamperBob2 on Saturday July 01, 2023 @04:29PM
(#36554801)
Wouldn't limiting the number of IP addresses that can log in as a single user avoid the botnet problem?
That said, someone mentioned that the login page alone involves 11 MB of data transfer, so maybe that is the problem they are confronted with.
Re:Confirmed: Twitter switched temporarily to rate limited mode
by tlogan on Saturday July 01, 2023 @04:38PM
(#36554929)
The solution is not simple but there is one (but it is complex). The question is always are going to be are you going to spend time and money to implement the right solution or just pick up a cheaper one.
Could it be that they just stop paying bills for Akrose and they shut down their service?
This isn't accurate at all.
Standard operating procedure for large-scale scraping is to use a botnet, it's a sybil attack where you can have tens or even hundreds of thousands of IPs ticking away at 10 requests per minute or some such. Done correctly, it's nearly impossible to detect which nodes are complicit from their access patterns or user agents.
That said, rate limits work relatively poorly against this. A proof of work approach like HashCash might work though.