braden_e
Re:AI web crawlers are destroying websites in their never-ending conte...
on Tuesday September 02, 2025 @01:32PM (#45106323)
There is a very large scale crawler that uses random valid user agents and a staggeringly large pool of ips. I first noticed it because a lot of traffic was coming from Brazil and "HostRoyale" (asn 203020). They send only a few requests a day from each ip so rate limiting is not useful.
I run a honeypot that generates urls with the source IP so I am pretty confident it is all one bot, in the past 48 hours I have had over 200,000 ips hit the honeypot.
I am pretty sure this is Bytedance, they occasionally hit these tagged honeypot urls with their normal user agent and their usual .sg datacenter.
Re:How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%
on Wednesday May 22, 2024 @02:08PM (#40444088)
ClaudeBot (second worst crawler) and GPTBot are the only ones that identify themselves in an obvious way. The rest I am blocking by network. I assume AI when the crawler is very aggressively downloading images - it must be costing them an absolute fortune!
Re:How we blocked TikTok's Bytespider bot and cut our bandwidth by 80%
on Wednesday May 22, 2024 @02:04PM (#40444026)
This is the worst behaved bot I have ever seen, I suspect it is AI related. I recently decided to block all the AI crawlers - unlike search engines I get nothing from them.
Re:The Stack Exchange Architecture
on Friday September 30, 2011 @06:11PM (#3058736)
Other than microsoft themselves:
myspace is still fairly huge but probably a mixed stack.
Plentyoffish is up there too - http://highscalability.com/plentyoffish-architecture