News for hackers. Stuff that matters.
Re:How we blocked TikTok's Bytespider bot and cut our bandwi... by braden_e on Wednesday May 22, 2024 @02:04PM (#40444026)
This is the worst behaved bot I have ever seen, I suspect it is AI related. I recently decided to block all the AI crawlers - unlike search engines I get nothing from them.
Re:How we blocked TikTok's Bytespider bot and cut our bandwi... by chptung on Wednesday May 22, 2024 @02:05PM (#40444049)
Same! Any chance you can share a list of the bots you're blocking so I can add them too?
Re:How we blocked TikTok's Bytespider bot and cut our bandwi... by braden_e on Wednesday May 22, 2024 @02:08PM (#40444088)
ClaudeBot (second worst crawler) and GPTBot are the only ones that identify themselves in an obvious way. The rest I am blocking by network. I assume AI when the crawler is very aggressively downloading images - it must be costing them an absolute fortune!
Re:How we blocked TikTok's Bytespider bot and cut our bandwi... by chptung on Wednesday May 22, 2024 @02:15PM (#40444171)
Thanks for these!