Re:AI web crawlers are destroying websites in their never-en...
by candlemas on Tuesday September 02, 2025 @02:44PM
(#45107318)
My site has also recently been getting massively hit by Brazilian IPs. It lasts for a day or two, even if they are being blocked.
Re:AI web crawlers are destroying websites in their never-en...
by dizlexic on Tuesday September 02, 2025 @02:55PM
(#45107468)
I've written my own bots that do exactly this. My reason was mainly to avoid detection so as part of that I also severely throttled my requests and hit the target at random intervals. In other words, I wasn't trying to abuse them. I just didn't want them to notice me.
TLDR it's trivial to send fake info when you're the one who controls the info.
Re:AI web crawlers are destroying websites in their never-en...
by kjkjadksj on Tuesday September 02, 2025 @01:37PM
(#45106389)
I wonder if you could implement a dummy rate limit? Half the time you are rate limited randomly. A real user will think nothing of it and refresh the page.
Re:AI web crawlers are destroying websites in their never-en...
by ronsor on Tuesday September 02, 2025 @02:30PM
(#45107142)
That will irritate real users half the time while the bots won't care.
Re:AI web crawlers are destroying websites in their never-en...
by kjkjadksj on Wednesday September 03, 2025 @01:45PM
(#45118561)
If they are a real user going on your site in 2025 then they have no alternative they are even interested in. They will blame their ISP and wait.
Meanwhile rate limiting the llm could potentially cost a lot of money in time and compute to people who don’t have our best interests at heart. Seems like a win to me.
I run a honeypot that generates urls with the source IP so I am pretty confident it is all one bot, in the past 48 hours I have had over 200,000 ips hit the honeypot.
I am pretty sure this is Bytedance, they occasionally hit these tagged honeypot urls with their normal user agent and their usual .sg datacenter.