Re:AI web crawlers are destroying websites in their never-en...
by kjkjadksj on Tuesday September 02, 2025 @01:37PM
(#45106389)
I wonder if you could implement a dummy rate limit? Half the time you are rate limited randomly. A real user will think nothing of it and refresh the page.
[ Parent
| on HN ]
Re:AI web crawlers are destroying websites in their never-en...
by ronsor on Tuesday September 02, 2025 @02:30PM
(#45107142)
That will irritate real users half the time while the bots won't care.
Re:AI web crawlers are destroying websites in their never-en...
by kjkjadksj on Wednesday September 03, 2025 @01:45PM
(#45118561)
If they are a real user going on your site in 2025 then they have no alternative they are even interested in. They will blame their ISP and wait.
Meanwhile rate limiting the llm could potentially cost a lot of money in time and compute to people who don’t have our best interests at heart. Seems like a win to me.