Re:AI crawlers haven't learned to play nice with websites
by CraigRood on Tuesday March 18, 2025 @12:04PM
(#43401047)
I've personally noticed loads of crawls too, but what has been interesting is they have been crawling effectively the same pages, over and over, for months.
Also, they don't seem to be caching DNS at all.
I guess I'm in the minority that I don't care too much about crawlers, even the use of my work for AI, but I think the current hit rates just can't be justified.
I guess I'm in the minority that I don't care too much about crawlers, even the use of my work for AI, but I think the current hit rates just can't be justified.