Re:The Cost of Being Crawled: LLM Bots and Vercel Image API ...
by cratermoon on Tuesday April 15, 2025 @02:04PM
(#43696395)
If they are well-coded,
they don't constantly crawl.
They use and pay attention to headers like ETag,
If-Modified-Since and/or If-None-Match and support conditional requests.
Badly behaving RSS readers on the other hand....
https://rachelbythebay.com/w/2024/05/27/feed/