News for hackers. Stuff that matters.

A comment on The Cost of Being Crawled: LLM Bots and Vercel Image API Pricing. Parent comment

Re:The Cost of Being Crawled: LLM Bots and Vercel Image API ... by cratermoon on Monday April 14, 2025 @10:01PM (#43688299)
> it clearly was not obvious how to block or challenge AI crawlers

https://xeiaso.net/notes/2025/anubis-works/

[ Parent | on HN ]
Re:The Cost of Being Crawled: LLM Bots and Vercel Image API ... by majorchord on Tuesday April 15, 2025 @06:11PM (#43699053)
Setting the user-agent to curl (and maybe others) completely bypasses anubis.
[ Parent | on HN ]