Re:Nepenthes is a tarpit to catch AI web crawlers
by taikahessu on Thursday January 16, 2025 @10:18AM
(#42726409)
Thanks, will look into that!
Re:Nepenthes is a tarpit to catch AI web crawlers
by breakingcups on Thursday January 16, 2025 @03:56PM
(#42730796)
Such as OpenAI, who will ignore robots.txt and change their user agent to evade blocks, apparently[1]
1: https://www.reddit.com/r/selfhosted/comments/1i154h7/openai_...
Re:Nepenthes is a tarpit to catch AI web crawlers
by zcase on Thursday January 16, 2025 @05:50PM
(#42731908)
For those looking, this is the best I've found: https://blog.cloudflare.com/declaring-your-aindependence-blo...
https://github.com/ai-robots-txt/ai.robots.txt
There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.