News for hackers. Stuff that matters.
Re:Nepenthes is a tarpit to catch AI web crawlers by jsheard on Thursday January 16, 2025 @10:15AM (#42726365)
For the "good" bots which at least respect robots.txt you can use this list to get ahead of them before they pummel your site.

https://github.com/ai-robots-txt/ai.robots.txt

There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.

Re:Nepenthes is a tarpit to catch AI web crawlers by taikahessu on Thursday January 16, 2025 @10:18AM (#42726409)
Thanks, will look into that!
Re:Nepenthes is a tarpit to catch AI web crawlers by breakingcups on Thursday January 16, 2025 @03:56PM (#42730796)
Such as OpenAI, who will ignore robots.txt and change their user agent to evade blocks, apparently[1]

1: https://www.reddit.com/r/selfhosted/comments/1i154h7/openai_...

Re:Nepenthes is a tarpit to catch AI web crawlers by zcase on Thursday January 16, 2025 @05:50PM (#42731908)
For those looking, this is the best I've found: https://blog.cloudflare.com/declaring-your-aindependence-blo...
Re:Nepenthes is a tarpit to catch AI web crawlers by maeil on Friday January 17, 2025 @01:57AM (#42734830)
This seemed to work for some time when it came out but IME no longer does.