News for hackers. Stuff that matters.

A comment on Nepenthes is a tarpit to catch AI web crawlers. Parent comment

Re:Nepenthes is a tarpit to catch AI web crawlers by observationist on Thursday January 16, 2025 @11:52AM (#42727731)
They have to follow the law, and the law explicitly allows scraping and crawling. Norms aren't enough; to get where most people seem to want to be, we need legislation that effectively makes robots.txt as legally binding and valid as a No Trespassing sign. There are significant consequences to the internet and public discourse if that happens, and it's not all good.
[ Parent | on HN ]
Re:Nepenthes is a tarpit to catch AI web crawlers by blibble on Thursday January 16, 2025 @11:58AM (#42727804)
this is a very US-ian view of the world

my site is not in the US, I am not a US citizen. US law does not apply to me.

under UK law: robots.txt is an access control mechanism (weak or otherwise)

knowingly bypassing it is likely a criminal offence under the Computer Misuse Act

good luck suing me because you got stuck when you smashed my window and climbed through it

[ Parent | on HN ]
Re:Nepenthes is a tarpit to catch AI web crawlers by tir on Thursday January 16, 2025 @12:26PM (#42728192)
>the law explicitly allows scraping and crawling.

Nepenthes also allows scraping and crawling, for as long as you like.

[ Parent | on HN ]