Re:Nepenthes is a tarpit to catch AI web crawlers
by jsheard on Thursday January 16, 2025 @11:43AM
(#42727632)
> No, put up a loginwall or paywall, authenticate users, and go private.
We know for a fact that AI companies don't respect that, if they want data that's behind a paywall then they will jump through hoops to take it anyway.
https://www.theguardian.com/technology/2025/jan/10/mark-zuck...
If they don't have to abide by "norms" then we don't have to for their sake. Fuck 'em.
Re:Nepenthes is a tarpit to catch AI web crawlers
by observationist on Thursday January 16, 2025 @11:52AM
(#42727731)
They have to follow the law, and the law explicitly allows scraping and crawling. Norms aren't enough; to get where most people seem to want to be, we need legislation that effectively makes robots.txt as legally binding and valid as a No Trespassing sign. There are significant consequences to the internet and public discourse if that happens, and it's not all good.
Re:Nepenthes is a tarpit to catch AI web crawlers
by blibble on Thursday January 16, 2025 @11:58AM
(#42727804)
this is a very US-ian view of the world
my site is not in the US, I am not a US citizen. US law does not apply to me.
under UK law: robots.txt is an access control mechanism (weak or otherwise)
knowingly bypassing it is likely a criminal offence under the Computer Misuse Act
good luck suing me because you got stuck when you smashed my window and climbed through it
If you want ToS or rules of conduct or any other constraints on interactions with your server or content, you have to engage the user in a legally binding contract. The technical mechanisms to do so require authenticating the user, requiring identification and segregation of individuals, each requiring separate, documented consent and agreement. Once there's a login, or unique session, you can impose whatever conditions you want, and you should.
Interacting in public comes with certain expectations and limitations. This should be communicated more clearly; the general public should know what it means to publish to a platform open to the public, for example.