yea, it comes across as an extremely entitled mobster take.

heads i win, tails you lose. we own all your content, and you better behave.

i can bet this is incentive-speak.

I don't think people are understanding.

There are norms and expectations for participating with the public internet; on the one hand, there's an expectation of good faith, and you're not being a good neighbor if you DOS a site with your crawler or scraper. On the other hand, it's technically legal to access any authorized content being served by a server, and laws like CFAA, the providers TOS, and other regulations shape the legal regime in which websites operate. In much the same way that there are different legal ramifications for behavior in public and behavior in private, the Supreme Court in the US has distinguished between Gate Up websites and Gate Down websites; in the former, then the technical operation and protocols largely govern the legal framework, while in the Gate Down scenario, the server operator can impose a contract - terms of service - which legally constrains the user to expected use.

You cannot set up a trap, in public, doing something deliberately malicious intended to cause harm, like consuming time, money, compute, etc. The law explicitly forbids that.

You can, however, use something like logins or paywalls, the equivalent of putting up a no trespassing sign, and require people behind the gate agree to things like "no scraping allowed" or whatever other terms and conditions you want to apply, and make any interactions in the Gate Down domain fully consensual and legal. You can deploy this software in such a private context, and that's actually a good thing, just like honeypots and such set up to tangle hackers and waste time.

It'd be just like setting up an obstacle course on a public highway - certain domains are regulated by certain legal frameworks, and the public internet has such a regulatory and technical framework. The price you pay for publishing something to the public is the implicit permission and authorization you grant to anyone or anything that connects over the protocol by which you're serving content.

Platforms conveniently failed to educate users of the consequences of their choice to publish content to the public internet. By choosing a Gate Up model, YouTube, Reddit, Facebook, Twitter, and so on imposed unfettered access within the legal constraints determined by the Supreme Court, technical protocols, CFAA, DMCA, and other factors. hiQ vs LinkedIn , https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn , established the legality of scrapers and crawlers, and determined the legal conditions by which your site is public or private.

Running Nepenthes in public is effectively operating malware, however morally justified it might feel.

Please provide a citation for a law that proscribes me from publically offering a service which consumes time while it is voluntarily engaged with.

Deliberately, maliciously interfering with the normal operation of a system, like the internet, isn't legal. Crawling and scraping are legal. That's the price we pay for having a global, publicly accessible internet.

If you want to protect your content, use the technical mechanisms that are available, like login systems or paywalls, or shared blocklists. You can choose to gatekeep your content, and by doing so, make it unscrapeable, and legally protected.

"Hacking clients" have nothing to do with this. It's people, software, and scrapers connecting to publicly served endpoints. If you want to deploy this behind a gate, in private, have at it. In public, maliciously? No bueno.

He's not interfering with any normal operation of any system. He is offering links. You can follow them or not, entirely at your own discretion. Those links load slowly. You can wait for them to complete or not, entirely at your own discretion.

The crawler's normal operation is not interfered with in any way: the crawler does exactly what it's programmed to do. If its programmers decided it should exhaustively follow links, he's not preventing it from doing that operation.

Legally, at best you'd be looking to warp the concept of attractive nuisance to apply to a crawler. As that legal concept is generally intended to prevent bodily harm to children, however, good luck.

> If you want to protect your content, use the technical mechanisms that are available,

> You can choose to gatekeep your content, and by doing so, make it unscrapeable, and legally protected.

so... robots.txt, which the AI parasites ignore?

> Also, consider that relatively small, cheap llms are able to parse the difference between meaningful content and Markovian jabber such as this software produces.

okay, so it's not damaging, and there you've refuted your entire argument

No, put up a loginwall or paywall, authenticate users, and go private. You can't have the benefits of a public site and try to impose your own terms and conditions. This is why LinkedIn had to go private and require logins; the public internet operates by technical protocols, and by serving content, you expressly grant permission to receive the content, without restriction or qualification. It's a proactive transaction on the part of the server operator. If you want to serve to any old person or system or software on the internet, and your server is configured to do so, you cannot impose arbitrary additional rules after the fact. You expressly consent to the user's interactions with your server by setting it up to serve content to whoever connects.

If you want ToS or rules of conduct or any other constraints on interactions with your server or content, you have to engage the user in a legally binding contract. The technical mechanisms to do so require authenticating the user, requiring identification and segregation of individuals, each requiring separate, documented consent and agreement. Once there's a login, or unique session, you can impose whatever conditions you want, and you should.

Interacting in public comes with certain expectations and limitations. This should be communicated more clearly; the general public should know what it means to publish to a platform open to the public, for example.

> No, put up a loginwall or paywall, authenticate users, and go private.

We know for a fact that AI companies don't respect that, if they want data that's behind a paywall then they will jump through hoops to take it anyway.

https://www.theguardian.com/technology/2025/jan/10/mark-zuck...

If they don't have to abide by "norms" then we don't have to for their sake. Fuck 'em.

They have to follow the law, and the law explicitly allows scraping and crawling. Norms aren't enough; to get where most people seem to want to be, we need legislation that effectively makes robots.txt as legally binding and valid as a No Trespassing sign. There are significant consequences to the internet and public discourse if that happens, and it's not all good.

this is a very US-ian view of the world

my site is not in the US, I am not a US citizen. US law does not apply to me.

under UK law: robots.txt is an access control mechanism (weak or otherwise)

knowingly bypassing it is likely a criminal offence under the Computer Misuse Act

good luck suing me because you got stuck when you smashed my window and climbed through it

>the law explicitly allows scraping and crawling.

Nepenthes also allows scraping and crawling, for as long as you like.

I guess it's an unpopular take but I don't see why it was flagged. It's a good point of discussion.