If AI companies want to sue webmasters for that then by all means, they can waste their money and get laughed out of court.
heads i win, tails you lose. we own all your content, and you better behave.
i can bet this is incentive-speak.
There are norms and expectations for participating with the public internet; on the one hand, there's an expectation of good faith, and you're not being a good neighbor if you DOS a site with your crawler or scraper. On the other hand, it's technically legal to access any authorized content being served by a server, and laws like CFAA, the providers TOS, and other regulations shape the legal regime in which websites operate. In much the same way that there are different legal ramifications for behavior in public and behavior in private, the Supreme Court in the US has distinguished between Gate Up websites and Gate Down websites; in the former, then the technical operation and protocols largely govern the legal framework, while in the Gate Down scenario, the server operator can impose a contract - terms of service - which legally constrains the user to expected use.
You cannot set up a trap, in public, doing something deliberately malicious intended to cause harm, like consuming time, money, compute, etc. The law explicitly forbids that.
You can, however, use something like logins or paywalls, the equivalent of putting up a no trespassing sign, and require people behind the gate agree to things like "no scraping allowed" or whatever other terms and conditions you want to apply, and make any interactions in the Gate Down domain fully consensual and legal. You can deploy this software in such a private context, and that's actually a good thing, just like honeypots and such set up to tangle hackers and waste time.
It'd be just like setting up an obstacle course on a public highway - certain domains are regulated by certain legal frameworks, and the public internet has such a regulatory and technical framework. The price you pay for publishing something to the public is the implicit permission and authorization you grant to anyone or anything that connects over the protocol by which you're serving content.
Platforms conveniently failed to educate users of the consequences of their choice to publish content to the public internet. By choosing a Gate Up model, YouTube, Reddit, Facebook, Twitter, and so on imposed unfettered access within the legal constraints determined by the Supreme Court, technical protocols, CFAA, DMCA, and other factors. hiQ vs LinkedIn , https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn , established the legality of scrapers and crawlers, and determined the legal conditions by which your site is public or private.
Running Nepenthes in public is effectively operating malware, however morally justified it might feel.
If you want to protect your content, use the technical mechanisms that are available, like login systems or paywalls, or shared blocklists. You can choose to gatekeep your content, and by doing so, make it unscrapeable, and legally protected.
"Hacking clients" have nothing to do with this. It's people, software, and scrapers connecting to publicly served endpoints. If you want to deploy this behind a gate, in private, have at it. In public, maliciously? No bueno.
The crawler's normal operation is not interfered with in any way: the crawler does exactly what it's programmed to do. If its programmers decided it should exhaustively follow links, he's not preventing it from doing that operation.
Legally, at best you'd be looking to warp the concept of attractive nuisance to apply to a crawler. As that legal concept is generally intended to prevent bodily harm to children, however, good luck.
> You can choose to gatekeep your content, and by doing so, make it unscrapeable, and legally protected.
so... robots.txt, which the AI parasites ignore?
> Also, consider that relatively small, cheap llms are able to parse the difference between meaningful content and Markovian jabber such as this software produces.
okay, so it's not damaging, and there you've refuted your entire argument
If you want ToS or rules of conduct or any other constraints on interactions with your server or content, you have to engage the user in a legally binding contract. The technical mechanisms to do so require authenticating the user, requiring identification and segregation of individuals, each requiring separate, documented consent and agreement. Once there's a login, or unique session, you can impose whatever conditions you want, and you should.
Interacting in public comes with certain expectations and limitations. This should be communicated more clearly; the general public should know what it means to publish to a platform open to the public, for example.
We know for a fact that AI companies don't respect that, if they want data that's behind a paywall then they will jump through hoops to take it anyway.
https://www.theguardian.com/technology/2025/jan/10/mark-zuck...
If they don't have to abide by "norms" then we don't have to for their sake. Fuck 'em.
my site is not in the US, I am not a US citizen. US law does not apply to me.
under UK law: robots.txt is an access control mechanism (weak or otherwise)
knowingly bypassing it is likely a criminal offence under the Computer Misuse Act
good luck suing me because you got stuck when you smashed my window and climbed through it
Nepenthes also allows scraping and crawling, for as long as you like.
The civics surrounding the rights and responsibilities of website and server operators really should be communicated better; a more informed general public would allow better regulation and legislation to be created.
Robots.txt should have the legal standing of a No Trespassing sign. You should be able to dictate terms to a reasonable degree on a public site; it shouldn't be legal to effectively DOS someone endlessly and repeatedly without recourse. As it stands, though, there's not a lot of protections for little guys. You have to implement a login mechanism, or some way of identifying unique visitors and requiring authentication, in order to impose your own rules on a site.
If I publish content at my domain, I can set up blocklists to refuse access to IP ranges I consider more likely to be malicious than not. Is that not already breaking the social contract you're pointing to wrt serving content public ? picking and choosing which parts of the public will get a response from my server ? (I would also be interested to know if there is actual law vs social contracts around behavior) So why shouldn't I be able enforce expectations on how my server is used? The vigilantism aspect of harming the person breaking the rules is another matter, I'm on the fence.
Consider the standard warning posted to most government sites, which is more or less a "no trespassing sign" [0] informing anyone accessing the system what their expectations should be and what counts as authorized use. I suppose it's not a legally binding contract to say "you agree to these terms by requesting this url" but I'm pretty sure convictions have happened with hackers who did not have a contract with the service provider.
Don't come out and announce that you intend to do an illegal thing, then do said illegal thing. There's not even plausible deniability at that point.
By publishing the software on a public endpoint, you risk running afoul of CFAA. One of the principles by which the internet operates is the express, explicit permission you grant to the public to connect to your server by the mere act of connecting your server to the internet. If you deliberately and maliciously interfere with the normal operation of authorized persons and systems - which is everyone and any software, if you are serving a public endpoint - then a tarpit like this is illegal. You don't get to pick and choose how you think the internet should work; it's got legal and technical requirements. Anything on a public endpoint is provided to the public. You're expected to follow technical protocols - deploying deliberately malicious software, no matter how self righteous you might feel about it, is a great way to get taken to court, lose access to hosting services, get sued, or any of a whole suite of negative legal outcomes.
The internet works because there's an implicit agreement of good faith participation in web protocols. It doesn't matter if it's somebody frustrated with big tech scrapers, or if it's big tech casually laying waste to little people; sometimes the laws are doing a good thing. In this case, it heavily suggests that running something like this is not good, and very likely is illegal, and depending on how much harm your instance causes, it could endanger your freedom.
Under 18 U.S.C. § 1030, amounts over $5,000 in aggregate damage/loss during any 1-year period brings the use of this software to the level of a felony offense. Subsequent offenses are treated as felonies at lower thresholds. Deliberately interfering with the sanctioned, legal, and authorized access to a server you are voluntarily publishing to the internet is not a good idea; you might trip up on a bored lawyer, or if you really get one good and waste a whole lot of someone else's compute, time, and money, they'll be motivated to sic a lawyer on you.
Crawlers shouldn't be expected to filter this. One of the costs of serving content to the public internet is the permission you grant to any and every person or thing that connects and you serve a piece of content.
By publishing to a public endpoint, you are granting permission to crawlers to access your content. Anything that is served by your server to any software is authorized by your having deployed and configured the server to do so.
Robots.txt is a good neighbor mechanism. It'd be great if crawlers respected it, but you aren't entitled to tell anyone they can't use the internet exactly how it was designed to be used. Getting hackernews hug of death, a popularity ddos, scraped by big tech, or any of a dozen other unfortunate things will happen if your site is public.
Also, consider that relatively small, cheap llms are able to parse the difference between meaningful content and Markovian jabber such as this software produces. It's only getting better, and thus far, "poisoning" of datasets is ragebait slop without any technical validity or relevance to the developers. Simply using unfiltered reddit is more damaging than including noise, and AI is getting to the point that noise gets filtered; you'd have to have an AI and architecture comparable to the one you're poisoning deliberately and specifically configured to disrupt the training run, spending as much compute on the disruption as is being spent on the learning.