If I know the creator of the page doesn't want his page used by my program I wouldn't do it.
>Discriminating based on User-Agent string is the unethical part.
Not being exploited by an AI company is unethical? Robots.txt is made for telling bot identified by user agent what they are allowed to read.
Specifically it's meant for instructing "automatic clients known as crawlers" [0]. A crawler is defined by MDN as "a program, often called a bot or robot, which systematically browses the Web to collect data from webpages." [1]
As generally understood, wget is not a crawler even though it may be used to build one. Neither is curl. A crawler is a program which systematically browses the web, usually to build a search index.
I see no evidence that Perplexity's crawler is ignoring robots.txt, I only see evidence that when a user does a one-off request for a specific URL then Perplexity uses Chrome to access the site.
Basically, OP is using the wrong tool for the job and complaining when it doesn't work. If he wants to be excluded from Perplexity for one-off requests (as distinct from crawling) he needs to reach out to them, there is no applicable RFC.
[0] https://www.rfc-editor.org/rfc/rfc9309.html
[1] https://developer.mozilla.org/en-US/docs/Glossary/Crawler
Clearly there are many other scenarios, and many that are more muddy, but overall when we get in to the business of trying to force people to consume content in particular ways it's a bit icky in my opinion.
The extreme end result of this is no more open web, just force people to download your app to consume your content. This is happening too and it sucks.
?
> Whether or not being an ass is unethical I'll leave to the philosophers.
The fences were reasonable because the demands were reasonable and both sides understood why they are there and respected these borders.
This peace has been broken, norms are thrown away and people who did this cheered for what they did. Now, the people are fighting back. People were silent because the system was working.
It was akin to mark some doors "authorized personnel only" but leaving them unlocked. People and programs respected these stickers. Now there are people and programs who don't, so people started to reinforce these doors.
It doesn't matter what you prefer. The apples are spoiled now. There's no turning back. The days of peace and harmony are over, thanks to "move fast break things. We're doing something amazing anyway, and we don't no permission!" people. If your use is benign but my filter is preventing that use, you should get mad at the parties who caused this fence to appear. It's not my fault to put a fence to protect myself.
To see the current state of affairs, see this list [0]. I'm very sensitive to ethical issues about training your model with my data without my consent, and selling it to earn monies.
I don't care about how you stretch fair-use. The moment you earn money from your model, it's not fair-use anymore [1].
[0]: https://notes.bayindirh.io/notes/Lists/Discussions+about+Art...
This is the ultimate goal already. We want every netizen (human or machine) to obey the written and unwritten rules and be a good netizen.
> Prompts will be for "content similar to X" and the bots will merely use what it has access to, rendering the fences moot.
Absolutely not. I don't want my content to end in an LLM, period. I don't license it that way, and I don't consent. Humans are always welcome to read it though.
An LLM is an hallucinating parrot anyway, so I don't want my words to be used in that LSD fueled computing chaos.
> And there will always be authors who don't mind their content being monitized or utilized by AI.
Yes, and there will always be authors who do mind their content being monitized [sic] or utilized by AI.
This is life.
The rules dictating what this means change over time as context changes. And this is definitely one or the largest shifts since the advent of the www.
> Absolutely not.
OK, but it seems in that case you'll have to delist from search engines and any other place where an LLM may get a preview of what a link that a human gave it is about. And there's likely a spiral here, as as you remove content references from more access points, fewer humans will come across them leading to fewer LLM-based queries, so you'll definitely get your wish. Just not the way you envisioned.
The LLM may hallucinate at times, but many including myself find the pros outweigh the cons, and will increasingly gravitate toward using such services for an increasing variety of tasks. This is the evolving state of the internet.
>Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
There are at least five lies here.
* It isn't made by Mozilla
* It doesn't use WebKit
* It doesn't use KHTML
* It isn't safari
* That isn't even my version of chrome, presumably it hides the minor/patch versions for privacy reasons.
Lying in your user agent in order to make the internet work is a practice that is almost as old as user agents. Your browser is almost certainly doing it right now to look at this comment.
Twenty years ago, I set up a web proxy on my Linux PC at home to change the User Agent because I was tired of getting popups about my web browser (Opera) not being Mozilla or Internet Explorer. It even contained the text "Shut the F up and follow w3c standards!" at first, until I realised that sites could use that to track me.
Discriminating based on User-Agent string is the unethical part.