... And what would you do to find out if an image or piece of text (that you didn't create) has been plagiarised by an AI model? You search for it on a search engine, of course!
Not saying I agree/disagree with the whole "LLMs trained on scraped data is unethical", but this way of thinking seems dangerous.
If companies like Theranos can prop up their value by lying, does that make it ok for Theranos competitors to also lie, as another example?
> To provide the best search experience, we need to collect data. We use web crawlers to gather information from the internet and index it for our search engine.
> You can identify our web crawler by its user agent
To anyone who's familiar with web crawling and indexing, these paragraphs have an obvious meaning: Perplexity has a search engine which needs a crawler which crawls the internet. That crawler can be identified by the User-Agent PerplexityBot and will respect robots.txt.
Separately, if you give Perplexity a specific URL then it will go fetch the contents of that URL with a one-off request. That one-off request does not respect robots.txt any more than curl does, and that's 100% normal and ethical. The one-off request handler isn't PerplexityBot, it's a separate part of the application that's probably just a regular Chrome browser that issues the request.
If you want to block access to a site, stop relying on arbitrary opt-in voluntary things like user agent or robots.txt. Make your site authenticated only, that’s literally the only answer here.