AI that answers questions using online sources isn't a web crawler, but a web researchers.
Once we get to the point of this AI running locally, or you just have a browser extension that grabs the data and sends it to the server, what is the difference between what it is doing now and that.
If I use an AI and I give it a specific URL I think the expectation is that it is acting like a user and not like a crawler so the robots.txt is meaningless.
That doesn't mean that perplexity is free of any blame here, if they are getting around paywalls or then using this loophole to gather up the data to train later. That isnt ok. I am also concerned about the reports of them using third party crawlers to get around blocking them.
But, purely on what this seems to be focused on and the example given. I don't see anything wrong with it when it is triggered based on a specific action of the user. And should be the expected behavior IMO.
Headline grabbing articles like this I feel like undermine the valid criticism of AI because it can be used to paint a picture of a vendetta against AI or worse as an example of why these arguments are wrong as a way to discredit large swaths of criticism.
We really need to be more mindefull about how we criticize the real problems with the technology.
There's a distinction that is being missed between:
* Web crawlers automatically accessing pages, such as recursively following links to index them for search engines
* A tool accessing a URL in direct response to a user request
robots.txt is only intended for the former. For instance, archive.is state:
> [Why does archive.is not obey robots.txt?] Because it is not a free-walking crawler, it saves only one page acting as a direct agent of the human user. Such services don't obey robots.txt (e.g. Google Feedfetcher, screenshot- or pdf-making services, isup.me, …)
Perplexity do (as far as I've been able to find) respect robots.txt for their scraping. What the investigations confirmed it was ignored for was users entering a URL to summarise.
There is still a conversation to be had. I think users should be able to browse the web with whatever tool they want to - whether that's a standard browser, a minimal reader mode that doesn't show ads, or a statistical summarisation tool - but it does pose a problem for sites that rely on standard browsers letting them show users advertisements, get them to sign up to mailing lists, collect data, etc. if users decide to choose tools that don't allow that.