News for hackers. Stuff that matters.
Re:Perplexity AI is lying about their user agent by lolinder on Saturday June 15, 2024 @07:32PM (#40693478)
Actually, no, the fact that it's a crawler is the most important fact. The reason why website operators care at all about robots accessing their site (as distinct from humans controlling a browser) is historically one of two reasons:

* The pattern of requests can be very problematic. Impolite crawlers are totally capable of taking down a website by hitting it over and over and over again for hours in a way that humans won't.

* Crawlers are generally used to build search indexes, so instructing them about URLs that would be inappropriate to have show up in a search is relevant.

The behavior that OP is complaining about is that when the user pastes a URL into Perplexity, Perplexity fetches that URL. Neither the traffic pattern nor the persistence profile are remotely similar to typical crawler behavior. As far as I can see there's almost nothing to distinguish it from someone using Edge and then using Edge's built-in summarizer.

There are no comments on this comment yet.