News for hackers. Stuff that matters.
Re:Perplexity AI is lying about their user agent by baxtr on Saturday June 15, 2024 @02:25PM (#40691638)
What will happen if:

Website owners decide to stop publishing because it’s not rewarded by a real human visit anymore?

Then perplexity and the like won’t have new information to train their models on and no sites to answer the questions.

I think there is a real content dilemma here at work. The incentives of Google and website owners were more or less aligned.

This is not the case with perplexity.

Re:Perplexity AI is lying about their user agent by lolinder on Saturday June 15, 2024 @02:29PM (#40691673)
What is a "visit"? TFA demonstrates that they got a hit on their site, that's how they got the logs.

Is it necessary to load the JavaScript for it to count as a visit? What if I access the site with noscript?

Or is it only a visit if I see all your recommended content? I usually block those recommendations so that I don't get distracted from the article I actually came to read—is my visit a less legitimate visit than other people's?

What exactly is Perplexity doing here that isn't okay that people don't already do with their local user agents?

Re:Perplexity AI is lying about their user agent by baxtr on Saturday June 15, 2024 @03:53PM (#40692276)
A visit is a human reader.

At the very least they get exposed to your website name.

Notice your product/service if you get lucky.

Become a customer at a later visit.

We are talking about cutting the first step off so that everything which may come afterwards is cut off as well.

Re:Perplexity AI is lying about their user agent by lolinder on Saturday June 15, 2024 @04:17PM (#40692419)
The behavior that TFA is complaining about is that when the user drops a link to a site into Perplexity it is able to summarize the content of that link. This isn't about the discoverability aspect of Perplexity, they're specifically complaining that the ad hoc "summarize this post" requests don't respect robots.txt [0]. That's what I'm arguing in favor of and that's the behavior that TFA is attacking.

[0] Which, incidentally, is entirely normal. robots.txt is for the web crawler that indexes, not for ad hoc requests.

Re:Perplexity AI is lying about their user agent by TeMPOraL on Saturday June 15, 2024 @04:51PM (#40692613)
In other words, content is bait, reward is a captured user whose attention - whose sanity, the finite amount of life - can be wasted or plain used against them.

I'm more than happy to see all the websites with attention economy business models to shut down. Yes, that might be 90% of the Internet. That would be the 90% that is poisonous shit.

Re:Perplexity AI is lying about their user agent by baxtr on Saturday June 15, 2024 @05:05PM (#40692689)
The attention economy will never die. Attention will only shift. From websites to aggregators like perplexity.
Re:Perplexity AI is lying about their user agent by TeMPOraL on Saturday June 15, 2024 @05:46PM (#40692932)
Perplexity isn't playing in the attention economy unless they upsell you, advertise to you, or put any other kind of bullshit between you and your goal. Attention economy is (as the name suggests) about monetizing attention; it does so through friction.
Re:Perplexity AI is lying about their user agent by baxtr on Saturday June 15, 2024 @06:31PM (#40693188)
I didn’t write they would. I said “like”. The next perplexity will show ads.

The attention economy will not die. Because it’s hasn’t for the last 100 years. The profits just shift to where the attention is now.

Re:Perplexity AI is lying about their user agent by TeMPOraL on Saturday June 15, 2024 @06:48PM (#40693276)
Fair enough, I agree with that. Hell, we may not need a next Perplexity, this one may very well enshittify couple years down the line - as it happens to almost any service offered commercially on the Internet. I was just saying it isn't happening now - for the moment, Perplexity has arguably much better moral standing than most of the websites they scrape or allow users to one-off browse.
Re:Perplexity AI is lying about their user agent by Dylan16807 on Saturday June 15, 2024 @07:39PM (#40693503)
There was a human reader on the other side of the summarization feature. And they did get exposed to the website name. Is that not enough? Would it be different if equivalent summarization was being done by a browser extension?
Re:Perplexity AI is lying about their user agent by alt227 on Saturday June 15, 2024 @02:54PM (#40691866)
> TFA demonstrates that they got a hit on their site

Whats stopping perplexity caching this info say for 24 hours, and then redisplaying it to the next few hundred people who request it?

Re:Perplexity AI is lying about their user agent by lolinder on Saturday June 15, 2024 @03:03PM (#40691932)
Then they don't get the extra hits. So is that it—is a "visit" important because of the data that you're able to collect from the visit?

Does this place HN's rampant use of archive.md on the same moral footing as Perplexity?

Re:Perplexity AI is lying about their user agent by denton-scratch on Saturday June 15, 2024 @03:07PM (#40691953)
> What exactly is Perplexity doing here that isn't okay that people don't already do with their local user agents?

It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it.

It's understood that site operators have a right to use the User-Agent to discriminate among visitors; that's why robots.txt is a standard. Crawlers that disrespect the standard have for many years been considered beyond the pale; thieves and snoopers. TFA's complaint is entirely justified.

Re:Perplexity AI is lying about their user agent by lolinder on Saturday June 15, 2024 @03:17PM (#40692028)
> It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it.

First, I'm ignoring the output of Perplexity. I have no reason to believe that they gave the LLM any knowledge about its internal operations, it's just riffing off of what OP is saying.

Second, PerplexityBot is the user agent that they use when crawling and indexing. They never claimed to use that user agent for ad hoc HTTP requests (which are notably not the same as crawling).

Third, I disagree that anyone has an obligation to be honest in their User-Agent. Have you ever looked at Chrome's user agent? They're spoofing just about everyone, as is every browser. Crawlers should respect robots.txt, but I'd be totally content if we just got rid of the User-Agent string entirely.

Re:Perplexity AI is lying about their user agent by denton-scratch on Saturday June 15, 2024 @07:10PM (#40693375)
> (which are notably not the same as crawling)

Is that a distinction without a difference?

I think the robots.txt RFC was addressed specifically to crawlers; so technically "ad hoc" requests generated automatically (i.e. by robots) aren't included. But the distinction operators would like to make is between humans and automata. Whether some automaton is a crawler or not isn't relevant.

Re:Perplexity AI is lying about their user agent by lolinder on Saturday June 15, 2024 @07:32PM (#40693478)
Actually, no, the fact that it's a crawler is the most important fact. The reason why website operators care at all about robots accessing their site (as distinct from humans controlling a browser) is historically one of two reasons:

* The pattern of requests can be very problematic. Impolite crawlers are totally capable of taking down a website by hitting it over and over and over again for hours in a way that humans won't.

* Crawlers are generally used to build search indexes, so instructing them about URLs that would be inappropriate to have show up in a search is relevant.

The behavior that OP is complaining about is that when the user pastes a URL into Perplexity, Perplexity fetches that URL. Neither the traffic pattern nor the persistence profile are remotely similar to typical crawler behavior. As far as I can see there's almost nothing to distinguish it from someone using Edge and then using Edge's built-in summarizer.

Re:Perplexity AI is lying about their user agent by Dylan16807 on Saturday June 15, 2024 @07:42PM (#40693508)
If explicitly telling it to access a URL is an access by automaton, then isn't every web browser load an access by automaton?
Re:Perplexity AI is lying about their user agent by BeefWellington on Sunday June 16, 2024 @03:39AM (#40695368)
The flaw with that example is your web browser isn't between other users and the website, turning 500 views into one.

And if we took the analogy to the other end, one could argue that all crawlers have to be kicked off manually at some point...

The problem is here in reality the differentiation is somewhat more understood.

The honor system web is going away, that's for sure.

Re:Perplexity AI is lying about their user agent by lolinder on Sunday June 16, 2024 @09:41AM (#40697001)
> your web browser isn't between other users and the website, turning 500 views into one.

There are a lot of people making this assumption about the way Perplexity is working, but there is no evidence in TFA that Perplexity is caching its ad hoc requests.

And even if they were, what's left unsaid is why it even would matter if 500 views turned into one. It matters either because of lost ad revenue or lost ability to track the users' behavior. Personally, I'm okay with moving past that phase of the internet's life and look forward to new business models that aren't built around getting large numbers of "views".

Re:Perplexity AI is lying about their user agent by Dylan16807 on Sunday June 16, 2024 @10:51AM (#40697514)
> The flaw with that example is your web browser isn't between other users and the website, turning 500 views into one.

So, a caching proxy? That has its own issues, but it's the opposite of access by automaton. One button press causes less than one access to the server. Though one button press still results in one user view, so it's only reducing loads in some ways.

But also is that happening here?

> And if we took the analogy to the other end, one could argue that all crawlers have to be kicked off manually at some point...

One button press causing a million page loads is access by automaton. The distinction seems pretty simple to me.

Re:Perplexity AI is lying about their user agent by ADeerAppeared on Saturday June 15, 2024 @03:59PM (#40692309)
> I think there is a real content dilemma here at work

It's not really a dilemma.

This is exactly what copyright serves to protect authors from. Perplexity copied the content, and in doing so directly competes with the original work, destroying it's market value and driving the original author out of business. Literally what copyright was invented to prevent.

It's the exact same situation as journalists going after Google & social media embeds of articles, which these sites propagandized as "prohibiting hyperlinking", but the issue has been the embedded (summary of the) content. Which people don't click through, and this is the entire point of those features for platforms like Facebook; Keeping users on facebook and not leaving.

This is why quite a few jurisdictions agreed with the journalists and moved to institute restrictions on such embedding.

By all practical considerations, perplexity is doing the exact same thing and trying to deflect with "we used an AI to paraphrase".

> The incentives of Google and website owners were more or less aligned.

The key difference here is that linking is and always has been fine. Google's Book search feature is fair use because the purpose is to send you to the book you searched for, not substitute the book.

Google's current AI summary feature is effectively the same as Perplexity. People don't click through to the original site, the original site doesn't get ad impressions or other revenue, and is driven out of business.

> What will happen if:

What will happen is what already is happening: Journalists are driven out of business, replaced by AI slop.

And then what? AI needs humans creating original content, especially for things like journalism and fact-finding. It'd be an eternal AI winter, all LLMs doomed to be stuck in 2025.

It's in every AI developer's best interest to halt the likes of Perplexity immediately before they irreparably damage the field of AI.

Re:Perplexity AI is lying about their user agent by skeledrew on Saturday June 15, 2024 @04:23PM (#40692456)
I see no competition. I use Perplexity regularly to give me summaries of articles or to do preliminary research. If I like what I'm seeing, then I go to the source. If a source chooses to block their content because they don't want it to be accessed by AI bots then they reduce even further the chance of me - and increasingly more persons - touching their site at all.
Re:Perplexity AI is lying about their user agent by __loam on Saturday June 15, 2024 @04:33PM (#40692503)
"Let us steal your content or you won't get any traffic" sounds extortionate
Re:Perplexity AI is lying about their user agent by skeledrew on Saturday June 15, 2024 @10:48PM (#40694320)
It is what it is. AI is increasingly being used to make lives easier. Those who choose to isolate from AI choose to isolate from the many using it.
Re:Perplexity AI is lying about their user agent by __loam on Sunday June 16, 2024 @02:30AM (#40695088)
We're burning long term value and the open web for shitty chat bots.
Re:Perplexity AI is lying about their user agent by ADeerAppeared on Sunday June 16, 2024 @06:54AM (#40696115)
You can say that, it doesn't matter. The statistics show that these tools reduce views.

And really, "I'm going to replace my entire news intake with the AI slop even if it's entirely hallucinated lies or propaganda" is perhaps not something you ought to say out loud.

Re:Perplexity AI is lying about their user agent by skeledrew on Sunday June 16, 2024 @05:13PM (#40700161)
Reality is view stats, etc don't matter to most users. All we want is to read/research in peace.

You're missing something crucial. Yes, there may be hallucinations, but that's why there's preference for AI that provides citations; they can be easily checked. And also, in my experience, the summaries are usually decent; the sources which tend to yield pretty broken summaries also tend to intersperse unrelated material (ads, previews to other things, etc) in the content, or they're doing wild transforms with JS for example to create unnecessary eye candy. And coincidentally, I'd rather avoid those sources where possible, so it becomes an even greater win for me as user to have AI prefilter said content. Thus the sources getting those precious views in the end become the ones that respect their users' time and aversion to distractions.

Re:Perplexity AI is lying about their user agent by baxtr on Saturday June 15, 2024 @05:02PM (#40692672)
>And then what? AI needs humans creating original content, especially for things like journalism and fact-finding. It'd be an eternal AI winter, all LLMs doomed to be stuck in 2025.

It's in every AI developer's best interest to halt the likes of Perplexity immediately before they irreparably damage the field of AI.

That’s exactly the problem and we all know that it will happen.

Re:Perplexity AI is lying about their user agent by bko on Saturday June 15, 2024 @02:42PM (#40691772)
How would an LLM training on your writing reduce your reward?

I guess if you're doing it for a living sure, but most content I consume online is created without incentive (social media, blogs, stack overflow).

I write a fair amount and have been for a few years. I like to play with ideas. If an llm learned from my writing and it helped me propagate my ideas, I'd be happy. I lose on social status imaginary internet points but I honestly don't care much for them.

The craziest one is the stack overflow contributors. They write answers for free to help people become better programmers but they're mad an llm will read their suggestions and answer questions that help people become better programmers. I guess they do it for the glory of having their handle next to the answer?

Re:Perplexity AI is lying about their user agent by morgante on Saturday June 15, 2024 @04:15PM (#40692402)
> I guess they do it for the glory of having their handle next to the answer?

Yes, it's hardly surprising that people find upvotes and direct social rewards more exciting than being slurped somewhere into GPT-4's weights.

Re:Perplexity AI is lying about their user agent by TeMPOraL on Saturday June 15, 2024 @05:38PM (#40692899)
But they get to enjoy both the social proof on SO and GPT-4 existing.

It's not like they're getting validation from most readers anyway. People who vote and comment on answers are playing the SO social/karma game and will continue to do so whether GPT-4 exists or not. Conversely, people who'll find answers via an LLM instead of viewing it on SO are people who wouldn't bother logging in to SO, even if they had accounts on it in the first place.

People are complaining about losing the audience they never had.

Re:Perplexity AI is lying about their user agent by int_19h on Saturday June 15, 2024 @03:54PM (#40692280)
Speaking as an SO contributor, I'm perfectly fine with having an LLM read my answers and produce output based on them. What I'm not okay with is said LLM being closed-weight so that its creator can profit off it. When I posted my answers on SO, I did so under CC-BY-SA, and I don't think it's unreasonable for me to expect any derivatives to abide by both the letter and the spirit of this arrangement.
Re:Perplexity AI is lying about their user agent by LikelyABurner on Sunday June 16, 2024 @08:58AM (#40696715)
This hits the nail completely on the head.

If the issue here was "just" training LLMs, like some AI bros want to deflect it to be, the conversation around this topic would be very different, and I would be enthusiastically defending the model trainers.

But that's not this conversation. These are companies that are trying to fold our permissively-license content into weights, close source it, and make themselves the only access point, all while pre-emptively perform regulatory capture with all the right DEI buzzwords so that the open source variants are sufficiently demonized as "alt-right" and "dangerous".

The thing that truly frightens me is that (even here on Hacker News) there is an increasing number of people that have fallen for the DEI FUD and are honestly cheering on the Sam Altmans of the world to control the flow of information.

Re:Perplexity AI is lying about their user agent by ADeerAppeared on Saturday June 15, 2024 @03:48PM (#40692245)
> How would an LLM training on your writing reduce your reward?

Because you're not getting the ad impressions anymore. The harsh reality is that people do not click on to sources, so when sites like Perplexity copy your content, you lose the revenue on that content.

This, in turn, drives all real journalism out of business. And then everyone's screwed, including these AI reposting sites.

Re:Perplexity AI is lying about their user agent by __loam on Saturday June 15, 2024 @04:34PM (#40692513)
It's a literal tragedy of the commons
Re:Perplexity AI is lying about their user agent by noduerme on Saturday June 15, 2024 @02:58PM (#40691900)
I think a concern for people who contribute on Stack Overflow is that an LLM will pollute the water with so many subtly wrong answers that the collective work of answering questions accurately will be overwhelmed by a tsunami of inaccurate LLM-generated answers, more than an army of humans can keep up with checking and debugging (or debunking).
Re:Perplexity AI is lying about their user agent by nradov on Saturday June 15, 2024 @04:25PM (#40692474)
It's nice that people are willing to create content on Stack Overflow so that Prosus NV can make advertising revenue from their free labor. But ultimately only a fool would trust answers from secondary sources like Stack Overflow, Quora, Wikipedia, Hacker News, etc. They can be useful sources to start an investigation but ultimately for anything important you still have to drill down to reliable primary sources. This has always been true, and the rise of LLMs doesn't change anything.

For what it's worth, the Stack Exchange terms of service do prohibit AI generated content. I'm not sure how they actually enforce that, and in practice as the LLMs improve it's going to be almost impossible to reliably detect.

https://meta.stackexchange.com/help/gen-ai-policy

Re:Perplexity AI is lying about their user agent by noduerme on Sunday June 16, 2024 @04:09AM (#40695500)
What is even more helpful than answers on S.O. are the comments. Of course it is only to begin an investigation. But who will want to clarify properly if most of the answers are LLM garbage, too many to keep up with?

It is not simply "nice", or for internet points, to take time to answer other people's questions.

Being able to pass on knowledge is the glue of society and civilization. Cynicism about the value or reason of doing so is not a replacement for a functioning structure to educate people who want to learn or to point them in the right direction.

Re:Perplexity AI is lying about their user agent by nradov on Sunday June 16, 2024 @12:05PM (#40698029)
We managed to pass on knowledge and keep civilization functioning before Stack Overflow existed. We'll be fine without it.
Re:Perplexity AI is lying about their user agent by noduerme on Monday June 17, 2024 @11:20PM (#40713774)
Not specifically Stack Overflow, but I don't think we are adapted to live in an environment where correct/useful knowledge is buried so deep in mountains of AI-generated advice which looks plausible but is in fact wrong.
Re:Perplexity AI is lying about their user agent by alt227 on Saturday June 15, 2024 @02:56PM (#40691882)
> The craziest one is the stack overflow contributors. They write answers for free to help people become better programmers.

In my experience they do it for points and kudos. Having people get your answers from LLMs instead of your answer on SO stops people from engaging with the gamification tools and so users get less points on the site.

Re:Perplexity AI is lying about their user agent by apantel on Saturday June 15, 2024 @05:09PM (#40692717)
Yeah. I don’t think people do much of anything for truly no reward. Most people want to directly impact and be recognized by others.
Re:Perplexity AI is lying about their user agent by nradov on Saturday June 15, 2024 @03:34PM (#40692146)
A lot of the public website content targeted towards consumers is already SEO slop trying to sell you something or maximize ad revenue. If those website owners decide to stop publishing due to lack of real human visits then little of value will be lost. Much of the content with real value for consumers has already moved to sites that require registration (and sometimes payment) for access.

For technical content of value to professionals, much of that is hosted by vendors or industry organizations. Those tend to get their revenue in other ways and don't care about companies scraping their content for AI model training. Like the IETF isn't going to stop publishing new RFCs just because Perplexity uses them.