News for hackers. Stuff that matters.
Re:Perplexity AI is lying about their user agent by gcanyon on Saturday June 15, 2024 @03:14PM (#40692016)
It seems self-evident to me that if a user tells a bot to go get a web page, robots.txt doesn't apply, and the bot shouldn't respect it. I understand others' concerns that, like Apple's reader, and other similar tools, it's ethically debatable whether a site should be required to comply with the request, and spoofing an agent seems in dubious territory. I don't think a good answer has been proposed for this challenge, unfortunately.
Re:Perplexity AI is lying about their user agent by lolinder on Saturday June 15, 2024 @03:25PM (#40692081)
> spoofing an agent seems in dubious territory.

Just to clarify, Perplexity is not spoofing a user agent, they're legitimately using a headless Chrome to fetch the page.

The author just misunderstood their docs [0]: when they say that "you can identify our web crawler by its user agent", they're talking about the crawler, not the browser they use for ad hoc requests. As you note, crawling is different.

[0] https://docs.perplexity.ai/docs/perplexitybot

Re:Perplexity AI is lying about their user agent by mattigames on Saturday June 15, 2024 @03:34PM (#40692148)
This is completely false, the user agent being used by Perplexity its _not_ the headless-chrome user agent, wich is close similar to this (emphasis on HeadlessChrome):

    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/119.0.0.0 Safari/537.36
They are spoofing it to pretend to be a desktop Chrome one:

    Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36
Re:Perplexity AI is lying about their user agent by lolinder on Saturday June 15, 2024 @03:40PM (#40692198)
Ah, you're correct, my bad.

I don't personally have a problem with spoofing user agents, but yeah, they're either spoofing or for some reason they're truly using a non-headless Chrome.

Re:Perplexity AI is lying about their user agent by jcrawfordor on Saturday June 15, 2024 @06:56PM (#40693316)
There's a difference here between "headless chrome" as a concept and "headless-chrome" the software. It's still pretty common to run browser automation with a full "headful" browser, in which case you would just get the normal user agent. headless-chrome is sort of an optimized option that comes with some downsides.