Re:Perplexity is using stealth, undeclared crawlers to evade...
by Spivak on Monday August 04, 2025 @11:45AM
(#44787451)
You don't middle click a bunch of links when doing research? Of all the things to point to I wouldn't have thought "opens a bunch of tabs" to be one of the differentiating behaviors between browsing with Firefox and browsing with an LLM.
Re:Perplexity is using stealth, undeclared crawlers to evade...
by daviddanielng on Thursday August 07, 2025 @01:43AM
(#44820992)
> simultaneously opens 3-8 pages (something a human can't do).
Can't you read?
Re:Perplexity is using stealth, undeclared crawlers to evade...
by fauigerzigerk on Monday August 04, 2025 @02:03PM
(#44789384)
>Doesn't o3 sort of already do this?
ChatGPT probably uses a cache though. Theoretically, the average load on the original sites could be far less than users accessing them directly.
Seems like a reasonable stance would be something like "Following the no crawl directive is especially necessary when navigating websites faster than humans can."
> What if it gets a bit smarter and tried to anticipate what you'll ask and does a bunch of crawling to gather information regularly to try to stay up to date on things (from your machine)?
To be fair, Google Chrome already (somewhat) does this by preloading links it thinks you might click, before you click it.
But your point is still valid. We tolerate it because as website owners, we want our sites to load fast for users. But if we're just serving pages to robots and the data is repackaged to users without citing the original source, then yea... let's rethink that.