News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment

Re:Perplexity is using stealth, undeclared crawlers to evade... by fergie on Tuesday August 05, 2025 @03:08AM (#44795123)
> why would the LLM accessing the website on my behalf be in a different legal category as my Firefox web browser accessing the website on my behalf?

It is illegal to copy stuff from the internet and then make it available from your own servers, especially when those sources have expressly asked you not to do it.

[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by danlitt on Tuesday August 05, 2025 @03:10AM (#44795137)
You need more qualifiers for this to be true. archive.org routinely archives content that the site operators would prefer to be taken down.
[ Parent | on HN ]