News for hackers. Stuff that matters.
Re:Perplexity is using stealth, undeclared crawlers to evade... by thoroughburro on Monday August 04, 2025 @10:14AM (#44786080)
> I think most people would draw a distinction between the two, and would at least agree the latter is more acceptable than the former.

No. I should be able to control which automated retrieval tools can scrape my site, regardless of who commands it.

We can play cat and mouse all day, but I control the content and I will always win: I can just take it down when annoyed badly enough. Then nobody gets the content, and we can all thank upstanding companies like Perplexity for that collapse of trust.

Re:Perplexity is using stealth, undeclared crawlers to evade... by hombre_fatal on Monday August 04, 2025 @10:25AM (#44786256)
Taking down the content because you're annoyed that people are asking questions about it via an LLM interface doesn't seem like you're winning.

It's also a gift to your competitors.

You're certainly free to do it. It's just a really faint example of you being "in control" much less winning over LLM agents: Ok, so the people who cared about your content can't access it anymore because you "got back" at Perplexity, a company who will never notice.

Re:Perplexity is using stealth, undeclared crawlers to evade... by ipaddr on Monday August 04, 2025 @11:35AM (#44787286)
It could be my server keeps going down because of llms agents keep requesting pages from my lyric site. Removing that site allowed other sites to remain up. True story.

Who cares if perplexity will never notice. Or competitors get an advantage. It is a negative for users using perplexity or visiting directly because the content doesn't exist.

That's the world perplexity and others are creating. They will be able to pull anything from the web but nothing will be left.

Re:Perplexity is using stealth, undeclared crawlers to evade... by gkbrk on Monday August 04, 2025 @10:20AM (#44786188)
> Then nobody gets the content, and we can all thank upstanding companies like Perplexity for that collapse of trust.

But they didn't take down the content, you did. When people running websites take down content because people use Firefox with ad-blockers, I don't blame Firefox either, I blame the website.

Re:Perplexity is using stealth, undeclared crawlers to evade... by glenstein on Monday August 04, 2025 @12:00PM (#44787697)
>But they didn't take down the content, you did.

That skips the part about one party's unique role in the abuse of trust.

Re:Perplexity is using stealth, undeclared crawlers to evade... by Bluescreenbuddy on Monday August 04, 2025 @10:23AM (#44786236)
FF isn’t training their money printer with MY data. AI scrapers are
Re:Perplexity is using stealth, undeclared crawlers to evade... by IncreasePosts on Monday August 04, 2025 @10:25AM (#44786264)
You don't win, because presumably you were providing the content for some reason, and forcing yourself to take it down is contrary to whatever reason that was in the first place.
Re:Perplexity is using stealth, undeclared crawlers to evade... by ipaddr on Monday August 04, 2025 @11:36AM (#44787310)
Llms attack certain topics so removing one site will allow the others to live on the same server.
Re:Perplexity is using stealth, undeclared crawlers to evade... by Den_VR on Monday August 04, 2025 @10:26AM (#44786281)
You can limit access, sure: with ACLs, putting content behind login, certificate based mechanisms, and at the end of the day -a power cord-.

But really, controlling which automated retrieval tools are allowed has always been more of a code of honor than a technical control. And that trust you mention has always been broken. For as long as I can remember anyway. Remember LexiBot and AltaVista?