News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment

Re:Perplexity is using stealth, undeclared crawlers to evade... by the_real_cher on Monday August 04, 2025 @12:08PM (#44787796)
In the same token the personal shoppers don't owe the store anything either.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by eddythompson80 on Monday August 04, 2025 @12:36PM (#44788178)
Surely they owe them money for the goods and service, no? I thought that's how stores worked.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by the_real_cher on Monday August 04, 2025 @02:12PM (#44789519)
Context friend. This article and entire comments sections is about questionable web page access. Context.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by eddythompson80 on Monday August 04, 2025 @02:51PM (#44789952)
You're replying in a store metaphor thread though. Context matters.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by recursive on Monday August 04, 2025 @01:04PM (#44788577)
Then they can't complain if they're barred entry.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by the_real_cher on Monday August 04, 2025 @02:22PM (#44789609)
http is neutral. it's up to the client to ignore robots.txt

You can block IP's at the host level but there's pretty easy ways around that with proxy networks.

[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by eddythompson80 on Monday August 04, 2025 @03:50PM (#44790622)
> http is neutral.

Who misled you with that statement?

[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by the_real_cher on Monday August 04, 2025 @05:29PM (#44791585)
Http doesnt have emotions or thought last time I checked.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by eddythompson80 on Monday August 04, 2025 @05:40PM (#44791666)
It seems that a 403 makes you sad though.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by the_real_cher on Monday August 04, 2025 @06:47PM (#44792193)
iproyal.com makes me smile again
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by eddythompson80 on Monday August 04, 2025 @07:01PM (#44792312)
And Cloudflare makes you cry. See, it's not neutral. Glad you learned something today. The more one learns everyday, the less stupid you become.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by drdaeman on Monday August 04, 2025 @05:31PM (#44791602)
IETF?
[ Parent | on HN ]