Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

Re:Perplexity is using stealth, undeclared crawlers to evade... by kldg on Monday August 04, 2025 @08:44PM (#44793081)

if you do analytics, it is not so hard, but then you need to store user data (if not directly, then worse, with a third party), which should be viewed as a liability. I see ~2/3 human traffic, ~1/3 bot traffic (I just parse user agent strings and count whitelisted browsers as human), but my main landing page is all dynamic-populated webgl. I just asked Gemini what it sees on website, and it states "The page appears to be loading, with the text "Loading room data...".[1] There are also labels for "BG", "FG", and "CURSOR", and a background weather animation." -so I can be feel reasonably confident I don't need to worry about AI, for now; it needs a machine-friendly frontend.

you could go proper insanomode, too. remaking The Internet is trivial if you don't care about existing web standards -- replacing HTTP with your own TCP implementation, getting off html/js/css, etc. being greenfield, you can control the protocol, server, and client implementation, and put it in whatever language you want. I made a stateful Internet implementation in Python earlier for proof-of-concept, but I want to port it and expand on it in rust soon (just for fun; I don't do serious biznos). you'll very likely have 100% human traffic then, even if you're the only person curious and trusting enough to run your client.

[ Parent | on HN ]

Re:Perplexity is using stealth, undeclared crawlers to evade... by 71bw on Tuesday August 05, 2025 @02:36AM (#44794939)

  > I made a stateful Internet implementation in Python earlier for proof-of-concept

Is there a repo or some other form of public access? I'd like to see this.

[ Parent | on HN ]

Re:Perplexity is using stealth, undeclared crawlers to evade... by kldg on Tuesday August 05, 2025 @10:51AM (#44798765)

it's not in a shareable state; is unsafe as-is. can share general idea and sample "webpage" files, though.

the server ("lodge") passes JSON to the client from what are called .branch files. the client receives JSON, parses it, then builds the UI and state representation from the JSON, then stored in that client's memory (self.current_doc and self.page_state in python client).

branches can invoke waterwheel (.ww) files hosted on the lodge. waterwheel files on the lodge contain scripts which define how patches (as JSON) are to be sent to the client. the client updates its state based on the JSON patch it receives. sample .branch and .ww from python implementation (in pastebin so to not make everyone have to scroll through this): https://pastebin.com/A0DEZDmR

[ Parent | on HN ]

Re:Perplexity is using stealth, undeclared crawlers to evade... by 71bw on Wednesday August 06, 2025 @02:43AM (#44808508)

I was right to ask, this seems extremely cool. Hit me up via mail [in bio] if you ever end up polishing it enough to share.

[ Parent | on HN ]