Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

Re:Perplexity is using stealth, undeclared crawlers to evade... by 71bw on Tuesday August 05, 2025 @02:36AM (#44794939)

  > I made a stateful Internet implementation in Python earlier for proof-of-concept

Is there a repo or some other form of public access? I'd like to see this.

[ Parent | on HN ]

Re:Perplexity is using stealth, undeclared crawlers to evade... by kldg on Tuesday August 05, 2025 @10:51AM (#44798765)

it's not in a shareable state; is unsafe as-is. can share general idea and sample "webpage" files, though.

the server ("lodge") passes JSON to the client from what are called .branch files. the client receives JSON, parses it, then builds the UI and state representation from the JSON, then stored in that client's memory (self.current_doc and self.page_state in python client).

branches can invoke waterwheel (.ww) files hosted on the lodge. waterwheel files on the lodge contain scripts which define how patches (as JSON) are to be sent to the client. the client updates its state based on the JSON patch it receives. sample .branch and .ww from python implementation (in pastebin so to not make everyone have to scroll through this): https://pastebin.com/A0DEZDmR

[ Parent | on HN ]

Re:Perplexity is using stealth, undeclared crawlers to evade... by 71bw on Wednesday August 06, 2025 @02:43AM (#44808508)

I was right to ask, this seems extremely cool. Hit me up via mail [in bio] if you ever end up polishing it enough to share.

[ Parent | on HN ]