Anubis: Proof-of-work proxy to prevent AI crawlers

Re:Anubis: Proof-of-work proxy to prevent AI crawlers by avodonosov on Thursday March 20, 2025 @05:25PM (#43429160)

Ideas:

- Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options.

- An easy way for good crawlers (like internet archive) to authenticate themselves. E.g. TLS client side authentication or simply an HTTP request header containing signature for the request (the signature in the header may be based on, for example, on their domain name and the TLS cert for that domain)

[ Parent | on HN ]

Re:Anubis: Proof-of-work proxy to prevent AI crawlers by ronsor on Thursday March 20, 2025 @06:59PM (#43430035)

> Make it generate cryptucurrency, so that the work is not wasted. Either to compensate for server expences hosting the content, or for some noble non-profit cause - all installations would collect the currency to a single account. Wasting the work is worse than these both options.

The last time[0] we did this, everyone had a meltdown and blocked it.

[0] See Coinhive, which conspicuously lacks a Wikipedia page.

[ Parent | on HN ]

Re:Anubis: Proof-of-work proxy to prevent AI crawlers by avodonosov on Thursday March 20, 2025 @08:13PM (#43430549)

This time the site content will not be available if the challenge computation is blocked.

[ Parent | on HN ]

Re:Anubis: Proof-of-work proxy to prevent AI crawlers by wslh on Thursday March 20, 2025 @08:03PM (#43430469)

It is mentioned at <https://en.wikipedia.org/wiki/Cryptojacking>

[ Parent | on HN ]

Re:Anubis: Proof-of-work proxy to prevent AI crawlers by avodonosov on Thursday March 20, 2025 @08:12PM (#43430537)

And the mention links to https://en.m.wikipedia.org/wiki/Coinhive which redirects to https://en.m.wikipedia.org/wiki/Monero#Mining_malware

[ Parent | on HN ]

Re:Anubis: Proof-of-work proxy to prevent AI crawlers by xg15 on Friday March 21, 2025 @11:31AM (#43436845)

Mining crypto can lead to misaligned incentives. Suddenly, people would have motivation to run this thing even on benign requests, etc.

[ Parent | on HN ]

Re:Anubis: Proof-of-work proxy to prevent AI crawlers by Silphendio on Friday March 21, 2025 @02:08PM (#43439109)

There's a German Wikipedia page: https://de.m.wikipedia.org/wiki/Coinhive

[ Parent | on HN ]

Re:Anubis: Proof-of-work proxy to prevent AI crawlers by throwawayEm8UE on Thursday March 20, 2025 @10:48PM (#43431355)

Maybe there should also be a protocol to support paying a small amount of crypto (on the order of 0.01-0.1 cents per request) to bypass the PoW. The bots can still scrape it by paying, but you now price in the externality.

[ Parent | on HN ]

Re:Anubis: Proof-of-work proxy to prevent AI crawlers by avodonosov on Thursday March 20, 2025 @11:16PM (#43431498)

Yes, I was also thinking of this.

"proof of pay" or "proof of a transaction"

An advantage of this would be not wasting electricity for the proof of work computation.

[ Parent | on HN ]

Re:Anubis: Proof-of-work proxy to prevent AI crawlers by avodonosov on Thursday March 20, 2025 @11:40PM (#43431602)

Alternatively to cryptocurrency, some other useful c9mputation. Like contribution to BOINC. Only need to find a way to prove the computation.

[ Parent | on HN ]

Re:Anubis: Proof-of-work proxy to prevent AI crawlers by kebokyo on Friday March 21, 2025 @01:45AM (#43432164)

I’m not sure how you can generate cryptocurrency with just sha256 hashes lol. Plus, there’s already a configuration file that lets you determine what you’re looking out for in terms of scrapers. I need to look at it more myself especially because I plan on using this on my own website soon, but I hope there’s already a good way to whitelist archival sites.

[ Parent | on HN ]