News for hackers. Stuff that matters.
Re:Creepy Crawlies by smallerize on Sunday August 30, 2026 @12:27PM (#49500112)
But the scraper is making way more requests and is paying for all that compute.
Re:Creepy Crawlies by tptacek on Sunday August 30, 2026 @12:44PM (#49500274)
This is the basic misconception behind the Anubis design. In an anti-spam system, there's a user/abuser reward asymmetry. Users make single requests and get some constant reward V. Abusers make millions of requests and also get some constant reward V. Proof-of-work exploits the asymmetry: costs are, by a factor of ~millions, disproportionately imposed on abusers.

That's exactly not what happens with scrapers. Scrapers make millions of requests, yes. But each one of those requests produces an average (token-denominated) reward V'. There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers.

That's before you even get to questions like "does the system actually work, or does a single solve get practically unbounded rewards".

Re:Creepy Crawlies by progval on Sunday August 30, 2026 @01:41PM (#49500916)
> There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers.

There is: users solve the PoW once, scrappers solve it every time they rotate IP addresses. Some of them change their IP address on every request, so they keep solving the PoW over and over.

Re:Creepy Crawlies by sunoforbrazilia on Sunday August 30, 2026 @04:15PM (#49502324)
"scrappers" opinion discarded
Re:Creepy Crawlies by tptacek on Sunday August 30, 2026 @05:09PM (#49502846)
That doesn't matter! Every time they successfully solve the challenge, they're getting the token reward. That's the missing asymmetry. In an anti-spam system, 99.999% of all transactions are unrewarded.
Re:Creepy Crawlies by graemep on Sunday August 30, 2026 @12:52PM (#49500357)
I wonder whether what Anubis is doing might be blocking agents that do not run JS or that have a timeout (or humans who give up as I just did)
Re:Creepy Crawlies by tptacek on Sunday August 30, 2026 @01:10PM (#49500558)
I suspect that's most of the value, and that the deterrent is about software construction and not about the proof-of-work, which I've been saying for the last year is a productive direction for Anubis to head towards. Instead, it's just sort of tormenting normal people, as a kind of performance of anger about AI.

Which: I get! If it's your site, perform all the anger you want.

Re:Creepy Crawlies by cyberax on Sunday August 30, 2026 @03:09PM (#49501793)
I have a small personal site with an exposed Git repository. I'm getting crawled from about 100000 residential IPs from several ASNs (mostly in Brazil). Each crawler makes a couple of requests per minute.

Anubis doesn't really help against this.

Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @02:00PM (#49512718)
This, blocking granularity is a country, not ip address.
Re:Creepy Crawlies by DarmokTanagra on Sunday August 30, 2026 @02:22PM (#49501343)
not if its running on someone else's device
Re:Creepy Crawlies by semiquaver on Sunday August 30, 2026 @12:32PM (#49500160)
no, a single solve returns a token which entitles the scraper to access the site as much as it wants for one week.
Re:Creepy Crawlies by NobodyNada on Sunday August 30, 2026 @12:40PM (#49500226)
The token should only be valid for one IP, so the scraper has to keep solving challenges if it rotates IPs.
Re:Creepy Crawlies by semiquaver on Sunday August 30, 2026 @12:48PM (#49500320)
OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource.

Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a site.

a $5000 ASIC-based device can hash SHA256 at 200TH/s or more.

Re:Creepy Crawlies by inigyou on Sunday August 30, 2026 @02:14PM (#49501248)
The way they're internally implemented doesn't allow pinning an IP. They buy a rotating proxy service from a vendor, and don't get to choose their source IP.
Re:Creepy Crawlies by semiquaver on Sunday August 30, 2026 @07:39PM (#49503938)
Why are you and other defenders of the Anubis approach so fixated on this one specific limitation of a certain type of scraping architecture? It’s hardly an immutable characteristic.

You say “they” as if all scrapers are a monolithic group with the same constraints and goals. Part of the problem is the massive diversity.

Re:Creepy Crawlies by inigyou on Monday August 31, 2026 @05:51AM (#49507767)
Why do you think anyone thinks it's immutable? Do you understand the basic concept of an arms race, or cat and mouse game?
Re:Creepy Crawlies by rstuart4133 on Sunday August 30, 2026 @10:02PM (#49504805)
It's not hard to test. Go to a page that demands PoW, change your IP and see what happens. I just did it. Spoiler: kernel.org asks for a new PoW.

If the source IP was an issue, you could do it other ways: for example, make the cookie rotate on every access, and insist there is a single stream of accesses.

Re:Creepy Crawlies by dspillett on Sunday August 30, 2026 @02:59PM (#49501705)
> In other words not a meaningful barrier.

And yet many report it works, at least for now, and the excess load due to scraping activity falls precipitously when Anubis or similar solutions are used. Maybe once this sort of challenge is used almost everywhere we'll see concerted attempts to get around it, but for now it is easier for them to just move on to another target.

> a $5000 ASIC-based device can hash SHA256 at 200TH/s or more

Peanuts for the big players, but many (almost all?) running smaller scale scraping operations are going to find $5000 rather prohibitive, and they are unlikely to be able to integrate it as they are probably running a “stock” scraper that they didn't write themselves.

Re:Creepy Crawlies by semiquaver on Sunday August 30, 2026 @08:15PM (#49504217)
You don’t need to spend $5000 to obtain the hash rate of a $5000 device on a rental basis. You may have heard of this thing called “the cloud”. Obtaining very high hash rates is effectively free, largely as a side effect of the crypto bust.

Not sure why anyone would characterize these scrapers im general as all being fly-by-night operations that don’t have two cents to scrape together.

Re:Creepy Crawlies by dspillett on Sunday August 30, 2026 @08:53PM (#49504433)
And yet you have not addressed the point that many report that tricks like Anubis work.

If they are so stupid an idea that they could never work, why do they seem to having the desired effect?

Re:Creepy Crawlies by Dylan16807 on Monday August 31, 2026 @01:20AM (#49505900)
A significant amount of the effect is everything outside the proof of work. Not the cost of hashing but the need to run the javascript that submits it.

If you've seen anyone post a comparison of crawl rate versus difficulty, I'd love to see it. There's probably some difference but I want to know how much of the overall effect it is.

Re:Creepy Crawlies by semiquaver on Monday August 31, 2026 @08:09AM (#49508681)
I never said the approach was so stupid it could never work. I just shared how it’s annoying and how I worked around my annoyance.

An even stupider approach would work exactly as well, like a form saying “type the letter y in this box to continue”. The only benefit is as a road bump. The moment anyone you are defending against so much as looks at the mechanics of solving the challenge it completely falls apart.

Re:Creepy Crawlies by dspillett on Tuesday September 01, 2026 @06:55AM (#49520258)
> I never said the approach was so stupid it could never work.

In that case I have read too deeply between the lines/tone. Sorry for the somewhat accusatory assumption of your opinion's details.

Re:Creepy Crawlies by LatticeAnimal on Sunday August 30, 2026 @01:24PM (#49500709)
Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently.

I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...

Re:Creepy Crawlies by inigyou on Sunday August 30, 2026 @02:15PM (#49501258)
You can do this on yours. Just have the client and server add an extra "2" after the challenge key or something. A different client which extracts the challenge key and does its own processing will only generate invalid responses.
Re:Creepy Crawlies by LatticeAnimal on Sunday August 30, 2026 @04:43PM (#49502583)
Cool, so then that invalidates the ASIC problem, right?

My earlier idea was to imagine that each day Anubis picks an entirely different problem-class. Ex: one day it is Sha256, the next it is prime factorization, the next it is twin-prime-finding, the next it is cracking elliptic curves, the next it is some kind of sorting / information theory problem...

All with the goal of adapting constantly so that scrapers have a harder time optimizing for the PoW problem (i.e. with Sha256 ASICs)

Re:Creepy Crawlies by inigyou on Monday August 31, 2026 @05:50AM (#49507764)
That doesn't work because there is a finite list to choose from.

Nobody is using ASICs on Anubis.

Re:Creepy Crawlies by CursedSilicon on Sunday August 30, 2026 @02:43PM (#49501555)
...That sounds like entropy? As in, the thing computers are bad at (truly random numbers) and /dev/urandom in your kernel already spits out an approximation of?
Re:Creepy Crawlies by xena on Monday August 31, 2026 @07:58AM (#49508585)
> I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...

This is something I'm working on in a prototype. Stay tuned!

Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @06:02PM (#49515372)
hashx kind of already does it.