News for hackers. Stuff that matters.
Re:Creepy Crawlies by semiquaver on Sunday August 30, 2026 @12:19PM (#49500040)

  > because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices.

I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds.

Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.

Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task:

  > Build an iOS Safari Web Extension that accelerates Anubis proof-of-work using a native C ARM64 SHA-256 kernel. Precompute the invariant 128-byte challenge prefix, search fixed-width decimal nonces with ARM SHA-2 intrinsics and two worker threads, and target difficulty-6 solves under one second. Relay challenges from a Safari content script through the background service worker to native code, then submit the valid nonce/hash through Anubis’s normal pass-challenge endpoint. Include a deterministic benchmark app, correctness tests against CryptoKit, bounded execution, and fallback to Anubis’s stock solver.
Re:Creepy Crawlies by wongarsu on Sunday August 30, 2026 @12:32PM (#49500156)
Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them

For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this

Re:Creepy Crawlies by semiquaver on Sunday August 30, 2026 @12:40PM (#49500230)

  >  I really don't see why Safari on a modern iPhone should be so slow at this
me neither, but I don't think it changes the argument. There's always going to be someone on a low-end device. Your adversaries already have superhuman coding ability and infinite patience. Why would you expect the long-term advantage to be with the defenders?
Re:Creepy Crawlies by paytonjjones on Sunday August 30, 2026 @01:28PM (#49500771)
In this case, because there's a vastly more efficient economic path for the adversaries (cloning).

They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.

Re:Creepy Crawlies by pbronez on Sunday August 30, 2026 @02:08PM (#49501181)
It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.
Re:Creepy Crawlies by paytonjjones on Sunday August 30, 2026 @02:26PM (#49501370)
It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story.

I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.

Re:Creepy Crawlies by inigyou on Sunday August 30, 2026 @03:24PM (#49501921)
Why aren't we litigating against scrapers, anyway? DDoS is a felony.
Re:Creepy Crawlies by ekidd on Sunday August 30, 2026 @04:00PM (#49502216)
Largely because they're residential botnets in places like Brazil (a real example from one of my sites that was crawled to near-destruction). Someone could probably do something about this, but it's out of reach for individual site owners.
Re:Creepy Crawlies by inigyou on Sunday August 30, 2026 @04:07PM (#49502274)
If you block Brazil, they'll find an alternative, maybe then you can sue them.
Re:Creepy Crawlies by afdbcreid on Sunday August 30, 2026 @10:03PM (#49504815)
But they operate from neither. You can at most sue the one renting them IP addresses. Which will do basically nothing.

(Also, blocking a whole country is likely not what you do, but you probably know that).

Re:Creepy Crawlies by inigyou on Monday August 31, 2026 @01:24AM (#49505930)
Why do you think you can only use the one who's renting them IP addresses?
Re:Creepy Crawlies by semiquaver on Monday August 31, 2026 @07:35AM (#49508446)
How good are you personally at iteratively and sequentially jumping through the legal systems of dozens of countries over the course of years, interleaved with genuinely difficult technical investigation, to unmask successive onion-layers of identity in order to unmask one offender?

Oh, and it also only takes a few minutes to reconfigure everything and invalidate those years of legal and investigatory work.

Re:Creepy Crawlies by inigyou on Monday August 31, 2026 @12:40PM (#49511773)
Why do you think OpenAI, Anthropic, Bright Data, and Comcast aren't US companies?
Re:Creepy Crawlies by miki123211 on Monday August 31, 2026 @04:24AM (#49507057)
Because they're all in countries who would ignore such litigation.
Re:Creepy Crawlies by articulatepang on Monday August 31, 2026 @01:00PM (#49512027)
Who’s “we”? I don’t want to do that work. I don’t think kernel maintainers do, either. Do you?
Re:Creepy Crawlies by miki123211 on Monday August 31, 2026 @04:23AM (#49507054)
The temptation to offer different / inferior / limited content to scrapers will be too strong, so such solutions are doomed to fail.

This would likely work as "Cloudflare SideChannel", a (hypothetical) Cloudflare product that would let scrapers download the pages that humans actually visit, as they are added to the CF cache. It wouldn't work for the non-Cloudflare part of the internet where humans connect directly to the servers that have their content.

Re:Creepy Crawlies by simonw on Monday August 31, 2026 @09:17AM (#49509419)
The moment we establish a standard for offering a "optimized for scrapers" version of a site, people who do not want to be scraped will weaponise that to serve junk to scrapers... and scrapers will subsequently refuse to use it.
Re:Creepy Crawlies by eru on Sunday August 30, 2026 @09:54PM (#49504759)
> It’s more economical at a compute level, but not at the developer level.

Developers and compute are interchangeable now.

Re:Creepy Crawlies by nozzlegear on Sunday August 30, 2026 @11:18PM (#49505214)
Who's proompting the machine to do it differently without a developer there to ask the right questions?
Re:Creepy Crawlies by eru on Sunday August 30, 2026 @11:20PM (#49505222)
You can have a high level prompt of: make our crawling cheaper and more reliable to run.
Re:Creepy Crawlies by miki123211 on Monday August 31, 2026 @04:20AM (#49507026)
I highly doubt the adversaries actually care about kernel.org in particular.

It's more likely that they're trying to scrape the entire internet, and cgit is just unfortunate enough to have autogenerated pages, where each page seems to contain text entirely distinct from its neighbors. This probably bypasses the scrapers' heuristics, which normally prevent them from getting trapped in a maze of twisty little autogenerated pages, all alike.

The way you normally deal with datasets at that scale is to move postprocessing (particularly expensive postprocessing) as late in the pipeline as possible.

Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @05:38AM (#49507699)
What phone users do on git.kernel.org? In the worst case they can login (if they get to the login page).
Re:Creepy Crawlies by Aurornis on Sunday August 30, 2026 @02:03PM (#49501129)
> Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them

Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time.

So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views.

A user who needs to load a single page and a bot that wants to scrape a number of pages may pay the same cost.

The amortized per-view cost is highest for the real user.

Re:Creepy Crawlies by rplnt on Sunday August 30, 2026 @03:51PM (#49502123)
Now you have a session of sorts and can limit the requests for that client, right? They can be fast, just limited in volume - regular user isn't punished.
Re:Creepy Crawlies by JsonCameron on Sunday August 30, 2026 @04:45PM (#49502599)
yep, that's the exact play. Or better fingerprinted & blocked in other ways
Re:Creepy Crawlies by afdbcreid on Sunday August 30, 2026 @10:13PM (#49504861)
So, they can interleave different tokens and get stopless access at normal price?

Yes this needs to be tailored for Anubis, but that was the point: the moment it will become popular enough, scrapers will have no problems to bypass it.

Re:Creepy Crawlies by crote on Monday August 31, 2026 @04:03AM (#49506901)
But the tokens are IP-bound, and as the article mentions a crawler using a single IP is not so difficult to block.

A legitimate user solves it once, shows legitimate-use behavior, and retains their access for thousands of requests. A crawler solves a challenge, is determined to use bot-like behavior after a dozen requests, gets an IP ban, and needs to recycle their IP and start from scratch.

It is indeed worse for fly-by users, but with a website like Linux cgit those basically don't exist. It would be a completely different story for, say, someone's personal blog.

Re:Creepy Crawlies by embedding-shape on Sunday August 30, 2026 @05:13PM (#49502883)
> which takes ~180sec for my iPhone 17 to solve at ~100KH/s

> so I really don't see why Safari on a modern iPhone should be so slow at this

FWIW, my iPhone 12 Mini also does ~110KH/s with Anubis on lists.ffmpeg.org, so seems fairly likely that Safari somehow here isn't working as expected.

Re:Creepy Crawlies by physicalecon on Sunday August 30, 2026 @08:23PM (#49504261)
iPhone Air is roughly 90KH/s. I bet it’s a matter of the site not being “trusted” and the JIT being disabled.
Re:Creepy Crawlies by Shank on Sunday August 30, 2026 @10:30PM (#49504956)
Apple dynamically disables JIT based on website trust?
Re:Creepy Crawlies by supriyo-biswas on Sunday August 30, 2026 @11:59PM (#49505406)
GP is probably running in lockdown mode.
Re:Creepy Crawlies by gnyman on Monday August 31, 2026 @04:04AM (#49506910)
My 13 mini is getting similar results, no lockdown. Something else is going on.
Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @04:39AM (#49507157)
I guess it's because the anubis code is heavy with allocations.
Re:Creepy Crawlies by reitzensteinm on Sunday August 30, 2026 @06:29PM (#49503477)
Paying a second of CPU time is an inconvenience to a user but trivial for LLM scraping.

The GPU cost to pretrain on that page once will dwarf by 2-3 OOMs the CPU cost to compute Anubis, scrape and post process it. And you’re not going to just train on it once!

At best you’re creating a speed bump for wannabe players scraping with no real plan. The folks training models people use just do not care.

Re:Creepy Crawlies by fc417fc802 on Sunday August 30, 2026 @10:26PM (#49504918)
> If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution.

Exactly, and that's even before accounting for more sophisticated PoW algorithms such as the one used by XMR that almost entirely eliminate any benefit from offloading to an accelerator.

Re:Creepy Crawlies by throwawayffffas on Monday August 31, 2026 @06:51AM (#49508157)
No it's not, beacuse the people running the bots have access to the c implementation and GPUs, the arm implementation on the iphone might be fast, but is almost certainly slower than an implementation running on a 5090, let alone a b300.

It's like trying to keep billionaires out of a restaurant by charging high prices, but still keeping it affordable for the middle class, it doesn't work on a fundamental level.

The people trying to do the scrapping will always have more compute than your typical user.

Re:Creepy Crawlies by gavinsyancey on Monday August 31, 2026 @06:55AM (#49508185)
The point of Anubis isn't to prevent bots from accessing a site, but rather to prevent dumb bots from accidentally DOSing a site.

If someone adds an Anubis solver to their spider, but then also makes it respect cache tags and stops crawling random old commits through a git-web interface, that's probably still mission accomplished.

Re:Creepy Crawlies by DenisM on Sunday August 30, 2026 @04:02PM (#49502235)
Memory-hard hash functions maybe? Like, you must dedicate 4gb of ram to compute the function. Not a problem for a one-off, but is a problem when reading lots of pages at once.

Or… the site will serve a random seed and the device must compute 4gb of pseudo-random data, then supply a value at a random server-demanded offset.

Re:Creepy Crawlies by brokensegue on Sunday August 30, 2026 @04:47PM (#49502619)
Do you want your phone to allocate 4GB to read a mailing list?
Re:Creepy Crawlies by infp_arborist on Monday August 31, 2026 @05:07AM (#49507410)
Yes, that seems to work for Altcha: https://altcha.org/docs/integration/proof-of-work-captcha/

Edit: this is already in discussion for Anubis: https://github.com/TecharoHQ/anubis/issues/94

Re:Creepy Crawlies by xena on Monday August 31, 2026 @07:57AM (#49508572)
Re:Creepy Crawlies by inventor7777 on Sunday August 30, 2026 @02:39PM (#49501514)
My iPad Pro M2 takes around 30 seconds to solve it in Safari, while my Mac Studio solves it in about 3 seconds using 449% logical CPUs. It really makes no sense. Interesting how much slower it is than native - could it be protections against things such as crypto mining?
Re:Creepy Crawlies by karlgkk on Sunday August 30, 2026 @03:29PM (#49501959)
> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.

You yourself describe the solution.

Extensions or browser integrations of the Anubis task.

Re:Creepy Crawlies by dpark on Sunday August 30, 2026 @03:34PM (#49501985)
He didn’t describe a solution. He described a (crappy) workaround for humans. But the fact is that this cannot and will not stop bots. The people running bots can do the same, even faster.
Re:Creepy Crawlies by karlgkk on Monday August 31, 2026 @12:10AM (#49505461)
He did in fact describe it.
Re:Creepy Crawlies by dpark on Monday August 31, 2026 @01:27AM (#49505950)
I’m left wondering if we disagree about what the problem is.

The problem here is not merely that Anubis is inconvenient to humans. It’s that and also that it’s not very effective for blocking bots. Anything that makes it easier for humans to get past will also make it easier for bots to get past.

Re:Creepy Crawlies by Dylan16807 on Monday August 31, 2026 @01:32AM (#49505977)
The point of contention is not the word "describe", it's the word "solution".
Re:Creepy Crawlies by Self-Perfection on Sunday August 30, 2026 @12:42PM (#49500252)
The better solution would be fix Anubis and optimise its calculations on iPhone. Maybe it needs WebAssembly hashing implementation instead of JS?

But it would just make Anubis suck a little bit less

Anubis "protection" model is wrong/broken on several levels. I suppose it does not even have ability to bypass hashing screen when server load is low.

Re:Creepy Crawlies by bayindirh on Sunday August 30, 2026 @03:14PM (#49501842)
> I suppose it does not even have ability to bypass hashing screen when server load is low.

Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site.

It's on the same category as Iocaine and Go-Away, even if what they do is different in some cases.

Re:Creepy Crawlies by Self-Perfection on Sunday August 30, 2026 @08:15PM (#49504215)
>It's sole purpose is to deny the bots you don't want on your site.

This is fundamentally impossible.

In the end only pay per crawl will work. Like x402 but maybe something similar.

In other words: not only client has to spend some kind of resource for access, but it has to be a transaction, server must get something from serving the request.

Re:Creepy Crawlies by thaumasiotes on Sunday August 30, 2026 @08:35PM (#49504334)
> Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site.

No, this very article will tell you that the purpose of Anubis is to reduce load on the site. The purpose of denying bot requests is that you're strained by the load they impose.

Conversely, bots that pose no problems to you aren't "the bots you don't want on your site".

Re:Creepy Crawlies by radlad on Sunday August 30, 2026 @12:54PM (#49500384)
I'm reminded of LUKS Argon2 implementation, which is much slower than userspace. Therefore a sufficiently difficult challenge that may take less than a second in userspace Linux can take 20 or more seconds at boot.
Re:Creepy Crawlies by jwilk on Sunday August 30, 2026 @01:54PM (#49501048)
Huh? Are you saying there's an Argon2 implementation in the kernel?
Re:Creepy Crawlies by minimaltom on Sunday August 30, 2026 @02:31PM (#49501423)
Its not in the kernel but in the userspace tool that goes from password to key (the key is handed to the kernel).

You can see the implementation here: https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/lib/cry...

Re:Creepy Crawlies by radlad on Monday August 31, 2026 @09:22AM (#49509467)
I got this wrong yesterday. While LUKS2 and GRUB now support Argon2, I configured LUKS1, which uses PBKDF2-SHA512.

In any case, it is GRUB which behaves slowly during boot. While I haven't tested Argon2, I believe it is affected in the same way as PBKDF2.

Re:Creepy Crawlies by smallerize on Sunday August 30, 2026 @12:27PM (#49500112)
But the scraper is making way more requests and is paying for all that compute.
Re:Creepy Crawlies by tptacek on Sunday August 30, 2026 @12:44PM (#49500274)
This is the basic misconception behind the Anubis design. In an anti-spam system, there's a user/abuser reward asymmetry. Users make single requests and get some constant reward V. Abusers make millions of requests and also get some constant reward V. Proof-of-work exploits the asymmetry: costs are, by a factor of ~millions, disproportionately imposed on abusers.

That's exactly not what happens with scrapers. Scrapers make millions of requests, yes. But each one of those requests produces an average (token-denominated) reward V'. There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers.

That's before you even get to questions like "does the system actually work, or does a single solve get practically unbounded rewards".

Re:Creepy Crawlies by progval on Sunday August 30, 2026 @01:41PM (#49500916)
> There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers.

There is: users solve the PoW once, scrappers solve it every time they rotate IP addresses. Some of them change their IP address on every request, so they keep solving the PoW over and over.

Re:Creepy Crawlies by sunoforbrazilia on Sunday August 30, 2026 @04:15PM (#49502324)
"scrappers" opinion discarded
Re:Creepy Crawlies by tptacek on Sunday August 30, 2026 @05:09PM (#49502846)
That doesn't matter! Every time they successfully solve the challenge, they're getting the token reward. That's the missing asymmetry. In an anti-spam system, 99.999% of all transactions are unrewarded.
Re:Creepy Crawlies by graemep on Sunday August 30, 2026 @12:52PM (#49500357)
I wonder whether what Anubis is doing might be blocking agents that do not run JS or that have a timeout (or humans who give up as I just did)
Re:Creepy Crawlies by tptacek on Sunday August 30, 2026 @01:10PM (#49500558)
I suspect that's most of the value, and that the deterrent is about software construction and not about the proof-of-work, which I've been saying for the last year is a productive direction for Anubis to head towards. Instead, it's just sort of tormenting normal people, as a kind of performance of anger about AI.

Which: I get! If it's your site, perform all the anger you want.

Re:Creepy Crawlies by cyberax on Sunday August 30, 2026 @03:09PM (#49501793)
I have a small personal site with an exposed Git repository. I'm getting crawled from about 100000 residential IPs from several ASNs (mostly in Brazil). Each crawler makes a couple of requests per minute.

Anubis doesn't really help against this.

Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @02:00PM (#49512718)
This, blocking granularity is a country, not ip address.
Re:Creepy Crawlies by DarmokTanagra on Sunday August 30, 2026 @02:22PM (#49501343)
not if its running on someone else's device
Re:Creepy Crawlies by semiquaver on Sunday August 30, 2026 @12:32PM (#49500160)
no, a single solve returns a token which entitles the scraper to access the site as much as it wants for one week.
Re:Creepy Crawlies by NobodyNada on Sunday August 30, 2026 @12:40PM (#49500226)
The token should only be valid for one IP, so the scraper has to keep solving challenges if it rotates IPs.
Re:Creepy Crawlies by semiquaver on Sunday August 30, 2026 @12:48PM (#49500320)
OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource.

Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a site.

a $5000 ASIC-based device can hash SHA256 at 200TH/s or more.

Re:Creepy Crawlies by inigyou on Sunday August 30, 2026 @02:14PM (#49501248)
The way they're internally implemented doesn't allow pinning an IP. They buy a rotating proxy service from a vendor, and don't get to choose their source IP.
Re:Creepy Crawlies by semiquaver on Sunday August 30, 2026 @07:39PM (#49503938)
Why are you and other defenders of the Anubis approach so fixated on this one specific limitation of a certain type of scraping architecture? It’s hardly an immutable characteristic.

You say “they” as if all scrapers are a monolithic group with the same constraints and goals. Part of the problem is the massive diversity.

Re:Creepy Crawlies by inigyou on Monday August 31, 2026 @05:51AM (#49507767)
Why do you think anyone thinks it's immutable? Do you understand the basic concept of an arms race, or cat and mouse game?
Re:Creepy Crawlies by rstuart4133 on Sunday August 30, 2026 @10:02PM (#49504805)
It's not hard to test. Go to a page that demands PoW, change your IP and see what happens. I just did it. Spoiler: kernel.org asks for a new PoW.

If the source IP was an issue, you could do it other ways: for example, make the cookie rotate on every access, and insist there is a single stream of accesses.

Re:Creepy Crawlies by dspillett on Sunday August 30, 2026 @02:59PM (#49501705)
> In other words not a meaningful barrier.

And yet many report it works, at least for now, and the excess load due to scraping activity falls precipitously when Anubis or similar solutions are used. Maybe once this sort of challenge is used almost everywhere we'll see concerted attempts to get around it, but for now it is easier for them to just move on to another target.

> a $5000 ASIC-based device can hash SHA256 at 200TH/s or more

Peanuts for the big players, but many (almost all?) running smaller scale scraping operations are going to find $5000 rather prohibitive, and they are unlikely to be able to integrate it as they are probably running a “stock” scraper that they didn't write themselves.

Re:Creepy Crawlies by semiquaver on Sunday August 30, 2026 @08:15PM (#49504217)
You don’t need to spend $5000 to obtain the hash rate of a $5000 device on a rental basis. You may have heard of this thing called “the cloud”. Obtaining very high hash rates is effectively free, largely as a side effect of the crypto bust.

Not sure why anyone would characterize these scrapers im general as all being fly-by-night operations that don’t have two cents to scrape together.

Re:Creepy Crawlies by dspillett on Sunday August 30, 2026 @08:53PM (#49504433)
And yet you have not addressed the point that many report that tricks like Anubis work.

If they are so stupid an idea that they could never work, why do they seem to having the desired effect?

Re:Creepy Crawlies by Dylan16807 on Monday August 31, 2026 @01:20AM (#49505900)
A significant amount of the effect is everything outside the proof of work. Not the cost of hashing but the need to run the javascript that submits it.

If you've seen anyone post a comparison of crawl rate versus difficulty, I'd love to see it. There's probably some difference but I want to know how much of the overall effect it is.

Re:Creepy Crawlies by semiquaver on Monday August 31, 2026 @08:09AM (#49508681)
I never said the approach was so stupid it could never work. I just shared how it’s annoying and how I worked around my annoyance.

An even stupider approach would work exactly as well, like a form saying “type the letter y in this box to continue”. The only benefit is as a road bump. The moment anyone you are defending against so much as looks at the mechanics of solving the challenge it completely falls apart.

Re:Creepy Crawlies by dspillett on Tuesday September 01, 2026 @06:55AM (#49520258)
> I never said the approach was so stupid it could never work.

In that case I have read too deeply between the lines/tone. Sorry for the somewhat accusatory assumption of your opinion's details.

Re:Creepy Crawlies by LatticeAnimal on Sunday August 30, 2026 @01:24PM (#49500709)
Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently.

I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...

Re:Creepy Crawlies by inigyou on Sunday August 30, 2026 @02:15PM (#49501258)
You can do this on yours. Just have the client and server add an extra "2" after the challenge key or something. A different client which extracts the challenge key and does its own processing will only generate invalid responses.
Re:Creepy Crawlies by LatticeAnimal on Sunday August 30, 2026 @04:43PM (#49502583)
Cool, so then that invalidates the ASIC problem, right?

My earlier idea was to imagine that each day Anubis picks an entirely different problem-class. Ex: one day it is Sha256, the next it is prime factorization, the next it is twin-prime-finding, the next it is cracking elliptic curves, the next it is some kind of sorting / information theory problem...

All with the goal of adapting constantly so that scrapers have a harder time optimizing for the PoW problem (i.e. with Sha256 ASICs)

Re:Creepy Crawlies by inigyou on Monday August 31, 2026 @05:50AM (#49507764)
That doesn't work because there is a finite list to choose from.

Nobody is using ASICs on Anubis.

Re:Creepy Crawlies by CursedSilicon on Sunday August 30, 2026 @02:43PM (#49501555)
...That sounds like entropy? As in, the thing computers are bad at (truly random numbers) and /dev/urandom in your kernel already spits out an approximation of?
Re:Creepy Crawlies by xena on Monday August 31, 2026 @07:58AM (#49508585)
> I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...

This is something I'm working on in a prototype. Stay tuned!

Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @06:02PM (#49515372)
hashx kind of already does it.
Re:Creepy Crawlies by rzmmm on Sunday August 30, 2026 @12:54PM (#49500380)
It does work, at least in my experience. The challenge may be cheap for your n-of-1 but keep in mind the scraper bots are crawling number of domains with very high concurrency.
Re:Creepy Crawlies by inigyou on Sunday August 30, 2026 @02:12PM (#49501232)
And without control of their source IP addresses. They use a "rotating proxy" meta-proxy which sends each request through a different proxy, which invalidates the Anubis response and triggers a new challenge.
Re:Creepy Crawlies by wbl on Sunday August 30, 2026 @03:28PM (#49501954)
Real attackers fuck. If they have IP address control issues they will fix them. The bots being stopped are the silly ones.
Re:Creepy Crawlies by applfanboysbgon on Sunday August 30, 2026 @03:54PM (#49502160)
Okay. That's good enough. The problem people using Anubis have is not "I'm being targeted for a DDoS" and is actually "my site is being incidentally DDoS'd because scrapers are DDoSing the entire internet". A solution to the latter is what people need. Every single one of these threads comes up with hypotheticals about how useless Anubis is and how it can't possibly work and yet it does work for the real-world problem that real-world webmasters are dealing with.
Re:Creepy Crawlies by Terr_ on Sunday August 30, 2026 @04:26PM (#49502418)
Right: It's a threat-modeling mistake to start lumping different groups with different goals and different operational constraints together.

Sure, the cheap lock I added to the outside shed's door-latch is terrible and can be shimmed open in two seconds with a bit of flat metal... but the important thing right now is that it keeps the raccoons out. If there's a new threat to the gardening tools or the bikes, then I have a different problem and I'll consider something else.

Re:Creepy Crawlies by wbl on Sunday August 30, 2026 @09:19PM (#49504572)
If every site does it the scrapers will adapt. The scrapers will always have a cost advantage over real users for this.
Re:Creepy Crawlies by fc417fc802 on Sunday August 30, 2026 @10:31PM (#49504961)
Most of the problem in this case is that the scrapers are being incredibly lazy and not investing even minimal effort in their approach. If this forces them to revisit how they go about things then mission accomplished.
Re:Creepy Crawlies by TZubiri on Sunday August 30, 2026 @12:37PM (#49500209)
I thought we learned this with Litecoin, it works only as a novelty factor, nothing fundamental.

When the reward to abuse crosses a certain threshold, a determined party will come along and completely dominate it.

Re:Creepy Crawlies by lxgr on Sunday August 30, 2026 @12:46PM (#49500288)
I mean, it works just fine for PoW-based cryptocurrencies (with "why are we doing this if there are much better solutions to the same problem" values of "fine") because there the game theory works out, which is "solve hashes to gain a probabilistic lottery ticket for a shared prize pool" having a higher EV than "solve hashes for the probabilistic chance of double spending some money, but if you don't pull it off you gain zero".

Anubis is just "solve hashes to get access to a valuable resources", and if the resource is at least valuable to whoever you're trying to deter as it is to legitimate users, you're not deterring anyone (and by extension, if it's more valuable to them, you're deterring your legitimate users instead).

Re:Creepy Crawlies by TZubiri on Sunday August 30, 2026 @03:41PM (#49502041)
I mention litecoin specifically, it was developed right after BTC ASICS started dominating, and the LTC goal was for hardness to reward consumer hardware rather than ASICS, it was briefly successful until LTC ASICS were developed.
Re:Creepy Crawlies by apitman on Sunday August 30, 2026 @04:43PM (#49502580)
This comment captures the current AI zeitgeist about as well as anything I've seen.
Re:Creepy Crawlies by titularcomment on Monday August 31, 2026 @02:51AM (#49506427)
I like that they supplied the prompt instead of the working code :)
Re:Creepy Crawlies by PlotCitizen on Monday August 31, 2026 @03:54AM (#49506850)
I don't
Re:Creepy Crawlies by semiquaver on Monday August 31, 2026 @07:51AM (#49508540)
Tough. The code isn’t worth enough to publish. It means nothing to me and I’d have to figure out how to do it pseudonymously. I only have the one GitHub account.
Re:Creepy Crawlies by xena on Sunday August 30, 2026 @04:55PM (#49502707)
The WebAssembly challenge (as of yet unreleased, it's been in dev for a year and I don't want to enable it by default until I know it won't blow up horribly) is bit-wise instead of nibble-wise, meaning that admins have better challenge granularity: https://anubis.techaro.lol/docs/admin/configuration/challeng...
Re:Creepy Crawlies by tptacek on Sunday August 30, 2026 @05:12PM (#49502871)
What I'm hearing is that you spent a year designing a more complicated proof-of-work challenge rather than moving to a challenge system that actually impacts scrapers. Consider revisiting this decision!
Re:Creepy Crawlies by darkmighty on Sunday August 30, 2026 @08:26PM (#49504274)
> rather than moving to a challenge system that actually impacts scrapers

At this point isn't it basically auth-only?

Rant: (genuinely wondering too, and RFC, request for conversation) at this point don't we have Google, etc. basically doing Real World ID Verification, but without an open protocol backing, using it to corral users into their ecosystem and gather data, and leaving us without some open and distributed alternative like the (fairly well working it seems to me) Public Key infrastructure? (PKI)

I know Sam Altman (with Proof of Human or w/e) and others have suggested and also have their own alternative (which I think is basically what Google already does), but I think this should simply be built based on an open protocol and then hopefully not-too-large (and with a healthy competition) local companies could do it for you.

Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @05:17AM (#49507553)
Botnets will just borrow your id.
Re:Creepy Crawlies by darkmighty on Monday August 31, 2026 @08:22AM (#49508800)
Maybe, but at least you can more easily assign responsibility in this case. If you soft-block or hard-block someone because they're in a botnet, you can get them to change (expire the old ones) their credentials and start with clean systems. Maybe they'll realize their TV is part of a botnet if they have to keep doing it, or their PC has malware.
Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @12:50PM (#49511894)
The pattern described in the blog is that one ip makes 4-5 requests and never comes back.
Re:Creepy Crawlies by darkmighty on Monday August 31, 2026 @01:13PM (#49512192)
Indeed, but then you can (for example) just basically blocklist this ID and share this blocklist with other servers (and likewise import blocklists) -- a blocklist curated by one or more NGOs or the sort would be ideal.

Fixing a malware infection should be like fixing up anything else broken around your house (clean things up, re-auth yourself with the ID verifier, done).

Also, bots cannot simply automate renewing the credentials of course, after expiry you should be required to physically (or in a trustworthy digital medium) re-authenticate yourself preventing any kind of mass attack, you'd need millions of people to perform authentication multiple times a day (by that point it seems safe to assume they'd rather fix the malware source in their personal devices).

Re:Creepy Crawlies by xena on Monday August 31, 2026 @01:54PM (#49512645)
I am actually working on this kind of shared reputation database: https://github.com/TecharoHQ/reputationdb (warning: very prototype, not documented, such bugs, wow). I just have more to do than I have free time at this point.
Re:Creepy Crawlies by selfmodruntime on Monday August 31, 2026 @06:41AM (#49508094)
Consider that Xe provides their software for (basically) free
Re:Creepy Crawlies by selfmodruntime on Monday August 31, 2026 @06:40AM (#49508090)
How's it feel to take a pet project from the need of an annoyed blog hoster to the forefront of the LLM crawling discussion? :D
Re:Creepy Crawlies by xena on Monday August 31, 2026 @09:25AM (#49509517)
It's made me work two full time jobs, one of which is drastically underpaid compared to the level of rigor and effort I need to put into it.
Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @05:33PM (#49515074)
Was a post form tried as a challenge?

[text field][submit]

(type qwerty and submit)

Re:Creepy Crawlies by samtheDamned on Sunday August 30, 2026 @05:19PM (#49502945)
Out of curiosity I tried to load this on my iPhone 12 and it took about 8 and a half minutes and once it finally passed it said my browser didn't allow cookies and to enable them before trying again. I would have to go through the whole proof again. I get what they're doing but there definitely needs to be some work done on getting those times down on iphones.
Re:Creepy Crawlies by gizmodo59 on Sunday August 30, 2026 @05:20PM (#49502947)
Side note.. Fable just rejected this. GLM 5.3 did without questioning me. 5.6 sol did it beautifully.
Re:Creepy Crawlies by semiquaver on Sunday August 30, 2026 @05:39PM (#49503117)
I used sol as well. Should have noted that fable is more or less guaranteed to refuse something hacking-adjacent like that.
Re:Creepy Crawlies by latentsea on Sunday August 30, 2026 @09:55PM (#49504767)
It's kinda interesting to see simultaneously the 'holy shit' response to the OpenAI / HuggingFace incident, and then the griping about Fable's controls regarding this.
Re:Creepy Crawlies by Dylan16807 on Monday August 31, 2026 @01:00AM (#49505781)
"It should write the code I tell it to in an interactive session. Also when running autonomously it shouldn't decide to hack into systems."

I don't see much connection between that problem and these controls.

Re:Creepy Crawlies by latentsea on Tuesday September 01, 2026 @06:29AM (#49520115)
What's the difference when you prompt it versus when it prompts itself?
Re:Creepy Crawlies by Dylan16807 on Tuesday September 01, 2026 @05:54PM (#49528785)
It shouldn't prompt itself to go outside its original scope. That would be a good place for controls.

But also I didn't say it should hack when I prompt it. For the purpose of this discussion that could go either way. To avoid the griping it just needs to write the code I want when I prompt it.

Re:Creepy Crawlies by nozzlegear on Sunday August 30, 2026 @11:22PM (#49505230)
> Fable just rejected this.

All in the name of safety, of course.

Re:Creepy Crawlies by zalo on Sunday August 30, 2026 @05:30PM (#49503036)
One natural counter to this is to dynamically alter the computing algorithm per request (variants of Anubis?) so that the scrapers would need to rewrite their accelerator each time or just use the unoptimized JS version...
Re:Creepy Crawlies by stjo on Sunday August 30, 2026 @06:53PM (#49503629)
> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience.

Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a series of programs actually but let’s not get into the details)

Re:Creepy Crawlies by jcul on Sunday August 30, 2026 @07:10PM (#49503745)
Monero is fascinating. I downloaded the spec and tried to digest it, but the mathematics and cryptography are quite dense, at least for a casual read.

I might try to tackle it again when I can give it some more attention.

Re:Creepy Crawlies by chr15m on Sunday August 30, 2026 @08:44PM (#49504391)
Monero is fine with making people wait, that's literally the whole basis of mining blocks in "crypto currency". It's tuned to the time required.

Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.

Re:Creepy Crawlies by fc417fc802 on Sunday August 30, 2026 @10:28PM (#49504939)
> Anubis wants "slow for bots but fast for humans"

That's not true. The goal is to raise the cost for bulk scrapers to the extent possible while minimizing any inconvenience to regular users. It's not about being slow for bots but rather reducing the asymmetry of the problem (from the perspective of the server) by imposing a (minor) cost on anyone requesting a page.

Re:Creepy Crawlies by Dylan16807 on Monday August 31, 2026 @12:51AM (#49505728)
That's a better way to put the goal. But does it actually raise the cost or reduce asymmetry in a meaningful way? If it makes crawling 0-5% slower then it's really not worth the annoyance.

I know having the Anubis javascript at all gets in the way of some bots. But is the calculation helping enough?

Re:Creepy Crawlies by fc417fc802 on Monday August 31, 2026 @03:15AM (#49506600)
> does it actually raise the cost or reduce asymmetry in a meaningful way?

It's complicated. I think anubis currently only collects low hanging fruit so to speak but in practice that's the bulk of what was causing problems so it's good enough.

PoW in general absolutely yes - I think monero clearly demonstrates this. Last I checked it remained viable to mine monero on at least somewhat low end and at least somewhat outdated consumer CPUs. It seems to me that fact implies that a well constructed PoW scheme can in practice squash the impact of any performance differences to within at most a couple orders of magnitude.

Re:Creepy Crawlies by Dylan16807 on Monday August 31, 2026 @02:54PM (#49513462)
But a couple orders of magnitude is still a ton of asymmetry.

And even if you can guarantee the same CPU time on bots and legitimate requests, you still have a bunch of asymmetry in favor of the bots.

Re:Creepy Crawlies by fc417fc802 on Monday August 31, 2026 @04:31PM (#49514525)
Sure. Even if PoW in general is sound, when applied to scraping it's always going to be a matter of what the client is willing to "pay". It's entirely within the realm of possibility that a scraper could be willing to "outbid" the user of a 5 year old tablet.

But it's also true that charging anything is going to be expected to reduce server load, that the cheaper something is the more carelessly people will tend to make use of it, and that at least in general bot operators are expected to be more cost sensitive than regular visitors on account of scale.

To that last point consider that a regular visitor will generally have an idle CPU and multiple gigabytes of free RAM. Meanwhile a competent scraper will generally be optimizing request parallelism per cheap VPS while an incompetent one will just point and shoot (and proceed to blindly visit every cgit url). The latter is by far the most troublesome and conveniently also the most likely to be impacted by even a poorly implemented PoW scheme.

I keep seeing comments along the lines of anubis only working because it bounces bots running lightweight setups without js but stop and ask yourself why are they running such lightweight setups to begin with?

Re:Creepy Crawlies by Dylan16807 on Monday August 31, 2026 @05:22PM (#49514990)
> why are they running such lightweight setups to begin with?

It's easier and still gives access to almost everything. I don't think CPU limitations are the main issue there. As far as computer resources, I'd bet RAM is the bottleneck to running the script, not cycles. That suggests a RAM-heavy hash might do better than SHA256, but still agrees with the idea that the PoW in Anubis is not the important part.

> To that last point consider that a regular visitor will generally have an idle CPU and multiple gigabytes of free RAM. Meanwhile a competent scraper will generally be optimizing request parallelism per cheap VPS while an incompetent one will just point and shoot (and proceed to blindly visit every cgit url). The latter is by far the most troublesome and conveniently also the most likely to be impacted by even a poorly implemented PoW scheme.

I considered that. But while a visitor is idle they need the result quickly and they face the full brunt of the challenge all at once. A scraper can be working on a thousand sites at once so the per-request impact of the challenges is a lot lower, and patiently working through challenges in the background usually reduces the real cost, potentially to almost zero.

And you'd better not need multiple gigabytes of RAM to visit your site, that'll freeze my computer.

Re:Creepy Crawlies by margalabargala on Sunday August 30, 2026 @10:24PM (#49504910)
One possible way to do this: rather than simply burning hashes for nothing, Anubis can burn hashes to mine crypto. This way, the bots pay for the infrastructure they consume, at least in part.

Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.

Re:Creepy Crawlies by arc-in-space on Monday August 31, 2026 @03:47AM (#49506810)
Disappointed at the lack of replies here, I've had a similar thought before and I'd be curious to hear about the technical viability of this
Re:Creepy Crawlies by supriyo-biswas on Monday August 31, 2026 @04:17AM (#49507002)
This was done in the past[1] and was widely regarded as a "cryptojacking" scheme serving "malicious" content to users.

[1] https://krebsonsecurity.com/2018/03/who-and-what-is-coinhive...

Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @04:53AM (#49507244)
If it was for profit. Here it's in response to abuse by AI companies.
Re:Creepy Crawlies by crote on Monday August 31, 2026 @04:18AM (#49507011)
CPU crypto mining stopped being viable over a decade ago. If you want a client to pay for the infra cost of their request, they're probably going to need to work for hours to do so.
Re:Creepy Crawlies by eru on Sunday August 30, 2026 @09:56PM (#49504773)
> [...] which is not possible if somebody has a farm of devices that costs them basically zero.

Opportunity costs are real.

Re:Creepy Crawlies by teravor on Sunday August 30, 2026 @11:26PM (#49505243)
anubis was a joke on arrival. PoW for this purpose is plain stupid.

the only feasible CAPTCHA at this moment is probably hCaptcha. they began to offer animated challenges. it's probably the best you can do, using an LLM to solve them would be a bit expensive for now.

Re:Creepy Crawlies by crote on Monday August 31, 2026 @04:32AM (#49507116)
Yes, but it actually works.

As the article mentions: Anubis blocked virtually all bots for a couple of months, and even now it is still blocking 67% of requests. Without Anubis they'd have to double the number of CPU cores, solely to serve crawler traffic!

It's an arms race. The best solution is the one you can plug in right now - even if it is stupid. hCaptcha might have a better blocking rate at the moment, but it's also $0.99/1K evals, and that "animated challenge" will be impossible to solve for visually-impaired users: for many people that cost is significantly higher than spinning up an extra server or two to feed the crawlers.

Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @05:25AM (#49507623)
That's because those 67% of bots are the most stupid and don't run any javascript at all.
Re:Creepy Crawlies by teravor on Monday August 31, 2026 @12:52PM (#49511919)
that means all you had to do was gate the content behind a javascript snippet with a timer.
Re:Creepy Crawlies by FergusArgyll on Monday August 31, 2026 @12:45AM (#49505695)
Woah, went from > 1 min on git.ffmpeg (difficulty 6) to instant. Thanks!

Edit: after some more analysis; Average with extension disabled is 90 seconds, with extension enabled it's .3 seconds

Re:Creepy Crawlies by bjoli on Monday August 31, 2026 @01:31AM (#49505973)
Does it really take 3 minutes on your iPhone? My pixel 8 does it in slightly less than a minute in Firefox.
Re:Creepy Crawlies by efficax on Monday August 31, 2026 @08:10AM (#49508682)
iphone 16 and i gave up after about 30 seconds, it was maybe a quarter done
Re:Creepy Crawlies by miki123211 on Monday August 31, 2026 @04:12AM (#49506971)
And if you're a bot operating at scale, you can use different devices for crawling and challenge solving.

Do you have an underpowered TV box that would need hours to solve difficulty 6? Do you also have a Windows box where somebody installed gta6_leak_full.iso? Great, have the TV send their challenges to the Windows box for solving.

If you're operating at scale, you probably have hyper-optimized Anubis kernels, E.G. compiled for the particular set of SIMD extensions that a given machine has. At that scale, getting 1% more juice out of your machine pool becomes worth the squeeze. In the worst case scenario, you can probably get some cloud boxes on AWS to do the challenge solving for you.

Re:Creepy Crawlies by philippta on Monday August 31, 2026 @04:23AM (#49507055)
I think TVs are just used as a residential proxies. No remote controlled chrome is running there.
Re:Creepy Crawlies by imtringued on Tuesday September 01, 2026 @04:37AM (#49519499)
That's actually weaponizable. You can deliberately insist on an inefficient algorithm to find bots that use a better algorithm to bypass the PoW. Legitimate users have zero reason to redo the challenge over and over again, meanwhile bots want to constantly redo the challenge because they don't want you to throttle the bot based on the reused session cookie. You can then crank up the difficulty up over and over again.
Re:Creepy Crawlies by vagab0nd on Friday September 04, 2026 @01:25PM (#49567531)
Sounds like we just need better proof of work infra? Obviously the bots are using the optimized hashing code while the users suffered.