News for hackers. Stuff that matters.
Re:Creepy Crawlies by wongarsu on Sunday August 30, 2026 @12:32PM (#49500156)
Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them

For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this

Re:Creepy Crawlies by semiquaver on Sunday August 30, 2026 @12:40PM (#49500230)

  >  I really don't see why Safari on a modern iPhone should be so slow at this
me neither, but I don't think it changes the argument. There's always going to be someone on a low-end device. Your adversaries already have superhuman coding ability and infinite patience. Why would you expect the long-term advantage to be with the defenders?
Re:Creepy Crawlies by paytonjjones on Sunday August 30, 2026 @01:28PM (#49500771)
In this case, because there's a vastly more efficient economic path for the adversaries (cloning).

They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.

Re:Creepy Crawlies by pbronez on Sunday August 30, 2026 @02:08PM (#49501181)
It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.
Re:Creepy Crawlies by paytonjjones on Sunday August 30, 2026 @02:26PM (#49501370)
It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story.

I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.

Re:Creepy Crawlies by inigyou on Sunday August 30, 2026 @03:24PM (#49501921)
Why aren't we litigating against scrapers, anyway? DDoS is a felony.
Re:Creepy Crawlies by ekidd on Sunday August 30, 2026 @04:00PM (#49502216)
Largely because they're residential botnets in places like Brazil (a real example from one of my sites that was crawled to near-destruction). Someone could probably do something about this, but it's out of reach for individual site owners.
Re:Creepy Crawlies by inigyou on Sunday August 30, 2026 @04:07PM (#49502274)
If you block Brazil, they'll find an alternative, maybe then you can sue them.
Re:Creepy Crawlies by afdbcreid on Sunday August 30, 2026 @10:03PM (#49504815)
But they operate from neither. You can at most sue the one renting them IP addresses. Which will do basically nothing.

(Also, blocking a whole country is likely not what you do, but you probably know that).

Re:Creepy Crawlies by inigyou on Monday August 31, 2026 @01:24AM (#49505930)
Why do you think you can only use the one who's renting them IP addresses?
Re:Creepy Crawlies by semiquaver on Monday August 31, 2026 @07:35AM (#49508446)
How good are you personally at iteratively and sequentially jumping through the legal systems of dozens of countries over the course of years, interleaved with genuinely difficult technical investigation, to unmask successive onion-layers of identity in order to unmask one offender?

Oh, and it also only takes a few minutes to reconfigure everything and invalidate those years of legal and investigatory work.

Re:Creepy Crawlies by inigyou on Monday August 31, 2026 @12:40PM (#49511773)
Why do you think OpenAI, Anthropic, Bright Data, and Comcast aren't US companies?
Re:Creepy Crawlies by miki123211 on Monday August 31, 2026 @04:24AM (#49507057)
Because they're all in countries who would ignore such litigation.
Re:Creepy Crawlies by articulatepang on Monday August 31, 2026 @01:00PM (#49512027)
Who’s “we”? I don’t want to do that work. I don’t think kernel maintainers do, either. Do you?
Re:Creepy Crawlies by miki123211 on Monday August 31, 2026 @04:23AM (#49507054)
The temptation to offer different / inferior / limited content to scrapers will be too strong, so such solutions are doomed to fail.

This would likely work as "Cloudflare SideChannel", a (hypothetical) Cloudflare product that would let scrapers download the pages that humans actually visit, as they are added to the CF cache. It wouldn't work for the non-Cloudflare part of the internet where humans connect directly to the servers that have their content.

Re:Creepy Crawlies by simonw on Monday August 31, 2026 @09:17AM (#49509419)
The moment we establish a standard for offering a "optimized for scrapers" version of a site, people who do not want to be scraped will weaponise that to serve junk to scrapers... and scrapers will subsequently refuse to use it.
Re:Creepy Crawlies by eru on Sunday August 30, 2026 @09:54PM (#49504759)
> It’s more economical at a compute level, but not at the developer level.

Developers and compute are interchangeable now.

Re:Creepy Crawlies by nozzlegear on Sunday August 30, 2026 @11:18PM (#49505214)
Who's proompting the machine to do it differently without a developer there to ask the right questions?
Re:Creepy Crawlies by eru on Sunday August 30, 2026 @11:20PM (#49505222)
You can have a high level prompt of: make our crawling cheaper and more reliable to run.
Re:Creepy Crawlies by miki123211 on Monday August 31, 2026 @04:20AM (#49507026)
I highly doubt the adversaries actually care about kernel.org in particular.

It's more likely that they're trying to scrape the entire internet, and cgit is just unfortunate enough to have autogenerated pages, where each page seems to contain text entirely distinct from its neighbors. This probably bypasses the scrapers' heuristics, which normally prevent them from getting trapped in a maze of twisty little autogenerated pages, all alike.

The way you normally deal with datasets at that scale is to move postprocessing (particularly expensive postprocessing) as late in the pipeline as possible.

Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @05:38AM (#49507699)
What phone users do on git.kernel.org? In the worst case they can login (if they get to the login page).
Re:Creepy Crawlies by Aurornis on Sunday August 30, 2026 @02:03PM (#49501129)
> Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them

Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time.

So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views.

A user who needs to load a single page and a bot that wants to scrape a number of pages may pay the same cost.

The amortized per-view cost is highest for the real user.

Re:Creepy Crawlies by rplnt on Sunday August 30, 2026 @03:51PM (#49502123)
Now you have a session of sorts and can limit the requests for that client, right? They can be fast, just limited in volume - regular user isn't punished.
Re:Creepy Crawlies by JsonCameron on Sunday August 30, 2026 @04:45PM (#49502599)
yep, that's the exact play. Or better fingerprinted & blocked in other ways
Re:Creepy Crawlies by afdbcreid on Sunday August 30, 2026 @10:13PM (#49504861)
So, they can interleave different tokens and get stopless access at normal price?

Yes this needs to be tailored for Anubis, but that was the point: the moment it will become popular enough, scrapers will have no problems to bypass it.

Re:Creepy Crawlies by crote on Monday August 31, 2026 @04:03AM (#49506901)
But the tokens are IP-bound, and as the article mentions a crawler using a single IP is not so difficult to block.

A legitimate user solves it once, shows legitimate-use behavior, and retains their access for thousands of requests. A crawler solves a challenge, is determined to use bot-like behavior after a dozen requests, gets an IP ban, and needs to recycle their IP and start from scratch.

It is indeed worse for fly-by users, but with a website like Linux cgit those basically don't exist. It would be a completely different story for, say, someone's personal blog.

Re:Creepy Crawlies by embedding-shape on Sunday August 30, 2026 @05:13PM (#49502883)
> which takes ~180sec for my iPhone 17 to solve at ~100KH/s

> so I really don't see why Safari on a modern iPhone should be so slow at this

FWIW, my iPhone 12 Mini also does ~110KH/s with Anubis on lists.ffmpeg.org, so seems fairly likely that Safari somehow here isn't working as expected.

Re:Creepy Crawlies by physicalecon on Sunday August 30, 2026 @08:23PM (#49504261)
iPhone Air is roughly 90KH/s. I bet it’s a matter of the site not being “trusted” and the JIT being disabled.
Re:Creepy Crawlies by Shank on Sunday August 30, 2026 @10:30PM (#49504956)
Apple dynamically disables JIT based on website trust?
Re:Creepy Crawlies by supriyo-biswas on Sunday August 30, 2026 @11:59PM (#49505406)
GP is probably running in lockdown mode.
Re:Creepy Crawlies by gnyman on Monday August 31, 2026 @04:04AM (#49506910)
My 13 mini is getting similar results, no lockdown. Something else is going on.
Re:Creepy Crawlies by GoblinSlayer on Monday August 31, 2026 @04:39AM (#49507157)
I guess it's because the anubis code is heavy with allocations.
Re:Creepy Crawlies by reitzensteinm on Sunday August 30, 2026 @06:29PM (#49503477)
Paying a second of CPU time is an inconvenience to a user but trivial for LLM scraping.

The GPU cost to pretrain on that page once will dwarf by 2-3 OOMs the CPU cost to compute Anubis, scrape and post process it. And you’re not going to just train on it once!

At best you’re creating a speed bump for wannabe players scraping with no real plan. The folks training models people use just do not care.

Re:Creepy Crawlies by fc417fc802 on Sunday August 30, 2026 @10:26PM (#49504918)
> If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution.

Exactly, and that's even before accounting for more sophisticated PoW algorithms such as the one used by XMR that almost entirely eliminate any benefit from offloading to an accelerator.

Re:Creepy Crawlies by throwawayffffas on Monday August 31, 2026 @06:51AM (#49508157)
No it's not, beacuse the people running the bots have access to the c implementation and GPUs, the arm implementation on the iphone might be fast, but is almost certainly slower than an implementation running on a 5090, let alone a b300.

It's like trying to keep billionaires out of a restaurant by charging high prices, but still keeping it affordable for the middle class, it doesn't work on a fundamental level.

The people trying to do the scrapping will always have more compute than your typical user.

Re:Creepy Crawlies by gavinsyancey on Monday August 31, 2026 @06:55AM (#49508185)
The point of Anubis isn't to prevent bots from accessing a site, but rather to prevent dumb bots from accidentally DOSing a site.

If someone adds an Anubis solver to their spider, but then also makes it respect cache tags and stops crawling random old commits through a git-web interface, that's probably still mission accomplished.