News for hackers. Stuff that matters.
Re:Creepy Crawlies by Aurornis on Sunday August 30, 2026 @02:03PM (#49501129)
> Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them

Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time.

So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views.

A user who needs to load a single page and a bot that wants to scrape a number of pages may pay the same cost.

The amortized per-view cost is highest for the real user.

Re:Creepy Crawlies by rplnt on Sunday August 30, 2026 @03:51PM (#49502123)
Now you have a session of sorts and can limit the requests for that client, right? They can be fast, just limited in volume - regular user isn't punished.
Re:Creepy Crawlies by JsonCameron on Sunday August 30, 2026 @04:45PM (#49502599)
yep, that's the exact play. Or better fingerprinted & blocked in other ways
Re:Creepy Crawlies by afdbcreid on Sunday August 30, 2026 @10:13PM (#49504861)
So, they can interleave different tokens and get stopless access at normal price?

Yes this needs to be tailored for Anubis, but that was the point: the moment it will become popular enough, scrapers will have no problems to bypass it.

Re:Creepy Crawlies by crote on Monday August 31, 2026 @04:03AM (#49506901)
But the tokens are IP-bound, and as the article mentions a crawler using a single IP is not so difficult to block.

A legitimate user solves it once, shows legitimate-use behavior, and retains their access for thousands of requests. A crawler solves a challenge, is determined to use bot-like behavior after a dozen requests, gets an IP ban, and needs to recycle their IP and start from scratch.

It is indeed worse for fly-by users, but with a website like Linux cgit those basically don't exist. It would be a completely different story for, say, someone's personal blog.