> I really don't see why Safari on a modern iPhone should be so slow at this
me neither, but I don't think it changes the argument. There's always going to be someone on a low-end device. Your adversaries already have superhuman coding ability and infinite patience. Why would you expect the long-term advantage to be with the defenders?They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.
I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.
(Also, blocking a whole country is likely not what you do, but you probably know that).
Oh, and it also only takes a few minutes to reconfigure everything and invalidate those years of legal and investigatory work.
This would likely work as "Cloudflare SideChannel", a (hypothetical) Cloudflare product that would let scrapers download the pages that humans actually visit, as they are added to the CF cache. It wouldn't work for the non-Cloudflare part of the internet where humans connect directly to the servers that have their content.
Developers and compute are interchangeable now.
It's more likely that they're trying to scrape the entire internet, and cgit is just unfortunate enough to have autogenerated pages, where each page seems to contain text entirely distinct from its neighbors. This probably bypasses the scrapers' heuristics, which normally prevent them from getting trapped in a maze of twisty little autogenerated pages, all alike.
The way you normally deal with datasets at that scale is to move postprocessing (particularly expensive postprocessing) as late in the pipeline as possible.
Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time.
So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views.
A user who needs to load a single page and a bot that wants to scrape a number of pages may pay the same cost.
The amortized per-view cost is highest for the real user.
Yes this needs to be tailored for Anubis, but that was the point: the moment it will become popular enough, scrapers will have no problems to bypass it.
A legitimate user solves it once, shows legitimate-use behavior, and retains their access for thousands of requests. A crawler solves a challenge, is determined to use bot-like behavior after a dozen requests, gets an IP ban, and needs to recycle their IP and start from scratch.
It is indeed worse for fly-by users, but with a website like Linux cgit those basically don't exist. It would be a completely different story for, say, someone's personal blog.
> so I really don't see why Safari on a modern iPhone should be so slow at this
FWIW, my iPhone 12 Mini also does ~110KH/s with Anubis on lists.ffmpeg.org, so seems fairly likely that Safari somehow here isn't working as expected.
The GPU cost to pretrain on that page once will dwarf by 2-3 OOMs the CPU cost to compute Anubis, scrape and post process it. And you’re not going to just train on it once!
At best you’re creating a speed bump for wannabe players scraping with no real plan. The folks training models people use just do not care.
Exactly, and that's even before accounting for more sophisticated PoW algorithms such as the one used by XMR that almost entirely eliminate any benefit from offloading to an accelerator.
It's like trying to keep billionaires out of a restaurant by charging high prices, but still keeping it affordable for the middle class, it doesn't work on a fundamental level.
The people trying to do the scrapping will always have more compute than your typical user.
If someone adds an Anubis solver to their spider, but then also makes it respect cache tags and stops crawling random old commits through a git-web interface, that's probably still mission accomplished.
For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this