it could be RAM-bound, which is very much NOT cheap nowadays :)
Even on a low-specced machine you can afford to have the currently-active website tab consume a few hundreds of megabytes of RAM. It was mostly sitting idle anyways, so most people aren't even going to notice. Multiply that by a few thousand concurrent visitors and you're burning hundreds of gigabytes without anyone caring.
Some scraper being forced to install hundreds of gigabytes of extra RAM in their crawling node? They will notice having to go from a $0.50 / h instance to a $15.00 / h one, solely for the extra RAM requirement.
100MB for 1 second just is not much of a deterrent.
On the other hand, the additional CPU usage rises the compute cost of scraping by several orders of magnitude. If you don't have to scrape this specific website, you'd be stupid not to move on and hit someone else.
Ideally the general cost of scraping would be high enough that it isn't affordable any more - especially anonymized - but considering the amount of money brainlessly being pumped into AI I doubt that'll happen any time soon.
You could of course also make the argument that the user's time is worth something as well and should be included as part of the cost, but that ship sailed a loooong time ago. If you care about that, you should be calling for the death of client-side Javascript and any form of advertising.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.