Or wget to save a set of pages for later.
I understand protecting commenting with captcha, or contact forms. But captcha on regular read-only access to public web pages in the style of Cloudflare is a bit ridiculous.
One thing contact forms should have is a static indication there's a captcha in use. I've filled all too many forms that just sent my written text to void, because I block some domains.
Sadly, there still doesn't seem to be much in the way of micropayment infrastructure.
If they'd like me to pay for access, they should return HTTP 402 Payment Required instead of letting me download the page for free. Perhaps they could also rate limit the network connection to prevent denial of service. Why straight up block automated user agents though? That sucks.
That's illegal.
They must first return HTTP 418 Know Your Customer Required.
Anti Money Laundering regulation killed it: KYC doesn't scale down to micropayment levels.
If you want to fix the web, you have to roll back the AML/KYC insanity. Until that happens, the web will stay broken, because paying with attention (ads) is magically exempt from the AML/KYC insanity, whereas paying with money or anything money-equivalent (fungible and transferable) is not.
Just finished reading about this and I completely agree. I can't imagine having a company and being literally obligated by law to violate everything I personally believe in about privacy and freedom just to help the government be even more efficient at marginalizing people.
Plus, it's not just benign read-only scrapers. Have you looked at the spam folder of your email recently? That's what every comment section and user bio and god knows what else would look like if you just blindly allow all automated traffic.