I stumbled upon this status code last year - had never heard of it before - and I bookmarked it and then forgot about it. Thanks for the reminder.

this is basically what they are doing, but instead of charging actual money they are making visitors spin the CPU ideally in a proof of work problem, which has the same outcome from the crawlers perspective.

This has existed for decades. The proof of CPU work is called "frontend frameworks"

I've talked with tons of publishers and all say the same thing:

"Hey, we'd happily give these companies clean data if they just paid us instead of building these scrapers."

I think there is a psychological aspect that made micropayments never work for humans but machines may be better suited for it.

This is ultimately the answer. If something has value, users should pay for it. We haven't had a good way to do that on the web, so it has resulted in the complete shitshow that most websites are.