Which is good. That's a desirable property. The distinction isn't available without also allowing fingerprinting. Something like cost-based attestation where humans and bots are treated equally is ideal.
> And before somebody makes the argument of "but that's centralised, big brother, blah blah whatever bullshit", let me remind you that every payment you make goes through either Mastercard or Visa.
That's an even bigger problem!
Is it? That's Cloudflare's whole selling point - keep the bots out. I can understand from a hacker perspective wanting bots to be able to roam the Internet as freely as people but that causes massive headaches for sysadmins, SREs, and DevOps. robots.txt is no good because it's opt-in.
Since we don't have chips that detect one's heart's intentions yet, the best we can do is treat bots and user agents the same, and address the problem of malicious activity in other ways. This can be rate limiting, paying per request (i.e. hashcash) or other mechanisms I don't have top of mind. But bot=deny and user=allow is not what Cloudflare does or seeks to do.
All of this comes from there being no universal way to prove you are a human on the Internet. If somebody were to invent a physical device (think YubiKey) that atttested that your activity is human without it being usable to identify/track you, we might have a shot at solving this without CAPTCHAs.
The device would be issued to you as an individual and any signs of it being abused could be reported to deactivate it. I have no idea how such a device would work, but I'm sure it's possible. With machine learning becoming more powerful, this is going to be needed one day.
And before somebody makes the argument of "but that's centralised, big brother, blah blah whatever bullshit", let me remind you that every payment you make goes through either Mastercard or Visa.