I'd be very interested in AI capabilities in detecting anomalous activity given that it should be very effective at pattern matching across large data sets. What would be required, however, is significant training with industry/application specific 'normal' browsing traffic patterns to determine acceptable tolerances, and then design escalating deterrence if and when tolerances are breached.
How is the scraper process stepping through the site? It must be programmatic, even if it's programmed to be random. What does that look like in traffic data in terms of timing and content requests?
What hazards can be put in place that a normal user would never go near, but a scraper is likely to trip over?
If scraping is being performed from multiple locations in parallel, what might that look like? ie. joining multiple sessions together that may show a more 'scraper' kind of behaviour.
Under IPv6 wouldn't detection start burning a lot of addresses? Yes, the pool is large, but that just means it's a scaling issue from IPv4 to IPv6, rather than a new problem requiring new thinking to solve? (assuming ways of detection were successful).
I'm being somewhat devil's advocate in my blocking rhetoric. The internet is in a weird confluence of wild-west and surveillance state. I like that there are ways and means of being anon-/pseudon-ymous; I like a bit of anarchaos, that's where the spirit of hacking lives. But I also don't want it sticking to my boots when I walk in the house. I'm also not fundamentally against scraping; there's definitely good scraping. There is an irony though, in protecting scraped data or services built on scraped data. On residential proxies though, I think they're 99% run by erstwhile criminals leveraging what were until recently referred to as botnets into a service they can launder themselves through as legitimate. I think most, basically all, residential proxies are run involuntarily or unwittingly. I believe blocking them would provide a greater society benefit than it would a drawback. Anything that promotes the sanitisation of end user devices can only be a good thing. The whole ecosystem is in a terrible state.
Maybe I'm hypocritical, or I'm trying to maintain mutually exclusive positions. This is where I'm at though.