This is the crux. It’s by design. Europe decided to generally legislate pro scraping because it was seen as a way to liberate data from big techs monopoly.

Which is exactly what happened with stable diffusion - say what you will about it, it’s distributing the ability to create value from AI among a larger number of benefactors than OpenAI