Tor and proxies solve that problem pretty nicely from experience.

Not saying this is right, but if the service provider wants to play cat and mouse then I’m happy to take part.

I don't see how using Tor would make your legal case any stronger

It would prevent the legal case from ever taking place, ideally.

The service provider will just block Tor.

And there are plenty of other proxies to choose from.

My point is, you can’t just put content online and expect to put restrictions on how I send the HTTP requests for it on my side. And if you think you can well I’ll do my best to prove you wrong.

Edit: I totally agree that the data they’re giving me is their property and I merely have a license to use it. I’m not advocating for copyright infringement or anything like that. But if the license allows me to use the content for X purpose, then the way I request the content (whether through a browser or a scraper) shouldn’t matter.

Just because you can do something doesn’t mean you should. Just because you can make an HTTP request doesn’t mean you have a right to do whatever you want with that data. You may not agree with that, but that’s what at least US law outlines.

There are plenty of fair use exemptions to copyright. That said, US law is not the highest of moral standards to aspire to. Especially US copyright law. We've been flaunting it online forever, because it's obviously ridiculous and still hasn't caught up with what's fair and equitable in 30 years.

If you're staying within the boundaries of fair use, you only have a Terms of Service to contend with. What you want to aspire to is meaningless versus what the law of the land is (which is what is enforced). That is my point. I don't disagree that most first world copyright law is overzealous and in a lot of scenarios, unreasonable.

"Whether you like it or not" isn't the legal part, it's the result.

This brings up the delightful distinction between malum prohibitum and malum in se.

That's why it is important to root out corruption from law making, nowadays called "lobbying" - this should be illegal. If you expose an endpoint to the public you can't restrict who or what can consume it. You can do throttling on your side but that's it. Otherwise this is just racism towards machines.

I don't think machines are yet considered a "protected class".

How do you outlaw lobbying?

It probably involves lobbying.

Why are we even arguing about this, is what I want to know. Call a referendum and let the people decide.

Not to mention, this still leaves a hole open as a scraping company can outsource the “scraping” to low-wage data-entry slaves in third-world countries.