We're talking a custom scraper written for this site and this site only.
Yes, I am expecting the people who spend hours inspecting the source of my site, and then writing a custom scraper for it, to spend 30 seconds reading the T&Cs first.
If you want people to read it put your content behind a sign up with a checkbox.
Seen here: https://kazuar.github.io/scraping-tutorial/
If you're using my data to hijack my traffic, without asking, you could have all the right justifications in the world but you're still a prick. Who knows, maybe your orphanage building app will move me tears once I hear about it and I'll give you free access.
EDIT - I can't reply to your comment below, but FWIW I agree that scraping sites in this manner is unethical. I am merely describing the logic that most scrapers go through for self justification and legal protection.
But whatever. It just saddens me that the internet is a constant "don't be a dick" battle with companies like the scraper guys.
(edit - understood :) )
Incidentally, I'll be shopping later. May I give you $5 for you to drive to London and take me there? I was going to steal your car, but then I figured I'd be a good citizen and demand you provide it to me at your own, prohibitive, loss.
A scraper company, funded by magic money (Knight Foundation grants) and $1m of VC, convinced a (UK) Government department to pay them to scrape our site for some analysis the department wanted. They'd never contacted us, never asked for permission, never asked if we could supply the data. Our company was bumping along at this point and having to lay people off. Income from a nice lucrative Government contract would have kept a couple more people in work.
The scraper company's FAQ was, in my view, full-on unethical:
> "we check the robots.txt file. If the site permits robots in general to scrape their site (NOT just GoogleBot!), then we will do so. We will make no effort to look for other terms and conditions as well."
You will ostentatiously "make no effort to look" for T&Cs in case they prohibit the significant contract you're about to sign with the Government? Whoa.
So how I feel about web scraping is simple: "don't be evil". If you're diverting income or traffic from the original site, don't do it. If you're genuinely adding value, go for it, but be open, be prepared to work with the original site, and be prepared to accede to their wishes.