And this is how you get the entire Internet locked behind login pages.

Abusing free and open leads to people locking down free and open.

I'm not sure that's true. Free and open drives more page views and is a more successful marketing strategy where it makes sense.

"lol" was my first thought as well, and I suspect many peoples, despite the down votes.

For non-commercial projects I always use at least a one second delay, even for large sites. For commercial projects I have actually advocated to clients that they contact the site's owners and ask to arrange a data use agreement. And a couple have really done it and been satisfied by the result which was cleaner data and continuous updates at a reasonable cost.

Seriously, why not? I've never lost anything doing this, you can always scrape (with delay) as Plan B. A little dose of ethics goes a long way!

Yeah, I totally agree with you, but also make sure you're not being an asshole with websites and hammer them to the point of bringing them down. Be respectful of their hardware _and_ take the data you need. No need to be burn everything down.

I certainly wouldn't want to burn it down as long as I still need the data, other than that I couldn't care less.