If I buy a map, can I use it to be a guide, giving people directions and make a living?
If I buy a story book, can I use it to read to children and collect money from their parents?
Or, if I spend some time to index all the books in the library, can I sell the index and make money? Or did I violate the rights of either the library or the books' copyright owners?
I think crawling and making use of the crawled data is not offending the copyright law because it's not actually a copy of the data. Rather, it's a service to transform the data and help other people get to the original data in an easier way.
Technically, they can crawl the data on the spot when receiving a request from their clients. In order to speed up the process, they somehow cache the data crawled from earlier occasions. But that's technical details, which is encapsulated. Otherwise, they could have stored URLs and offsets in the pages, rather than the data itself. I don't think that breaks the law, or there will be no way to avoid breaking the law to refer to anything.
Another possible related issue is the question of derivative works (e.g. the index of a library). There's a Wikipedia article about derivative works, it's a bit too complex to summarize here: http://en.wikipedia.org/wiki/Derivative_work
Both films and music (plus songwriting too) have been explicitly enshrined by US copyright law as having separate rights for home use and public performance. Each has multiple spheres of statutory licensing organizations with special exemptions from anti-trust law.
I'm not very familiar with US copyright legislation but I assumed that the copyright laws there are similar without checking any sources - which was a mistake apparently.
But, really, they can't stop it. An when (not if) nominative aggregated data is out, maybe the general population will begin to actually understand how expensive Facebook actually is.
Everything from stock prices to the temperature is in theory uncopyrightable, but you can still run into trouble if you just scrape some site and republish it.
This could have been an interesting case.
And regarding robots.txt - it's by no means anything more than a suggestion and common courtesy.. it's a way to suggest to crawlers what to ignore and how to behave with your site, to their benefit and yours - it's not by itself a legally binding, well, anything..... it was just a convention for all parties to unofficially cooperate.
It depends, re: the oft cited example of phone books being uncopyrightable,
If you run a public website, you have to accept that people may crawl your website. If you want to prevent it, don't let them crawl it. That's what robots.txt is for.