You can't copyright facts, but you can copyright collections of facts.
This could have been an interesting case.
And regarding robots.txt - it's by no means anything more than a suggestion and common courtesy.. it's a way to suggest to crawlers what to ignore and how to behave with your site, to their benefit and yours - it's not by itself a legally binding, well, anything..... it was just a convention for all parties to unofficially cooperate.
You can't copyright facts, but you can copyright collections of facts.
It depends, re: the oft cited example of phone books being uncopyrightable,
Re:How I got sued by Facebook
by Lazlo_Nibble on Tuesday April 06, 2010 @05:50PM
(#1245763)
There has to be some kind of creative component to the compilation. And arguably Facebook didn't even compile the data in the first place -- it'd be like a college claiming a compilation copyright on the band fliers, roommate requests, and lost-pet ads pinned to a bulletin board in the student union building.
Everything from stock prices to the temperature is in theory uncopyrightable, but you can still run into trouble if you just scrape some site and republish it.