You can't copyright facts, but you can copyright collections of facts.
It depends, re: the oft cited example of phone books being uncopyrightable,
Re:How I got sued by Facebook
by Lazlo_Nibble on Tuesday April 06, 2010 @05:50PM
(#1245763)
There has to be some kind of creative component to the compilation. And arguably Facebook didn't even compile the data in the first place -- it'd be like a college claiming a compilation copyright on the band fliers, roommate requests, and lost-pet ads pinned to a bulletin board in the student union building.
This could have been an interesting case.
And regarding robots.txt - it's by no means anything more than a suggestion and common courtesy.. it's a way to suggest to crawlers what to ignore and how to behave with your site, to their benefit and yours - it's not by itself a legally binding, well, anything..... it was just a convention for all parties to unofficially cooperate.