It's probable they would lose. But it's far, far more probable that Pete would be bankrupt well in advance of that point in time, and that's how the court system works.
Well, they claim it is their right. The courts will decide if it actually is.
Plenty of places make claims to rights and make you sign waivers (dojos, for example), but fail in court. This could be interesting.
If you run a public website, you have to accept that people may crawl your website. If you want to prevent it, don't let them crawl it. That's what robots.txt is for.
If I buy a map, can I use it to be a guide, giving people directions and make a living?
If I buy a story book, can I use it to read to children and collect money from their parents?
Or, if I spend some time to index all the books in the library, can I sell the index and make money? Or did I violate the rights of either the library or the books' copyright owners?
I think crawling and making use of the crawled data is not offending the copyright law because it's not actually a copy of the data. Rather, it's a service to transform the data and help other people get to the original data in an easier way.
Technically, they can crawl the data on the spot when receiving a request from their clients. In order to speed up the process, they somehow cache the data crawled from earlier occasions. But that's technical details, which is encapsulated. Otherwise, they could have stored URLs and offsets in the pages, rather than the data itself. I don't think that breaks the law, or there will be no way to avoid breaking the law to refer to anything.
Another possible related issue is the question of derivative works (e.g. the index of a library). There's a Wikipedia article about derivative works, it's a bit too complex to summarize here: http://en.wikipedia.org/wiki/Derivative_work
Both films and music (plus songwriting too) have been explicitly enshrined by US copyright law as having separate rights for home use and public performance. Each has multiple spheres of statutory licensing organizations with special exemptions from anti-trust law.
I'm not very familiar with US copyright legislation but I assumed that the copyright laws there are similar without checking any sources - which was a mistake apparently.
But, really, they can't stop it. An when (not if) nominative aggregated data is out, maybe the general population will begin to actually understand how expensive Facebook actually is.
Everything from stock prices to the temperature is in theory uncopyrightable, but you can still run into trouble if you just scrape some site and republish it.
This could have been an interesting case.
And regarding robots.txt - it's by no means anything more than a suggestion and common courtesy.. it's a way to suggest to crawlers what to ignore and how to behave with your site, to their benefit and yours - it's not by itself a legally binding, well, anything..... it was just a convention for all parties to unofficially cooperate.
It depends, re: the oft cited example of phone books being uncopyrightable,
"If you collect information from users, you will: obtain their consent, make it clear you (and not Facebook) are the one collecting their information, and post a privacy policy explaining what information you collect and how you will use it."
"You will only use the data you receive for your application, and will only use it in connection with Facebook."
"By "application" we mean any application or website (including Connect sites) that uses or accesses Platform, as well as anything else that receives data." - note that by this definition a Facebook crawler is an applicatiom
"You will only use the data you receive for your application, and will only use it in connection with Facebook."
"You will have a privacy policy or otherwise make it clear to users what user data you are going to use and how you will use, display, or share that data."
"You will not transfer the data you receive from us (or enable that data to be transferred) without our prior consent."
"You will make it easy for users to remove or disconnect from your application."
etc.
These things are easy to look up, it took me about five minutes to find them. In fact, they have the right to limit the way even publicly faced content on their site is used. Things like copyright and data protection laws come in mind. You might not agree with that but it is absolutely in their right to do so.
The FB Terms & Conditions don't mention robots.txt at all.