Google can do it because Facebook doesn't really want to go up against money in making this a test case, and Google has baskets of cash.
Facebook doesn't give a tinker's damn about prohibiting your value creation.
Or because Google would simply comply with their demands and remove all mention of facebook from their indexes... That would be pretty interesting.
Ha! Yeah, I hadn't thought of that aspect. Half of their userbase wouldn't even be able to sign on any more!
rather more than half I think
Disallow: /directory/people/*
Disallow: /directory/pages/*
Done. Google crawls Facebook. Every social media monitoring company crawls Facebook (and every other social network). Tons of other companies do the same every day. I know this because many of these companies are our customers.
What really irks me is that Pete collected valuable data and whoever he shared it with was probably able to derive added value from that data - in ways that Facebook is not doing. Facebook is prohibiting value creation.