http://www.wired.com/techbiz/media/news/2005/09/68928
http://cci.edu.au/publications/search-engine-liability-copyr...
Parker v Yahoo takes the view that putting content online is implicitly allowing indexing. By the same logic, explicitly blocking an engine would mean the opposite.
http://blog.ericgoldman.org/archives/2008/10/search_engine_c...
The second one includes some great information, but if you read about the specific cases (starting on page 6) you can see it is still very much up for debate. Actually it seems that the courts have ruled in Google's favor almost every time, and most of these are for caching and not indexing.
If a site really really wants to block a particular engine run by a for-profit company, or make it impractical (either legally or technically) they can.