The robots.txt discussion is a worthy one. For example, is it against net neutrality (the concept) to only accept being indexed by Google? I think so. In a world where every site only accepts Google and other well known search engines to index them tgere is no room for a new search engine to appear.