Re:The Definition of Evil: Microsoft's Search Wars Hurt Us All
by blasdel on Monday November 23, 2009 @04:34PM
(#958197)
Even if the spider doesn't respect robots.txt, the act of indexing is really obvious -- your webserver does not have to respond to their HTTP Requests if you don't want it to.
It's not as facile as 'lock your doors' -- instead just not answering when they knock.
It's not as facile as 'lock your doors' -- instead just not answering when they knock.