I assume a correct robots.txt, plus attempt to kill off robots.txt evaders; if you could do that reliably (the second bit is the tricky bit, of course), you could selectively exclude any bots you want.