Re:Digg banned from Google?
by blauwbilgorgel on Wednesday March 20, 2013 @01:12PM
(#5408644)
Incorrect. To configure your robots.txt to not be crawled at all use:
Allow indexing of everything with:
It seems they are at this very moment struggling to change things.
[ Parent
| on HN ]