Re:Robots.txt is a suicide note
by tomjen3 on Tuesday May 10, 2011 @02:17AM
(#2531584)
No, it is about not being willing to waste bandwidth and server capacity on unworthy projects (no person will ever search for my site through baidu but it still being indexed).
Google and archive.org is one thing, I will be happy to support them.
Google and archive.org is one thing, I will be happy to support them.