News for hackers. Stuff that matters.
Re:Robots.txt is a suicide note by tomjen3 on Tuesday May 10, 2011 @02:17AM (#2531584)
No, it is about not being willing to waste bandwidth and server capacity on unworthy projects (no person will ever search for my site through baidu but it still being indexed).

Google and archive.org is one thing, I will be happy to support them.

There are no comments on this comment yet.