Re:Testing 3 million hyperlinks, lessons learned
by Aissen on Thursday June 07, 2012 @08:28AM
(#4078629)
He explains how and why in the article, and gives arguments. You do none.
The problem with whitelist-only robots.txt is that they favor monopolies and startups are the ones getting the "fuck you". But maybe you don't care about that.
The problem with whitelist-only robots.txt is that they favor monopolies and startups are the ones getting the "fuck you". But maybe you don't care about that.