News for hackers. Stuff that matters.
Re:What one may find in robots.txt by ceequof on Monday May 18, 2015 @10:39AM (#9564677)
Googlebot caches robots.txt for a very, very long time. If you disallow a directory it may take months for the entire googlebot fleet to start ignoring it. Google's official stance is that you should manage disallow directives through webmaster tools.

There are no comments on this comment yet.