Re:What one may find in robots.txt
by ceequof on Monday May 18, 2015 @10:39AM
(#9564677)
Googlebot caches robots.txt for a very, very long time. If you disallow a directory it may take months for the entire googlebot fleet to start ignoring it. Google's official stance is that you should manage disallow directives through webmaster tools.
[ Parent
| on HN ]
There are no comments on this comment yet.