Re:What one may find in robots.txt
by Ded7xSEoPKYNsDd on Monday May 18, 2015 @07:31AM
(#9563802)
Googlebot won't follow the link because it's listed in robots.txt.
[ Parent
| on HN ]
Googlebot caches robots.txt for a very, very long time. If you disallow a directory it may take months for the entire googlebot fleet to start ignoring it. Google's official stance is that you should manage disallow directives through webmaster tools.