News for hackers. Stuff that matters.
Re:What one may find in robots.txt by Ded7xSEoPKYNsDd on Monday May 18, 2015 @07:31AM (#9563802)
Googlebot won't follow the link because it's listed in robots.txt.
Re:What one may find in robots.txt by ceequof on Monday May 18, 2015 @10:39AM (#9564677)
Googlebot caches robots.txt for a very, very long time. If you disallow a directory it may take months for the entire googlebot fleet to start ignoring it. Google's official stance is that you should manage disallow directives through webmaster tools.
Re:What one may find in robots.txt by ryanlol on Monday May 18, 2015 @11:01AM (#9564825)
Yes, but it will index it.