News for hackers. Stuff that matters.

A comment on Nano is no longer a GNU project. Parent comment

Re:Nano is no longer a GNU project by tzs on Wednesday June 22, 2016 @12:07PM (#11954953)
They are more accessible here on HN, though, because crawlers that obey robots.txt will not see it on git.savannah.gnu.org but will see it here on HN.
[ Parent | on HN ]
Re:Nano is no longer a GNU project by yellowapple on Wednesday June 22, 2016 @02:44PM (#11956057)
I suspect that if a crawler is being used to farm email addresses for spamming, it's highly unlikely that robots.txt would be any deterrent whatsoever.
[ Parent | on HN ]
Re:Nano is no longer a GNU project by mysidia1 on Thursday June 23, 2016 @09:02AM (#11960530)
Spammers' crawlers use URLs obtained from search engines and public sources. If the whole directory is blocked in robots.txt, it WILL reduce crawling activity massively.
[ Parent | on HN ]