News for hackers. Stuff that matters.

A comment on Nano is no longer a GNU project. Parent comment

Re:Nano is no longer a GNU project by yellowapple on Wednesday June 22, 2016 @02:44PM (#11956057)
I suspect that if a crawler is being used to farm email addresses for spamming, it's highly unlikely that robots.txt would be any deterrent whatsoever.
[ Parent | on HN ]
Re:Nano is no longer a GNU project by mysidia1 on Thursday June 23, 2016 @09:02AM (#11960530)
Spammers' crawlers use URLs obtained from search engines and public sources. If the whole directory is blocked in robots.txt, it WILL reduce crawling activity massively.
[ Parent | on HN ]