Re:GitHub Says ‘No Thanks’ to Bots — Even if They’re Nice
by jgeralnik on Sunday December 30, 2012 @04:45AM
(#4985255)
Not connected to github, but look at https://github.com/robots.txt, specifically the first 2 lines.
Nuuton is currently crawling the web. The plans include crawling Github (actually, Github has a specific and exclisive crawler built for it). Is that permitted? If so, what are the rules? If not, to whom may I speak regarding it? I know DuckDuckGo does it, but I don't know if they are crawlin gyour site or just using what the Bing index currently has.