Re:Show HN: Effort to clone unmaintained SourceForge project...
by nadams on Saturday June 27, 2015 @11:57AM
(#9790533)
Honestly I think ignoring robots.txt in this case is acceptable. Even if he programs in code to respect robots.txt - once the management at sourceforge get wind of what he is doing - what is stopping sourceforge from putting up robots.txt everywhere blocking him?
[ Parent
| on HN ]
Re:Show HN: Effort to clone unmaintained SourceForge project...
by ihatehn on Saturday June 27, 2015 @11:57PM
(#9792383)
Look at their current robots.txt; they're already prohibiting robots to crawl the actual source code: http://sourceforge.net/robots.txt