News for hackers. Stuff that matters.
Re:Show HN: Effort to clone unmaintained SourceForge project... by nadams on Saturday June 27, 2015 @11:57AM (#9790533)
Honestly I think ignoring robots.txt in this case is acceptable. Even if he programs in code to respect robots.txt - once the management at sourceforge get wind of what he is doing - what is stopping sourceforge from putting up robots.txt everywhere blocking him?
Re:Show HN: Effort to clone unmaintained SourceForge project... by ihatehn on Saturday June 27, 2015 @11:57PM (#9792383)
Look at their current robots.txt; they're already prohibiting robots to crawl the actual source code: http://sourceforge.net/robots.txt