News for hackers. Stuff that matters.
Re:Poll: Should this site be visible to Yahoo, MSN crawlers? by icey on Wednesday April 16, 2008 @01:57PM (#165395)
Why not just robots.txt out all search engines? Of course, if you did that, it'd be pretty nice to have a "Search" button up there in the orange bar.
Re:Poll: Should this site be visible to Yahoo, MSN crawlers? by Tichy on Wednesday April 16, 2008 @08:23PM (#165813)
How do you robot.txt for search engines? If anything, you could ban all robots, I suppose. I wouldn't like that - it is akin to those social networks where you can't get your data out.