News for hackers. Stuff that matters.
Re:Poll: Should this site be visible to Yahoo, MSN crawlers? by wastedbrains on Wednesday April 16, 2008 @07:03PM (#165760)
Is there a way to block only the useless calls? The x that fail? Or to robots.txt anything that isn't main story pages so it wouldn't follow every individual comment link?

It might be good to still be indexed on the primary single page for each discussion... Then again I only use google, so it doesn't bother me any.

There are no comments on this comment yet.