News for hackers. Stuff that matters.
Re:Poll: Should this site be visible to Yahoo, MSN crawlers? by gojomo on Wednesday April 16, 2008 @03:34PM (#165533)
Is there a reason for any "/x?" URLs to be indexed?

If not, a simple robots.txt would fix the crawler-traffic problem without prejudicially banning any upstart search engines:

  User-Agent: *
  Disallow: /x?

There are no comments on this comment yet.