News for hackers. Stuff that matters.
Re:Poll: Should this site be visible to Yahoo, MSN crawlers? by Alex3917 on Wednesday April 16, 2008 @02:02PM (#165403)
I wonder if one could improve the efficiency of the crawlers by dynamically updating the robots.txt based on votes and time.

There are no comments on this comment yet.