News for hackers. Stuff that matters.
Re:Poll: Should this site be visible to Yahoo, MSN crawlers? by jules on Wednesday April 16, 2008 @01:44PM (#165369)
Why does it matter how many useless urls they visit? If you're trying to fix the "expired link" issue -- there are better ways to fix that.

I think you should ban google too if you ban the others. The only disadvantage I can see is that people from here who want to find something they've seen on news.yc can't use google anymore.

Re:Poll: Should this site be visible to Yahoo, MSN crawlers? by jsrn on Wednesday April 16, 2008 @02:04PM (#165407)
> [...] can't use google anymore.

I think pg should just make a search link on top of the site pointing to

http://www.searchyc.com/

or

http://nycs.bigheadlabs.com/

Re:Poll: Should this site be visible to Yahoo, MSN crawlers? by pchristensen on Wednesday April 16, 2008 @01:59PM (#165398)
Blocking a crawler is a one liner that could be done in a couple seconds. Tackling the limitations of a continuation based architecture is a bigger problem.