News for hackers. Stuff that matters.
Re:Poll: Should this site be visible to Yahoo, MSN crawlers? by gojomo on Wednesday April 16, 2008 @02:21PM (#165429)
I use Yahoo search occasionally and have often been impressed at it finding exact material, with a precise search, that Google couldn't.

Separate from the local implications on News.YC, society benefits by having multiple competing search engines. Seceding from all search indexes but Google contributes to a monopoly/monoculture.

Can you just warn off the stupid crawlers from valueless URLs with some combination of robots.txt, META robots directives, and 'nofollow'? (I'm happy to help craft these signposts, with a description of the URLs to be spared.)

There are no comments on this comment yet.