News for hackers. Stuff that matters.
Re:Hacker News bans Google and all other search engines by pg on Monday March 15, 2010 @10:47PM (#1194797)
Don't worry, it doesn't mean anything. The software for ranking applications runs on the same server, and it is horribly inefficient (something 4 people use every 6 months doesn't tend to get optimized much). This weekend all of us were reading applications at the same time, and the system was getting so slow that I banned crawlers for a bit to buy us some margin. (Traffic from crawlers is much more expensive for us than traffic from human users, because it interacts badly with lazy item loading.) We only finished reading applications an hour before I had to leave for SXSW, so I forgot to set robots.txt back to the normal one, but I just did now.
Re:Hacker News bans Google and all other search engines by g0atbutt on Monday March 15, 2010 @10:53PM (#1194807)
That makes sense. Thanks for the explanation pg!
Re:Hacker News bans Google and all other search engines by lisper on Tuesday March 16, 2010 @12:34AM (#1195004)
Dude, why not just get yourself another server?
Re:Hacker News bans Google and all other search engines by sbhat7 on Tuesday March 16, 2010 @12:59AM (#1195052)
"something 4 people use every 6 months doesn't tend to get optimized much"
Re:Hacker News bans Google and all other search engines by _delirium on Tuesday March 16, 2010 @04:21AM (#1195289)
Yeah, but the usual case is still to throw stuff like that on a spare "random odds and ends go here" server, not a production one whose performance/responsiveness you care about.
Re:Hacker News bans Google and all other search engines by kmt on Tuesday March 16, 2010 @09:52AM (#1195654)
It likely has a dependency on HN: looking up users, karma, etc., which is just simpler to do on the same machine.
Re:Hacker News bans Google and all other search engines by markbao on Tuesday March 16, 2010 @10:13AM (#1195713)
Ranking applications by HN karma? I hope not.
Re:Hacker News bans Google and all other search engines by davidu on Tuesday March 16, 2010 @11:04AM (#1195826)
They've been clear that they do look at your HN posts and comments, etc.

It may or may not be used for ranking in specific, but it's definitely used and they've always been upfront about that when asked.

Re:Hacker News bans Google and all other search engines by idoh on Tuesday March 16, 2010 @02:59PM (#1196517)
Also, they use hacker news to communicate to applicants. I applied a while back, and one day while browsing I got a message through HN, something like "pg has a question for you" in the header bar. Clicking through is the question, and place to put the answer.

Basically, there's a whole messaging system in hacker news that most people don't see.

Re:Hacker News bans Google and all other search engines by markbao on Tuesday March 16, 2010 @06:09PM (#1196952)
Yeah, this is known :) but an algo that incorporates HN karma wouldn't be the best thing :P
Re:Hacker News bans Google and all other search engines by Xichekolas on Tuesday March 16, 2010 @04:37PM (#1196744)
I was told once that they don't rank on karma, but if they recognize your name from discussions around here it helps your case.
Re:Hacker News bans Google and all other search engines by snprbob86 on Tuesday March 16, 2010 @12:42AM (#1195020)
I'd be really worried about some algorithm somewhere deciding to crawl less frequently even after the temporary ban has been lifted.
Re:Hacker News bans Google and all other search engines by jsrfded on Tuesday March 16, 2010 @01:01AM (#1195057)
Robot bans can render your archive.org records inaccessable, zap all your former code 200 urls out of the major search indices, and yes, kill your inclusion in the list of topical sites.

Much better to set a temporary Crawl-Delay directive. Otherwise you're not just telling the engines to pause crawling you, you're telling them "take all of my pages out of your index."

Re:Hacker News bans Google and all other search engines by Andys on Tuesday March 16, 2010 @01:29AM (#1195101)
What are the details of crawl-delay? If its supposed to be "number of seconds between hits", for some sites, 1 hit per second may as well be "not crawled at all"..
Re:Hacker News bans Google and all other search engines by ComputerGuru on Tuesday March 16, 2010 @01:43AM (#1195124)
Right, but his point is that at least this way you won't get delisted :)
Re:Hacker News bans Google and all other search engines by duck on Wednesday March 17, 2010 @09:15AM (#1198452)
Why is this still on the front page of HN? This seems like such a non-story.
Re:Hacker News bans Google and all other search engines by ks on Saturday March 20, 2010 @07:47PM (#1207234)
Have you considered using last-modified/etag headers to control the crawlers? If you detect a crawler, and it requests a page that it has visited before, you could just return a 304 without doing anything else.
Re:Hacker News bans Google and all other search engines by robryan on Tuesday March 16, 2010 @01:49AM (#1195137)
On a related note the threads page seems to load very slowly now. I guess because it would place heavy load on the DB grabbing all your comments from different threads.