News for hackers. Stuff that matters.

Tell HN: The front page of Hacker News has been deindexed from Google

You can confirm this by searching for 'hacker news' in Google; the #1 ranking URL is /newest, rather than the front page. This isn't term specific - the site doesn't appear for other terms that it usually ranks well for, such as "news.ycombinator.com" or "hn".

I've checked the usual technical reasons (html head canonical/robots meta tag, http headers, robots.txt issues) but I don't see anything untoward.

I'll keep looking into it, but I'm posting this here in case the admins/mods have made any changes recently that could have had an effect. There's a possibility that the URL has been removed by Google for some particular reason, though I can't think of many pages that deserve it less than HN.

I'll update this thread if I see anything, but hopefully someone else will post an answer before I figure it out....

( 38 comments | 66 points | on HN )
Re:Tell HN: The front page of Hacker News has been deindexed... by Matt_Cutts on Thursday June 27, 2013 @07:34PM (#5955374)
It's not that PG has a grudge against Google (or vice versa) or anything like that. I believe that search engine bots crawl Hacker News hard enough that PG blocks most crawling by bots. In the case of Google, he does allow us to crawl from some IP addresses, but it's true that Google isn't able to crawl/index every page on Hacker News.

Here's a link where I answered the same question about three weeks ago: https://news.ycombinator.com/item?id=5837004 , so this isn't a new issue. In fact, PG has been blocking various bots since 2011 or so; https://news.ycombinator.com/item?id=3277661 is one of the original discussions about this.

And to show this isn't a Google-specific issue, note that Bing's #1 result for the search [hacker news] is a completely different site, thehackernews.com: http://www.bing.com/search?q=hacker+news

In general, I think PG's priority is to have a useful, interesting site for hackers. That takes precedence and is the reason why I believe PG blocks most bots: so that crawling doesn't overload the site.

Re:Tell HN: The front page of Hacker News has been deindexed... by Roedou on Thursday June 27, 2013 @07:44PM (#5955408)
Thanks for that Matt; I didn't see that recent post or your comment, so sorry for dragging you back here to repeat yourself.

Looks like I'm going to have to stop relying on searching 'hn' when using a different computer, and start typing in the full URL. First world problems are such a burden.

Re:Tell HN: The front page of Hacker News has been deindexed... by Matt_Cutts on Thursday June 27, 2013 @07:48PM (#5955426)
No worries at all. I don't think the HN thread from three weeks ago made it to the front page (I happened to see it while browsing on /newest). I figured someone would notice and ask about this, so I'm happy to have the chance to explain.
Re:Tell HN: The front page of Hacker News has been deindexed... by mkbrody on Thursday June 27, 2013 @08:37PM (#5955573)
Hey Matt,

I'm sorry to reach out to you directly on a public forum like this, but my company's website encountered a major negative SEO attack last month and we were hit with a manual penalty by Google today. I thought you might be interested to hear about what happened, and I of course I would like to resolve it as I do my best to always keep my company's SEO efforts within Google's guidelines. Please reach out via email to me at [email protected] if we can help each other fix this! Thanks again for everything you do to help make the web a better place, and in advance I understand if you're too busy to respond.

Best regards,

Mike B.

Re:Tell HN: The front page of Hacker News has been deindexed... by nicholasreed on Thursday June 27, 2013 @09:47PM (#5955750)
Don't apologize to just Matt, you're pseudo-apology and better-sent-as-an-email question pissed me off. Why would you take up three extra lines with a BS platitude and a signature? Please keep a personal request for assistance to better-suited channels.
Re:Tell HN: The front page of Hacker News has been deindexed... by karolist on Friday July 05, 2013 @08:33AM (#5994576)
Just downvote and move along, why the hostility.
Re:Tell HN: The front page of Hacker News has been deindexed... by sswaner on Thursday June 27, 2013 @09:06PM (#5955652)
Re:Tell HN: The front page of Hacker News has been deindexed... by chintan on Thursday June 27, 2013 @09:25PM (#5955712)
Doesn't Googlebot respect Crawl-Delay in robots.txt? PG has set it to be 30 secs - https://news.ycombinator.com/robots.txt - which IMHO should not cause any load issues given the overall traffic profile of HN.
Re:Tell HN: The front page of Hacker News has been deindexed... by kanamekun on Thursday June 27, 2013 @10:02PM (#5955782)
Googlebot doesn't respect the crawl-delay setting in robots.txt. https://groups.google.com/forum/#!topic/Google_Webmaster_Hel...

As I understand it, the best way to lower the crawl rate is to log into Google Webmaster Tools and manually lower your crawl rate. The crawl rate delays expire every 90 days, so I set a calendar reminder to renew the crawl delay every 3 months. https://support.google.com/webmasters/answer/48620?hl=en

Re:Tell HN: The front page of Hacker News has been deindexed... by jlgreco on Thursday June 27, 2013 @08:10PM (#5955491)
Mmm, seems kind of like a feature. In fact, maybe PG should robots.txt google entirely. It seems like HN has been getting mentions in other media with increasing frequency. If you can't find the site just because google doesn't doesn't list it, then I have to wonder what you are actually doing here. This wouldn't be the first way that HN sets a bar for new users either; the "Create Account" form is already hidden under "submit".

HNSearch works great for HN specific searches anyway.

Re:Tell HN: The front page of Hacker News has been deindexed... by JoeCortopassi on Thursday June 27, 2013 @07:00PM (#5955254)
This has happened before, and usually has a non-pitchforky reasoning (e.g. PG pulled it temporarily because of network/server issue). I'm sure it will be back soon, and we will have a rather reasonable answer as to why. There are way to many google employees, that frequent and enjoy HN, for it to be banned for some arbitrary reason
Re:Tell HN: The front page of Hacker News has been deindexed... by AsymetricCom on Thursday June 27, 2013 @07:27PM (#5955357)
And of course, the network has specific functions for censorship as required by child protection laws. "Just a network error" really doesn't guarantee that the network wasn't doing something nefarious itself.
Re:Tell HN: The front page of Hacker News has been deindexed... by gee_totes on Thursday June 27, 2013 @06:13PM (#5955067)
If you are using DuckDuckGo, you can use the !hn bang to send your query to hnsearch.com
Re:Tell HN: The front page of Hacker News has been deindexed... by eliben on Thursday June 27, 2013 @08:32PM (#5955558)
This is trivial to do in any modern browser without DDG.
Re:Tell HN: The front page of Hacker News has been deindexed... by Roedou on Thursday June 27, 2013 @07:04PM (#5955263)
I found this old thread, where pg had blocked most of the Google bots, and it caused Google to think the site was down:

https://news.ycombinator.com/item?id=3277661

Could be a similar issue? I'll take a look.

Re:Tell HN: The front page of Hacker News has been deindexed... by Roedou on Thursday June 27, 2013 @07:15PM (#5955316)
pg also commented that he doesn't want traffic from Google anyway: https://news.ycombinator.com/item?id=5808990

In which case, he should add: <meta name="googlebot" content="noindex"> to the html head of every page.

(I have to say, that's a smart way of avoiding any Eternal Septembering, but it'd be a shame. I often use Google to find old HN threads that I vaguely remember from months or years ago.)

Re:Tell HN: The front page of Hacker News has been deindexed... by dnautics on Thursday June 27, 2013 @07:17PM (#5955322)
you may want to consider using hnsearch.
Re:Tell HN: The front page of Hacker News has been deindexed... by saurik on Thursday June 27, 2013 @07:42PM (#5955402)
(Google's search is often better for this purpose as it has features like synonym/typo fixing and as it indexes entire pages lets you match keywords across entire threads: hnsearch is myopic on individual posts.)
Re:Tell HN: The front page of Hacker News has been deindexed... by snogglethorpe on Thursday June 27, 2013 @10:48PM (#5955914)
Yup...

Even sites with good search functions are often still way outclassed by google search with "site:..."—and most sites don't have good search functions...

Re:Tell HN: The front page of Hacker News has been deindexed... by glitch273 on Thursday June 27, 2013 @06:21PM (#5955104)
Matt Cutts browses this site. Maybe he knows the reason why?
Re:Tell HN: The front page of Hacker News has been deindexed... by jffry on Thursday June 27, 2013 @07:57PM (#5955457)
Re:Tell HN: The front page of Hacker News has been deindexed... by mattparlane on Thursday June 27, 2013 @08:19PM (#5955525)
The .org site hasn't been delisted, so it's obviously not based on content:

https://www.google.co.nz/search?q=site:news.ycombinator.org

Re:Tell HN: The front page of Hacker News has been deindexed... by meritt on Thursday June 27, 2013 @06:23PM (#5955111)
This is most likely the same reason digg's frontpage was deindexed. There's no "content" per se, it's just links. Someone will notice, add an exception, and all is well.

Unlike Digg, HN has a substantial amount of content in the comments pages though, which are heavily indexed.

Edit - All the comment pages are still indexed just fine. It's /only/ the front-page. Which, imo, doesn't really matter anyway.

Re:Tell HN: The front page of Hacker News has been deindexed... by Roedou on Thursday June 27, 2013 @06:55PM (#5955233)
That wasn't the reason for Digg's issue at all: Google had tried to manually deindex some pages from the site, but made a mistake and pulled the whole domain. They reincluded it shortly after.
Re:Tell HN: The front page of Hacker News has been deindexed... by pstuart on Thursday June 27, 2013 @06:33PM (#5955148)
The comments are the real content of this site.
Re:Tell HN: The front page of Hacker News has been deindexed... by aidscholar on Thursday June 27, 2013 @06:40PM (#5955172)
Sounds like overaggressive spam detection.
Re:Tell HN: The front page of Hacker News has been deindexed... by joepawl on Thursday June 27, 2013 @06:45PM (#5955194)
This sounds like the case. Google is getting aggressive with its Panda updates, and as a previous commenter noted, the HN homepage is just links. Since that triggers Panda, it's a good bet that Google went a little overboard (not unprecedented).
Re:Tell HN: The front page of Hacker News has been deindexed... by mahranch on Thursday June 27, 2013 @07:04PM (#5955266)
> the HN homepage is just links. Since that triggers Panda

To be more specific, Panda is triggered by low quality/duplicate content. 'Penguin' is triggered by spammy/bad backlinks.

I'm not saying you're wrong (a page of links would look pretty low quality to google's algo), I just wanted to add on for clarity's sake.

Re:Tell HN: The front page of Hacker News has been deindexed... by joepawl on Thursday June 27, 2013 @08:58PM (#5955633)
Yes, I see where I was unclear. It's not the links themselves, but the lack of original, robust content.
Re:Tell HN: The front page of Hacker News has been deindexed... by Roedou on Thursday June 27, 2013 @06:53PM (#5955223)
The Panda update definitely pushed lazy/low-quality pages down the SERPs, but doesn't tend to deindex pages.

Also: while the page doesn't have any of its own unique content, it presumably still has high engagement and low bounce rate.

Re:Tell HN: The front page of Hacker News has been deindexed... by joepawl on Thursday June 27, 2013 @09:03PM (#5955645)
That's a good point. Hadn't thought the issue through that far.
Re:Tell HN: The front page of Hacker News has been deindexed... by eli on Thursday June 27, 2013 @07:43PM (#5955405)
Please don't post on HN to ask or tell us something (e.g. to ask us questions about Y Combinator, or to ask or complain about moderation). If you want to say something to us, please send it to [email protected].

http://ycombinator.com/newsguidelines.html

Re:Tell HN: The front page of Hacker News has been deindexed... by malandrew on Thursday June 27, 2013 @08:21PM (#5955530)
I too had noticed this. It's unfortunate because searching via Google with site:news.ycombinator.com in the query is much better than HN's own search when you have a good idea what you're looking for (spearfishing search vs BFS)
Re:Tell HN: The front page of Hacker News has been deindexed... by chacham15 on Thursday June 27, 2013 @06:54PM (#5955227)
This isnt the first time this has happened and I suspect that it wont be the last.
Re:Tell HN: The front page of Hacker News has been deindexed... by gscott on Thursday June 27, 2013 @10:12PM (#5955798)
The pagerank has fallen from a 6 to a 3 as well.
Re:Tell HN: The front page of Hacker News has been deindexed... by godgod on Thursday June 27, 2013 @06:53PM (#5955221)
Google is evil. Screw them. I refuse to use Google or their services. Make the switch. They deindex a lot of sites they don't agree with. Not saying that is the case here but they've been known to do it.
Re:Tell HN: The front page of Hacker News has been deindexed... by quantumpotato_ on Thursday June 27, 2013 @07:09PM (#5955288)
Link backing up your claims?
Re:Tell HN: The front page of Hacker News has been deindexed... by mindstab on Thursday June 27, 2013 @07:57PM (#5955456)
Switch to what?