Looks like I'm going to have to stop relying on searching 'hn' when using a different computer, and start typing in the full URL. First world problems are such a burden.
I'm sorry to reach out to you directly on a public forum like this, but my company's website encountered a major negative SEO attack last month and we were hit with a manual penalty by Google today. I thought you might be interested to hear about what happened, and I of course I would like to resolve it as I do my best to always keep my company's SEO efforts within Google's guidelines. Please reach out via email to me at [email protected] if we can help each other fix this! Thanks again for everything you do to help make the web a better place, and in advance I understand if you're too busy to respond.
Best regards,
Mike B.
As I understand it, the best way to lower the crawl rate is to log into Google Webmaster Tools and manually lower your crawl rate. The crawl rate delays expire every 90 days, so I set a calendar reminder to renew the crawl delay every 3 months. https://support.google.com/webmasters/answer/48620?hl=en
Here's a link where I answered the same question about three weeks ago: https://news.ycombinator.com/item?id=5837004 , so this isn't a new issue. In fact, PG has been blocking various bots since 2011 or so; https://news.ycombinator.com/item?id=3277661 is one of the original discussions about this.
And to show this isn't a Google-specific issue, note that Bing's #1 result for the search [hacker news] is a completely different site, thehackernews.com: http://www.bing.com/search?q=hacker+news
In general, I think PG's priority is to have a useful, interesting site for hackers. That takes precedence and is the reason why I believe PG blocks most bots: so that crawling doesn't overload the site.