Another important point about this thread: this is a very common issue that regularly comes up, and no site is immune from it. I regularly see major sites have a firewall that auto-configures itself to block Googlebot, and the webmaster doesn't know what's going on. Raising awareness about this problem and how to fix it adds to the importance of replying.
Of course, there are people who want to take advantage of people like Matt to give themselves an economic advantage, and I have no sympathy for those people if they can't get someone in Google to help them as easily.
It comes down to karma.
That's exactly the problem. Google is notorious for providing horrible customer support, so why some people get personalized help, while the rest of us are stuck with the uncaring robots?
Giving occasional personalized help is a great way to avoid a total black hole while still creating an escalation path. It's like having EXPLAIN QUERY stack traces on NewRelic. You don't need a deep trace of every transaction; you need a deep trace from a few representative squeaky wheels, and you use that information to improve the underlying self-serve process. This is something Google, with their "1/3 of the company parses logs", excels at.
You'll note neither Matt nor Pierre offered to do anything special for PG. They didn't say "Oh! I love HN, and we want it to succeed; I'll go push out a manual override that makes sure we use only the one crawler you unblocked." They gave him exactly the info he'd get from Webmaster Tools plus general knowledge, and told him how to use Webmaster Tools to fix it. Maybe they'll go figure out how to implement that "you're blocked!" hover-over in SERPs, but again, that would benefit everyone, not just HN. There's no special treatment here.
[1] http://blog.wordtothewise.com/2008/04/troubleshooting-a-post...
The point is if you can build relationships with Google employees rather than with Google itself then you too can get this sort of amazing help.
There's nothing sinister or crazy about this. Rather than being one in a million, become one in a circle of friends or acquaintances.
PG knows his shit. He knows how to block crawlers, he knows how get SEO done, he's no dummy. "So what?" you say. Well I work in web development and I meet with people all the time and explain to them how to run a site and get rankings up, etc. I tell them I'll give them the tools, show them how to use them and even leave detailed manuals on how to work everything. These people are so into it at our first meeting and they're excited to get into it and work this stuff.
Then the job is done, the tools are handed over, and the lessons are learned. What do they do? Nothing. Then they do nothing. Then even more nothing. The site is never updated and they do no maintenance whatsoever and eventually it becomes outdated and dead for all intents and purposes. A year later they want to know why they're not ranking in Google. I ask if they've checked or even once used any of the tools I gave them and the answer is always no. Then they demand I fix it. I kindly remind them they're the ones who wanted to have Larry in accounting run the site because he's "good at computers" and to save money by not hiring an IT guy.
So the point I'm making is that I'm sure there are a ton of people out there who are trying to get customer service without trying to help themselves or using the resources readily available. I think it's great that Google is showing they're willing to help those who help themselves.
I think google isn't treating everyone equally but they are treating people fairly. There's a difference. I'd prefer fairness. Not all "customers" are created equal so you compensate by treating them with fairness instead.
Apparently he doesn't use Google Webmaster Tools, though.
With regard to this issue doing the above is not evidence of knowing your shit in the area in question.
Being bold is an important aspect of getting to know your shit.
Also, besides this, this action has gotten us talking about how Google throttles crawling, and this discussion has further reduced my opinion of the World's Largest Adware Vendor.
Now if I were to ban Google searchbots based on what PG is saying I would simply limit them to the first page or two of listing. I think this can be done in a robots.txt?
I think it's particularly good that this is being done publicly, too. There's billions of sites out there and Google can't provide this for all of them, but since they're doing it publicly others can learn from it and know how to address it.
If Matt was artificially boosting HN's ranking that would be disturbing, but they're fixing an indexing issue.
Lots of folks here like to give Matt Cutts this aura of an angel, or some sort of saint. Those who have known his actions over the past 5-10 years know better. Just ask guys like Aaron Wall and Rand. Matt Cutts is just a pawn that is there to do damage control for Google. That's all he is.
P.S. Just curious--is this the same AznHisoka from BlackHat World, the "Blackhat SEO forum"? http://www.blackhatworld.com/blackhat-seo/members/137345-azn...
I understand why you have to do this, but I wish there was at least some transparency regarding confirmation that an issue has at least been looked at, as opposed to just filed as a spam report.
I've only reported one issue to Google before, and the site in question, though incredibly obviously bad, is still the top link on the SERP for the business.
Specifically, there's a pizza place here in Halifax, NS that looks like it didn't renew it's domain in time, and some people are squatting it with an old copy of the site plus their own spammy links. The site[1] displays a copy of the site from 2008 or so, with a banner on the top which reads:
> To previous domain owner: We bought this domain after expiration so it's not our fault that you lost it. We put old content for this domain only to avoid losing good quality of it from SEO point of view. If it's a problem for you contact us ASAP!
I reported it to Google months ago and never heard a word about it, but today when I search for the business name - which I'd imagine a lot of people do when looking for menus, phone numbers, etc. - the "compromised" site still ranks first on the SERP[2]. I wonder how many people even notice the banner telling customers that <jedi>this is not the page you're looking for</jedi>.
[1] http://jessys.ca
[2] https://encrypted.google.com/search?hl=en&q=jessys%20piz...
Clearly that's not the case, as advertising makes 99% of their revenues.
Remember, organic doesn't have to mean non-commercial. In fact, if I search for best buy, the first organic result is bestbuy.com
I'm not disagreeing with the fact that ads are subject to a whole different objective function. I'm saying that if the best results were already in the organic, then the ads would either be redundant (for the user) or otherwise less useful than the organic (by definition, really).
All the people in this thread, including Google-related users, are trying to do is figure out what changed. That could just as easily have been at Google or HN's! It could also just as easily been a correct change.
If it turns out it was something HN changed and they did not use or even aware of the proper Google tools available for everyone to optimize their site, then I hardly think any reasonable person could object if it is pointed out to them - even if it's by Google personnel.
Umm...no. They're not tweaking the algo, they're explaining to PG that he should stop blocking the crawlers, or that he should verify the site in google webmaster tools, then change the crawl rate.
Seems like "bending over" to me:
"but I'm pinging the right people to ask whether we can get this fixed pretty quickly"
I can usually get the hacker news by just googling for "hn"
Secondly, HN is one of the most popular sources of tech news. By responding to this issue, Google is helping bring out possible solutions to a common problem - that crawlers are too frequent and affect the performance of a site. People like you and me can read the responses and learn what can be done to control this.
Third, popular websites like HN deserve attention like this because of the following they have garnered over the years. If,say "hahla blog" doesn't show as a top result in Google - it will take time for Google to verify whether it should even be shown as a top site. But, if something like HN or Amazon.com doesn't show as top result when users explicitly searches for it - that is a darn good use case for Google to fix the issue. It is an indication of something is definitely wrong somewhere that needs to be fixed.
from google's point of view, a user who types "hacker news" into google is almost certainly looking for news.ycombinator. therefore, it is highly desirable for the site to be the #1 result.
I think your best bet is to realize that, regardless of size, companies are always made up of people. And people are opinionated, subjective, and prone to making decisions that fall outside of some standard set of rules.
If fairness is treating people based on merit then it's totally fine. Matt isn't in here saying he's going to manually manipulate the result or change anything on Google's end (save for maybe correcting a flaw that would benefit everyone who's in a similar situation as HN). All he's doing is suggesting possible causes, trying to make a diagnosis, and just troubleshooting.
Most people who want this attention don't deserve it. They're the type of people who won't use the resources available like the google help documents or even learning how to administrate a site for best results in search engines. Plus, most people's website just aren't worthy of this attention.
I could go on but lets look at the reality here. We're all knowledgeable of how the web works here so let's just admit that HN is a damn popular site and it definitely seems like a fluke to have it rank as it was when this was submitted.
Does <title> make any difference?
I don't think of this site as "Hacker News". I think of it as ycombinator, and the subdomain, news.
Should users of hackernews.com think of that site as something else, e.g. whatever is between the title tags?
A searchable list of domain names, ranked by popularity. Or even a searchable list of main page titles. Is that how some users are using Google? If so, Google does not need a full, current cached copy of the crawlable web to provide that.
I suppose it is because the top hits below HN are perhaps even less about "hacker news" ?
Its pg's fault not googles, and I dont see why they should care. Maybe from their standpoint it would be more beneficial to google users who are used to typing in 'hacker news' to visit this site, but since when did that matter to google?
Also don't get me wrong I love both google and hackernews. I just find whats going on in this thread interesting..