I don't even understand the theory of how there is a profit motive here.

How does Google make more money by asking webmasters to hide parts of their site from the indexer?

Doesn't Occam's Razor say that a more likely explanation is that innocent sites are being caught in the never-ending arms race between spammers and Google?

If Google didn't care about the "indie" web, why would Google employees be taking their time to coach indie website operators on how to improve their site's rankings?

I know this is a frustrating situation for everyone involved. But the cynicism that says there is some ulterior motive cuts deep, especially coming from someone I respect like you, Matt.

I would imagine you actually do see the profit motive for that activity, and yet you rejected the statement above by saying, "I don't even understand the theory of how there is a profit motive here. How does Google make more money by asking webmasters to hide parts of their site from the indexer?"

Could you explain how these things connect in your view?

There are sometimes quite a lot of ads on the search results page these days, but they are still clearly distinguished, and I haven't heard anything about that policy changing.

The conflict is between "the world's information" (mission) and "the world's information, minus some troublesome long-tail bits that few people care about and aren't profitable anyway" (business motive).

> why would Google employees be taking their time to coach indie website operators on how to improve their site's rankings

This "coaching" is actually Google pushing the spam problem onto content authors, promoting "solutions" that benefit Google while still hurting legit publishers. "Spam has gotten so bad that our algorithms can't classify it without misclassifying you. You can either hide some of your content as a proof that it's not link spam, or have us hide even more on presumption that it is." Yes, this gives Google a powerful tool for weeding out fake/malicious content, but it also hurts the user experience and the mission in the long run, compared to solutions that allow for more manual review, more use of inbound link disavowal, or finer-grained penalties that target only the possibly "low-quality" pages without spilling over to entire sites. Those solutions would probably shift more cost to Google, however.

What's really bad is that some of Google's recent recommendations (e.g. "use the noindex robots meta tag") will remove real knowledge not just from Google but from other present and future search engines (even if those engines wouldn't have the same problems Google did with that content). And even if the algorithm's limitations/errors are eventually resolved, the recommendations will live on much longer in legacy content (and in SEO folklore), meaning that some content will be erased from search permanently for no reason. These are long-term costs that are borne not by Google but by users, publishers, and competing search engines.

Personally, I want to be able to search ALL THE THINGS even if it means that I sometimes have to wade through more crap, and that it's harder work for the search engine to weed out people who are gaming the system. Maybe Google can't offer me that without degrading its more mainstream users. Even if that's the case, it still feels like a departure from principles (or maybe a declaration of defeat) for Google to be making that choice. Even more so when it hurts the ability of other search engines to step in and index the stuff that Google doesn't want.

The reason I want to see Google hew strictly to "the mission" is that it offers a way to let these big-picture issues override things that are more immediately measurable. I know I became an extensive user of Google services mainly because of the way its mission-oriented thinking was visible in its product design choices. That intangible feeling was more important than any particular feature. Keeping it alive requires the ability to say, "Screw it, we're not going to go this route no matter its obvious benefits, simply because it's not in line with the mission." But I also know that it's not possible to do that in every decision; even here in the non-profit, mission-driven Mozilla project, principles alone can't win every fight. (See http://robert.ocallahan.org/2014/05/unnecessary-dichotomy.ht... for an interesting take on this.) And these things tend to look different from the inside; maybe Panda was a case where de-indexing legit content was the least-bad path available. But from the outside, every time "pragmatism" wins over "principle" is a little chink in the armor. And for Google, the outside view is the only one I have.

Agreed 100%. I often start looking at the search results beyond page 10, since that skips past a lot of the "sites that Google's algorithm thinks are relevant, but are actually useless". And thanks to their ranking algorithms, the linkfarms and spammy sites tend to cluster together so it's easy to skip past them too (or just add -site:spammydomain.com to the query and try again).

Yes, yes, yes, yes, this please! Google used to still have this in the first half of the 2000s or so and it was glorious.

Truly having the world's information at your fingertips. Not just Wikipedia and some high ranking sites, but as your queries got more and more specific, you'd still get results from increasingly strange and dusty corners of the web. Yes that meant you had to evaluate the value and trustworthiness of results yourself (but really, you still do), and it was so very much worth it.

Obviously storage and networking costs would make it impossible today, but it might be feasible in the not-so-distant future. Either way, sooner or later people will realize the absurdity from a public-interest point of view of having "all the world's information" in the hands of a single corporation whose sole legal purpose is maximize its owners' profit --like it happened with other media in the past, like broadcast news.

Ha! HN loves GOOG. Try frequenting this site as an Apple employee...

Maybe they get paid by competition. After all removing competition with clever TOS tricks is right up Googles alley:

http://pando.com/2014/05/29/after-google-bought-nest-it-remo...

Looks like google saw what bookface was doing and copied. Remove from existence and charge for exposure.

make sure that all of your indexed pages are of the highest quality possible and that they are fantastic representatives of your website

JohnMu then gave some examples of very thin pages. It is about a site with 99% of such pages and problems with spammy UGC. Not about maximizing search advertising revenue. It is about quality like described in: http://googlewebmastercentral.blogspot.com/2011/05/more-guid...

I'd like if there was a traffic graph (is an algorithmic penalty visible, trends, or are they just down to 10% of 1.5 years ago?). We have to go by word and own research now. Googling "Google Maps Mania" shows a botched domain transfer as a likely culprit for the low rankings/visitors. The redirect is not 301. Old pages do not redirect to content on the new domain ( http://www.mapsmaniac.com/2013/02/beneath-thunderdome.html ). So any links to mapsmaniac before the domain transfer do not contribute to the new blog at googlemapsmania.blogspot.com . Domain authority is zero'd out. Many well-ranking pages now show a "Page not Found" to Google. Then another site pops up on another domain with the same name. How can Google know for sure the two are from the same owner if the redirects are not set up right?

Fix the redirect. Restore the old links and redirect them to the new site. But maybe much of that is already too late for them. For anyone else in the process of moving a site, see: http://www.mattcutts.com/blog/moving-to-a-new-web-host/

We need to stop pretending Google is a rich Mozilla, They're a poor Microsoft.

http://ycharts.com/companies/MSFT http://ycharts.com/companies/GOOG

That's why Twitter, which is a tiny company with no profits and probably not much more growth potential, has a higher market cap than some insurance companies an retail chains with good growth, real money and solid long term market potential. "Feelings".

"What about Facebook? What about Twitter? I have no idea what Twitter is good for. But if it flips out every tyrant in the Middle East, I'm interested."

I guess it's just pretty hard to allocate resources in the traditional sense that most people are used to because of leverage that companies like google can provide, but people try anyways.

All in all, I think google is just trying to increase confidence from advertisers in it's content network.

[0]: http://www.cfr.org/technology-and-foreign-policy/technology-...

Which is hardly to say that you can unload the entire company at that price, or that it'll last, but it's a step above mere 'feelings'. :b

Meanwhile Google has done terrible things including ripping off bandwidth providers, and not to mention they hold a monopoly on search. conspiracy theorists seemed ill concerned on one organization responsible for answering all their web queries. could go on about outsourcing of jobs, h1b visas, causing a recession, etc etc.

not that any of these companies are 'evil' or even 'wrong' in my eyes, but why do others treat tech companies with such endearment and throw so much hate at a handful of other companies?

we're thinking geography has a lot to do with it. Many influencers on the east and west coasts know a lot of people that work in these companies, and write very positive things about them. not completely altruistic, as they are also heavily invested in them.