If the results suck, by all means, rail on Google and tell us we're doing a crappy job, but there's no need to suspect an ulterior motive.
Every so often, some well meaning salesperson will email one of the search quality mailing lists because they think there's a problem with how we are indexing/ranking one of our Adsense or Adwords customer's sites. Every single time, I've seen the person sternly told by a VP to never email those lists again.
However, once such revenue-enhancing feedback loops exist, they can lead to self-reinforcing behaviors without any conscious intent. Simply the fact that Google has been so wildly profitable, even with search quality stagnant or declining for many spablum-heavy queries, could mean certain radical avenues of inquiry aren't considered. ("We're obviously doing OK overall, our search share, usership, and profits are up every quarter. Let's not rock the boat by starting a war with these content mills, which would surely include lawsuits and congressional hearings calling attention to our market power.")
Even the choice of overall budget for search quality comes into play. The current level of spending has, with its mixture of both qualified success and failure against spablum, done just fine for Google's margins. But I think Google could spend twice as much on search quality, and still be profitable. So, why doesn't it? Well, I don't blame Google for being profit-maximizing, but that means someone is controlling a relative-effort lever that makes search just so good, and no better, because of a tradeoff in which the marginal profitability of even better search is considered.
Also, what if Google's search quality gets monotonically better given the web it has to work with, but the overall economic impact of Google's ad programs and dominant-focal-point-rankings is simultaneously making the average web content worse? Then, each generation of Google's search tech may be better, ceteris paribus, but the net pollution effect still boosts Google's revenues via the three mechanisms I listed: longer search sessions, greater attention to paid areas, a more difficult environment for competitors.
Even your anecdote is not completely reassuring; for an AdSense/AdWords customer, it's not in Google's self-interest to favor those sites in natural results. By ranking highly in organic results, they might not need to buy as much paid placement! So that VP's stern warning can be explained by either a dedication to search independence, or to simply ad-revenue-maximizing self-interest.
Will I ever get an advanced search operator that will filter all AdSense-carrying results out of my searches? (I'm going to make one on Blekko, if possible.)
You do have a point there. We do get sued a lot, and in general courts have upheld our view that search results are our constitutionally protected speech, but we are a lot more lawsuit shy than we used to be.
>But I think Google could spend twice as much on that search quality, and still be profitable. So, why doesn't it?
I'm not sure it would help. We have a _lot_ of people working on search quality, and even more working on the infrastructure that supports it. More people would certainly let us scratch more itches and peer in more dark corners, but I'm not sure it would produce the kind of dramatic improvements you are looking for.
>Also, what if Google's search quality gets monotonically better given the web it has to work with, but the overall economic impact of Google's ad programs and dominant-focal-point-rankings is simultaneously making the average web content worse?
That's something a lot of people worry about. I hope that having a financial incentive to make content results in more good content, or at least more content that isn't horrible. Even the existence of content-farms is probably a net positive. For all the crap they produce, a lot of it is filling niches in which there simply isn't any content, and the stuff they produce _is_ better than nothing.
>Will I ever get an advanced search operator that will filter all AdSense-carrying results out of my searches?
You really want the New York Times filtered out of your searches?
These indirect mechanisms for avoiding quality when it conflicts with profitability may not seem malicious on their surface. In fact, here's what they might sound like instead:
I'm not sure [spending more on search quality] would help.
Even the existence of content-farms is probably a net positive. For all the crap they produce, a lot of it is filling niches in which there simply isn't any content, and the stuff they produce _is_ better than nothing.
And yes, in many searches, I would gladly sacrifice NYTimes results to get rid of EHow and its siblings. (I didn't ask to eliminate AdSense sites from every results-page, just an option to do so when I'm in a MFA-polluted category.)
Something like so:
how to make blueberry pie -adsense
edit: didn't notice that previous commenter had said the same thing and moultano had replied "you want the new york times filtered out of the results?"
No, I want an advanced flag that filters ALL sites with adsense out when I use it, including the New York Times. If I want results from NYT then I won't use the flag when I search. He didn't exactly answer the question of will we ever get a filter-out-adsense flag
Big changes I can think of in the last few years that we've publicly announced:
* The index is several orders of magnitude larger.
* Most documents that you see in results can now go from crawl to serving in a matter of minutes.
* Ranking for the extreme head is pretty much SEO free.
* Improved ranking for long tail documents with little data available about them.
Those are just the really big launches I can remember off the top of my head.
Most of our day to day work in general falls into two categories.
1. Making our existing systems faster, fresher, higher quality. 2. Searching for brand new sources of data, most of which don't work out that well (like in any other research area.) and occasionally finding the diamond in the rough that dramatically changes search results enough for at least the SEO community to notice.
Can you explain what you mean by this?
They haven't done it with scraped links and content farms, but they have definitely optimised their business to do well in search engines.
Equally (and while I very much respect the work you guys are doing and trying to do) there are massive head terms where I know how much the top-ranking sites are paying for links. When they stop, they slip. The job is not done. (But you know that).
In general though, I agree with you, and it will get better.
it would be hard for me to say that the search results from google
have increased in quality at all in the past five years
Well, I know SQUAT about Google's search team. But just as a thought experiment, let's ask whether the 'net has changed in the last five years? It seems to me that whatever "algorithm" was in place back then would be gamed to death by now. Spammers would have all their dreck at the top of every search result. And that's not even counting the criminals who want their drive-by download payloads delivered...There's a constant war against spam and crime, and while Google has a stupendous amount of money to invest in winning it, the other side has an amazing war chest and a massive army fighting against them.
I have no idea, maybe it only takes an hour a week for one engineer to keep things ticking along. But I can easily imagine that the team has to do a lot of work just to keep results from getting worse.
What if spablum doesn't prevent a person from eventually finishing their task, but lengthens their browsing/searching session, requiring a few reformulations? Up to a certain threshold -- when the person gives up entirely or tries a competitor -- such delaying tactics may give Google more AdWords and AdSense impressions and clickthroughs.
Next, as the quality of natural search results drops, the value of the paid-placement slots goes up -- for both advertisers and users. This is one giant hole in the argument that it's effortless to switch to another search engine, if it offers better natural search results. Lots of the value from Google searches now comes from the relevant ads -- only if Google were to let competitors run ads from the same pool would this switching cost be eliminated.
Finally, the quality arms-race is something deep-pocketed, heavily-PhD'd Google can fight indefinitely... but adds an immense barrier to new shoestring operations. Google, given its position, would be perfectly happy if search quality just hovers where it is (or even declines slightly) -- as long as the cost for potential competitors to achieve that same search quality, at scale, grows moreso than for Google.