Big changes I can think of in the last few years that we've publicly announced:
* The index is several orders of magnitude larger.
* Most documents that you see in results can now go from crawl to serving in a matter of minutes.
* Ranking for the extreme head is pretty much SEO free.
* Improved ranking for long tail documents with little data available about them.
Those are just the really big launches I can remember off the top of my head.
Most of our day to day work in general falls into two categories.
1. Making our existing systems faster, fresher, higher quality. 2. Searching for brand new sources of data, most of which don't work out that well (like in any other research area.) and occasionally finding the diamond in the rough that dramatically changes search results enough for at least the SEO community to notice.
Can you explain what you mean by this?
They haven't done it with scraped links and content farms, but they have definitely optimised their business to do well in search engines.
Equally (and while I very much respect the work you guys are doing and trying to do) there are massive head terms where I know how much the top-ranking sites are paying for links. When they stop, they slip. The job is not done. (But you know that).
In general though, I agree with you, and it will get better.
it would be hard for me to say that the search results from google
have increased in quality at all in the past five years
Well, I know SQUAT about Google's search team. But just as a thought experiment, let's ask whether the 'net has changed in the last five years? It seems to me that whatever "algorithm" was in place back then would be gamed to death by now. Spammers would have all their dreck at the top of every search result. And that's not even counting the criminals who want their drive-by download payloads delivered...There's a constant war against spam and crime, and while Google has a stupendous amount of money to invest in winning it, the other side has an amazing war chest and a massive army fighting against them.
I have no idea, maybe it only takes an hour a week for one engineer to keep things ticking along. But I can easily imagine that the team has to do a lot of work just to keep results from getting worse.