SEO: When you cut through all the BS, the entire goal here is to make a less good match come first. And it works (sorta). Just consider crap sites like Experts Exchange that we've only learned about because they pollute many searches.

Feedback effect: Thanks to google, less people do less collecting of good links. Why bother when you can google for it? So there's less good information for google to use in ranking links. Bear in mind that when google started, nearly every home page had a long list of links to all the pages that particular user liked and frequently used. I used to have one; I've long since deleted it; my blog has some outgoing links that I like, but relatively few. If I twittered, I'd probably post a lot of outgoing links, but of dubious value; there's no gardening of just the perfect page of 100 links going on anymore.

(I think this also partially explains why some (generally more specialized, so less effected by other things) results feel dated -- legacy links that are still hanging around from days when links were still used that way.)

Feedback effect: Thanks to google, ten sites tend to be more important than any other sites on any given topic. This results in certain sites becoming increasingly important. Wikipedia is the chief example here. Why is there only one Wikipedia and not a dozen? Chiefly because it's gotten all the google juice. If you want your wiki article on foo to show up in google, you naturally write it on Wikipedia, not Fooipedia. The result here is that all google searches feel increasingly the same -- of course Wikipedia is always in the top ten, or maybe something like Stack Overflow for a technical search.

----

So, these days, if I don't see something interesting in the top ten, I often click on the link to page 10 (or 20, or 100) of the results. Often more interesting. For example, google for "mashed potatos".

Top 10 results: "Perfect mashed potatoes" (SEO), allrecipies.com (always in top 10 for any recipe search), foodnetwork.com, Wikipedia, about.com, nytimes, etc. Pictures of mashed potatos. All generic and useless.

Page ten results: Dairy-free mashed potatoes. _Potato_ free mashed potatos! Caramelized Onion Horseradish Red Mashed Potatoes! A poem about eating them. At least marginally more interesting and quirky. What I would have expected out of google circa 1997.

Now, if you want something "quirky", why are you searching for a generic term? What kind of "useful" result do you want from a search on mashed potatoes? If you give them a crappy search query, they should be giving you as generic of results as possible.

One thing I've found is that if you are looking for something specific, don't search for something generic. If you wanted something "quirky", why didn't you do "mashed potatoes quirky"? Then you get a restaurant that features mashed potatoes heavily in their recipes, a carmelized onion mashed potato recipe, a mashed potatoes festival, several more "interesting" recipes, and a book called "Grinning in His Mashed Potatoes".

It sounds to me like the results have improved, not gotten worse, if you aren't getting a poem about mashed potatoes on the first page of search results for just "mashed potatoes".

Until they implement it, you can use the CustomizeGoogle add-on for Firefox that lets you filter results out on your side.

I have started tailoring my searches in odd ways to help them out. Ex: Adding a the year when I want current results. But, without useful links it's all GIGO.

A related point: whoever first owns the data from all link aggregators (digg, reddit, mixx, etc) and all URL shorteners (bit.ly, tinyurl, ad nauseum), and weighs those results more heavily is going to have an awesome search engine... albeit better for entertainment than productivity.

i will humbly disagree. i think folks who browse the web are different from those who search the web. search is what gets you the most relevant results, therefore more opportunity for ad money.

Except I'd never find the obscure stuff that Google helps me find every day. I don't think a lot of links to manuals,mailing lists, etc show up on any of the sources you mention.

Google already does this, by surfacing youtube videos in many search queries.

That is not the entire purpose of SEO. There's good sites out there that don't provide their content in a way that can be indexed by spiders. SEO often solves that. There certainly are plenty of people making bad websites and trying to make them rank, it's Google's job to weed out the useless information.

People still collect links (shameless self promotion: http://internetmindmap.com ), they still have huge blogrolls, there are human powered search engines, a vast amount of directories for every imaginable niche...

Google doesn't need the perfect page of 100 links and I doubt it ever did.

Your mashed potatos example does not make sense. Google gave you generic info for your generic search query. How is that bad?

Now the fact that certain sites dominate a very wide range of search queries, is an interesting point. Personally, I would just add a sidebar or something similar, to be occupied by the "staple" sites, such as wikipedia, about.com etc.

One thing I noticed is that searches no longer require that all words in the query be present in the search results. Adding a + before a word is now required to ensure that it's present in results. That frequently results in me having to do 2-3 searches to find something that could previously be found with one.

This seems to be an optimization for their average user, but is really inconvenient for people searching for system errors, mathematical/cs theory terms, or other queries where acronyms are common.

User testing must have somehow played a role, I hate to blame non-technical users but... still I can't believe anyone gets better results when the keywords are optional.

The odd thing is I remember switching from AltaVista to google, before google you always had to discount the first bunch of results, but google was just so amazingly accurate. And yet now I find myself skipping the top results in a google search.

I remember when google bombing first started, it didn't bother me much, but then google tried to counter it and I could swear searches got a bit worse. And recently they've gotten even worse.

It's a shame this seems to be destiny of all truly great things.

Plus, google's handling of punctuation (e.g. f-script) is a pain since (even with the +) it will do weird substitutions and consider blanks good enough.

Last night I was searching for the syntax of DEFTYPE when used with various types (i.e. MEMBER, SATISFIES, OR, etc.) and the #1 his for "deftype member" was the personal MySpace page of some guy.

I think they're optimizing for "social" results now.

It would probably be a highly praised feature to seperate off a second index like that, as specifically searching for programming language concepts and documentation can be difficult (the C# and .NET problem).

And, yes, the results are different. I'd agree with you they seem enhanced (eg., classified results) and fresher for popular culture, but somewhat worse for domain specific queries.

Here is a search that I have had problems with :

octave "--eval"

Your site does pretty well with this (fourth link is somewhat relevant).

Google was basically filled with dead end forum postings and SEO spam.

DuckDuckGo was more helpful and brought me to the MS TechNet article with full documentation on MS Communicator Policy configuration.

Bing was surprisingly the most helpful and brought me to the Communicator Team posting from 2008 which shows me where I should have been able to find the logs.

It looks like my work has blocked/disabled this feature on a global setting even though I haven it enabled locally.

Previously it was equivalent to ("hyphenated word" OR +hyphenatedword). But now it seems to behave almost the same as the unquoted (hyphenated word).

Just to make matter worse, when I tried out my example just now I found that the first result (a wikipedia page) for "hyphenated word" doesn't even include the phrase!

Notably, you return a result for a specific osCommerce error message that I wrote about a while back (!); Google doesn't even know I exist.

You might try to track page age; some of the results I get are from 5 years ago and as noted by others, aren't always useful today. But that's a harder problem for another time.

On the more important metric of 'has quality gotten worse in the last couple of years', I would say 'sort of'. The direct quality of results HAS suffered, but on the other hand, google have implemented the user-wiki thing that allows you to modify, to a degree, which sites are less relevant.

I will add that I think google needs to rethink it's keyword fuzziness, in the past it used to be acceptable if the results didn't exactly match what you were searching for, but these days that is becoming more of a problem. If I search for a bunch of words, I typically know I want those words, by all means suggest 'did you mean ... ?' but the fuzziness in the results needs to be pulled back.

Seriously, the problem is advanced users get unexpected results with the query expansion and refinement layers they've added on. While dropping obscure words is helpful when grandpa has misformed queries, it's maddening for a technical user looking for a very specific, infrequent keyword. However for grandpa, it's probably a better experience for a generalized result.

Using +'s works well enough, but it's disappointing we have to use a less efficient method of querying now.

It does work as described! - which may very well not be as desired. If you want a more mathmatical-logical use of operators then you need to go find another search engine.

http://www.google.com/support/forum/p/Web+Search/thread?tid=...

I am afraid that Google is now more of a 'social search engine' than a 'hacker search engine'.

Regarding the query [Linux asynchronous IO] returning older results, here's a tip. Above the search results click the "Show options" link to open up what we call "toolbelt" mode. From there, you can click to show only results from (say) the last year, or in a certain date range.

Toolbelt mode is really handy, e.g. if you search for a product, you can click "Show options" and then click the "Fewer shopping sites" link to get more reviews and manufacturer pages instead of comparison shopping sites.

?

Although I alternate search engines regularly, I do think the way Google indexes for specialized searches is pretty smart.

Only 5 topics, unfortunately.

Thanks for the link!

http://www.reddit.com/r/programming/comments/9s6p2/google_is...

It is starting to get incredibly difficult to do specific searches (such as for documentation). It also seems that SEO have become incredibly successful in gaming Google.

It would actually be good if there were some competition.

Strange that with regard to current events the web seems to have little historical memory, and yet with regard to current technology it has too much.

http://news.google.com/archivesearch?q=2002+house+vote+socia...

Dropping keywords also annoys me. If the keywords don't exist then tell me that so I can adjust my search. Don't give me a long list of results that I have to click through before realizing you screwed up the search.

The only reason the google search bar is still my default is because I use it as a quick and easy calculator.

I've been doing home remodeling a bit lately, and it's clear to me that there are NO home remodeling linkerati. But there are plenty of SEO guys out there and it only takes a few low quality links to top a lot of searches. So search for home remodeling stuff and you see plenty of adsense spam.

They should drop yahoo answers, any site with an affiliate link, and any of the internet marketer sites ( like ezinearticles.com ) from their index

My friend had an idea to make search rankings go by the number of ads on a page, the less ads the higher the ranking. I guess Google would never do that because of their adsense program.

FWIW, Bing nailed it on the first query.

http://blindsearch.fejus.com/

I was amazed to discover that I was consistently choosing the blind results from Yahoo! as best for my own searches.

"Hashmap", the term you searched for, is a term popularized by the Java world; I think most C programmers still call them "hashtables". (Hash map is a better term, as many excellent map implementations aren't actually based on hashes; see "Judy" for example.) A search for "C concurrent hashtable" gave me a lot of useful results. (You are also suffering from C's lack of a coherent community here. Lots of people write this sort of thing, but few think to share it. Hence, not many search results.)

For those of you who have been forced to use + or quotes to get google to return what you want, we would love if you would post the specific example queries on this thread. (Some people did, but a few of the complaints were vague.) Also, in general you can click the 'Dissatisfied? Help us improve' link in the footer of google search results pages to report problems. We check those out, but if you wanted to include the word "hackernews" in the info you submit, that would let us see all your reports easily.

One thing I have noticed is YouTube's search is dreadful. It's embarrassing for a company like Google to not offer an adequate search engine on their own site. The grace period of the YouTube purchase is long over. I shouldn't have to retype my search 5 different ways to find what I'm looking for. These days I've just given up and search for simple terms and look through 10 pages of results or try related videos until I find what I'm looking for. Not super impressed with Gmail's search functions either. My mailbox is mirrored on Gmail and my local mail client (Apple Mail) Mail will almost always find what I want when Gmail gets confused. Apple isn't even a search company. Google can do a lot better.

Before google got so effective people put more work into maintaining collections of links to other sites; hand-curated omnibus directories like yahoo and dmoz or narrower fields.

As search became better this was increasingly a poor use of time; you could, after all, just go and google stuff to find it again, so why bother maintaining a hand-curated set of favorite links.

A consequence of this is that link-actions-taken-for-reasons-consistent-with-pagerank's-assumptions dropped, and reducing the signal-to-noise ratio insofar as pagerank is concerned; similarly link-actions-taken-for-reasons-contrary-to-pagerank's-assumptions increased (SEO), further dropping the signal-noise ratio.

Even extremely robust SEO-detection techniques can only cut away the increase in noise; now that people are less incentivized to publicly post high-quality hand-curated link collections there's less for the algorithm to go on.

This is why there's talk of twitter and fb vis-a-vis search; twitter and fb are where the info the search engines need currently lives.

(!) I know they use additional techniques these days but whatever they're throwing into the mix doesn't seem to be counteracting the overall utility decay.

I think in future search engines should lean a particular way depending on your profession - "Google for Medical Personell", "Google for Architects", "Google for Programmers".

I guess a single keyword search is not going to get anything sensible out. The answer lies in being able to interact with such systems and get what you want. Solving the search/retrieval with a single query is a very hard problem as it depends on the context of the user. Try searching for the word "Execute". Does it show pictures of a person being hung or a how-to for the first hello world program in a terminal?

on a similar note - http://guillaume-nargeot.blogspot.com/2009/08/think-twice-be... (probably saw this on HN)

"The Modbookish" is an inconsequential six member Ning social network, but doing a Google search on "Modbookish" is interesting. There are the annoying 'Google thinks you are stupid' hits like the site in which there is a sentence ending with "mod" followed by a sentence beginning with "Bookish". The remaining results are sites that link to the Modbookish. However, the Modbookish itself is NOT in the results. As Google ads are on the site, there is no doubt that the Google bot visited it shortly after it went online. Nonetheless, Google has refused to put the site into its web index for a couple months now. Clearly, (and not without cause), it is assuming that the pages that link to the site are link spam. It is probably waiting for some semi-authoritative link to appear before it indexes it.

So, why is this bad? It undermines Google search purpose: to provide the most relevant results. When Google got started, I recall that when Sergey or Larry were asked why Google search was better they often offered the Harvard example. If one enters the search term "Harvard" with no other terms into a web search box, there is really only one reasonable search result: the Harvard University website. However, in the search engines of the time, the university often ended up on page five or six. On Google, it was search result number one. By the same token, if you enter "Modbookish" (no spaces) there is only one reasonable result, and Google doesn't offer it. In contrast, both Duck-Duck-Go and Yahoo list it in the first spot.

Why is this really bad? Many (Most?) websites do not do their own site indexing; they let Google do it for them. Clearly if Google refuses to index a certain important but only occasionally referenced page, that is a problem.

http://www.google.com/search?q=linux%20asynchronous%20io&...

'Linux async io' -> select 'show options' and then 1-year view on the LHS.

Sadly for the concurrent hashmap I got to 'concurrent hashmap "c" -java', then saw this very discussion page at #3 and I'm in an infinite loop until further notice.

http://www.cederman.com/2009/09/google-continues-to-disappoi...

Your "Red Room" query is hard in a couple ways. First, it looks like that root page used to have the words on the page: "The Red Room Doors open 6pm $18 Pre-Booked" And it's also tough because it looks like the name changed to the "2nd Degree Bar & Grill" at some point. The fact that you can type [red room] and get a suggestion for [red room st. lucia] is actually pretty helpful in my book because it leads you to the answer that the name changed.

I actually did a search on "recency" first, and Google didn't correct me and the first hits were a couple of dictionary entries, which is why I searched for the longer query. I don't know why, but I guess I figured there might be a blog post perhaps complaining about Word lacking common dictionary words and marking them as misspelled. In hindsight, including "recency" in the query was silly, but I'd prefer to get nothing back instead of Google assuming I want something else, which is the issue I was trying to highlight.

I agree that the suggestion of "red room st lucia" is a good one, and it helped me find what I was looking for! However the problem I was trying to show was Google's new approach at suggesting a first hit without search terms you've entered. As another example, I was trying to find my old (and embarrassing!) Geocities page recently, so I searched for [tim cederman geocities] and the first hit was my own blog.

In my experience Google has not kept up with the assault of companies trying to beat its algorithms!

Let me guess:

1. Google's Pagerank patent is about to expire, and they have to make something new?

2. New ranking algorithm adjustment to compete with Bing but failed?

And similar discussions on reddit:

http://www.reddit.com/r/programming/comments/9qro1/

http://www.reddit.com/r/programming/comments/9s6p2/

I find myself alternating between google and bing these days, for some technical searches sometimes bing works better (especially - and unsurprisingly - for Microsoft-related technologies).

Some competition from bing may lead us to a better google (just like competition from apple made microsoft's new OS better).

However, I do think it could be possible that the pagerank algorithm does not scales good enough for the growth of Web pages, and especially with the "SEOs" out there always trying to come up with new ways to game the system.

That being said, I do feel often like Google's search result quality is decreasing. Bias due to using it exclusively perhaps?

And it's duckduckgo.com that helped me.

My behavior has changed slightly though. Increasingly I'm using twitter, reddit, delicious, & hacker news to search and browse for content. Google site search for Hacker news has also come in handy.

It's possible that Google doesn't give you perfect results, merely the best of a bad lot. If so, it suggests the problem is the web, not the search of it.

And the web certainly has changed.

http://www.google.com/#hl=en&source=hp&q=site%3Avldt... http://www.bing.com/search?q=site%3Avldtr.com

As long as the Google has the most up to date index there is no good alternative for me.

You mention trying Yahoo. Did it give you better results?

See also here: http://news.ycombinator.com/item?id=464393

Another example: when I search for "vldtr is now a couple of months old" (from my post up there) Google returns this thread, Bing returns nothing.

Different metric, same result.

Edit: redundancy