Google algorithm change launched
Earlier this week Google launched an algorithmic change that will tend to rank scraper sites or sites with less original content lower. The net effect is that searchers are more likely to see the sites that wrote the original content. An example would be that stackoverflow.com will tend to rank higher than sites that just reuse stackoverflow.com's content. Note that the algorithmic change isn't specific to stackoverflow.com though.
I know a few people here on HN had mentioned specific queries like [pass json body to spring mvc] or [aws s3 emr pig], and those look better to me now. I know that the people here all have their favorite programming-related query, so I wanted to ask if anyone notices a search where a site like efreedom ranks higher than SO now? Most of the searches I tried looked like they were returning SO at the appropriate times/slots now.
(
183 comments
| 688 points
| on HN
)
I see filestube's auto-generated search listing pages ranking on Google all the time. Pages like: http://www.filestube.com/m/matt+cutts http://www.filestube.com/g/google+scraper
Same goes for freshwap: http://www.freshwap.net/387/dl/Google+Matt+Cutts
These sites will give out an auto-generated page for every keyword you enter into it. Apparently, Google loves to index them... there are 126 million pages of files tube indexed in Google. I thought indexing search listing pages of other search engine was against Google's policies.
You threw a vitamin pill into a bucket of mud?
http://www.google.com/search?q=nstoolbar+bottom+bar&ie=u...
You'll notice that efreedom.com shows on the first page with content taken directly from stackoverflow. While stackoverflow does show in the results, the exact page that efreedom copies does not. Anyway, I'm glad you guys are taking this seriously.
For reference here is what I see right now - http://dl.dropbox.com/u/1437645/googlesearchresult.png
It seems like just showing a few more results from the "real" site would solve the problem.
#2) stackoverflow.com/questions/3977343/nstoolbar-on-bottom-of-window
#3) stackoverflow.com/questions/tagged/nstoolbar
#9) efreedom.com/Question/1-3977343/NSToolbar-Bottom-Window
The #9 result is copied from stackoverflow.com/questions/3977343/nstoolbar-on-bottom-of-window, which is the same as the #2 result.
This actually is really hard to do as well as Google has given that the stackoverflow page doesn't have the word "bar" on it anywhere, while the efreedom page does.
Neither Bing nor DDG seem to have either the StackOverflow page nor the Efreedom page. I would give an uninformed guess that they probably can't match the StackOverflow page because it doesn't contain "bar" and are probably doing some sort of hand demotion/removal for efreedom. As a result, you get no pointers to the information you want instead of two.
Um...
Two SO links on the first page above any efreedom urls is a positive thing, but even so, if content is apparently considered good enough to make it to the first page, it should come from the original source rather than a scraper.
1. The SO results shown don't address the query I submitted as well as the efreedom one does.
2. Why doesn't the stackoverflow article that is being duped show in the results?
http://www.learn-to-hack.com/
Several of the other results are rather dubious as well. The reason I bring it up is because the Squidoo lens that comes up is something I made, and while certainly not perfect it's still a much better than many of the SEO spam sites and fake eBooks that rank above it. (And plus the ad revenue is going to charity rather than some shady organized crime ring.)
Anyway sorry if it's a faux pas to complain about my own stuff, but I feel like it's a legitimate problem with the way Google works.
Definitely not a faux pas. Thanks for the example. My biggest annoyance in threads like these is people who write essays about their site losing traffic but then aren't willing to provide an url for people to check out.
Don't flag this because there's no link or "citation". Matt Cutts is the web spam guy at Google.
I noticed today that a search for "mubarrek london" returns a page of results where every result on the first page, besides the top one, is spam from www.88searchengines.com, www.30searchengines.com, www.70searchengines.com, etc.
I know this might not be related to topic of scraper sites directly, but not sure how else one can easily report these types of things.
http://i.imgur.com/kDwPd.png
Are you sure there isn't something else going on?
http://www.tristanperry.com/pics/SpammyResult.jpg
Searching from the UK, if that changes anything.
I get similar results as he does, with personalization, and, in Incognito mode.
Following query is from google.com and contains no efreedom on the front page. http://www.google.com/search?hl=en&q=gcc+optimization+fl...
Now, the same query from google.ie (ireland site) contains 2 efreedom on the top page!
http://www.google.ie/search?hl=en&q=gcc+optimization+fla...
Why this strange search behavior to a query which has no relevance to user's location?
P.S. I was logged on to my google account while searching, not sure if that has any effect whatsoever.
hl refers to the UI language and gl to localized results.
So, does this mean that Google search now takes into account my past search result clicks (or rather mis-clicks)? Or ranks contents in someway that proves efreedom is somehow more relevant to me?
Is deleting them from search history enough to correct this effect?
Also, I cleared my web history and searched again. The results are same now signed in or signed out. Problem solved! :)
Thanks!
In the future is there going to be some way for webmasters to do something like rel="canonical" across domains so if I want to syndicate a piece of content across two properties I own I can indicate which one is the original source? My understanding is that rel="canonical" is only meant to be used between pages on the same root domain today but I could be mistaken.
Is Google the only engine that supports it currently or is it standardized?
As for the long list of crap between the question and the answer, use AdBlock to remove that various DIV's.
They used to be scraped from usenet. They may well still be, but I don't have any current evidence. Are they still doing that?
Create something like:
http://www.google.com/search?q=google+-google.com&ie=utf...
I suspect you could hack together a quick greasemonkey script to automatically inject your negative site list when the URL is generated
query: Mailengine with .NET API
The right page that should show up is
http://stackoverflow.com/questions/1720900/mailengine-with-n...
If you google the link below you'll see that the page is indexed.
http://stackoverflow.com/questions/1720900/mailengine-with-n...
However, the page doesn't show up when you use the query I mentioned.
How about sites that rank well with no content, just navigation? Here's an example:
http://www.collegegrad.com/entryleveljob/entrylevelaccountin...
It's generally a high quality site, but that page has absolutely no relevance to the query except for a title tag and some internal anchor text. The search terms aren't even on the page.
If I remember correctly, it used to rank #1 for "accounting entry level jobs," and now it's down to #8. My question is why is it even ranking at all? It's not even low quality content. It's no content.
The motives here don't offend me.
I'm not trying to "out" this site. I don't think they've done anything wrong or manipulative—they just have some pages with no content that are ranking well. I think that this type of issue should be on Matt's radar. Most of the site is of extremely high quality, and it deserves the high rankings that it gets.
It's a page that is solely navigational structure, yet it ranks very well for a very specific keyword because of title tags and anchor text. We've already seen that Google has a problem (that they're working towards fixing) with low content from low and medium quality sites. What are they doing about low quality content (or non-content) from high quality sites?
http://www.google.com/#q=major+online+dating+sites+koopa&...
We submitted an article to ezinearticles from our blog.koopa.com. Just wondering how our blog is not remotely listed but ezine which we posted are article through our blog has? How does this work when our authors submit to article directories?
On that same note I notice a bunch of other sites just below which have copies ezines article to the tee and are ranking higher then the our original blog that posted it.
Is this related to the current algorithm change or just that our blog may not be indexed yet? Thanks for the update once again. I love the fact that you and the google team are constantly updating and changing your algo to give value to rightful content owners.
Example: http://www.google.com/search?q=Any+yii+way+to+get+the+previo...
In that particular instance it is result number 3 (and the original is number 1), but on more than one occasion it was the top result and it's never what you are looking for.
A major usability problem for me with auto-corrected results is the two lines with links: "Showing results for <link>" and "Search instead for <link>". Interpreting those two lines takes me out of the flow of analyzing search results. I haven't yet been conditioned to click on the second link. I never want to click on the first link, because google is already showing me those results.
I frequently search for unusual acronyms, terms, variables, made-up words, and gibberish that shows up in logs, among other things that google likes to "correct" for my benefit. I realize those use cases are not typical, but current auto-correction behavior can be extremely aggravating in those cases.
Auto-suggest a correction all you want, but auto-correcting and suggesting the original is going too far I think, unless there are zero hits, or some equivalently strong algorithmic determination is made that I couldn't possibly want the query the way I typed it.
Following on from one of the comments here, namely the idea that "value is in the eye of the beholder", I'd like to raise our own plight. I run a number of aggregator sites - the largest and oldest of them being celebrifi.com, which was a PageRank 5 until Google de-indexed us in December (along with some of our other sites, but interestingly not all).
A little background - the purpose of the sites is to aggregate, organize, rank and add context to what's happening in the news, with each site focusing a specific vertical. Think Techmeme, but with more context.
I'll be the first to admit, there is no original content, but I strongly believe that we "add value" by figuring out what exactly is going on in any given story or blog post.
We add value to publishers, by always linking to the original source (indeed, many publishers directly request that we add their feeds to the sources we track), we respect copyright by only displaying a short snippet of the original text and only displaying thumbnail images and we add value to users by giving them easy access to a lot more content on the same topic/story, all in the same place.
Google's Quality Guidelines clearly state that duplicate content is penalized, and that is totally fine with us, but is it right to totally de-index a site for duplicate content? I wouldn't even want to rank above the original source for any given piece of content, as I respect the hard work that writers and publishers put into creating quality content, but aggregators who add value have a role to play in the content ecosystem. Digg, for example, uses the "wisdom of the crowd" to aggregate and rank content - hence adds value. Topix takes a local approach to aggregating content, and uses comments to rank content - hence adds value. We take a verticalized approach to aggregating and ranking content, and hence I believe that we add value.
As mentioned above, we got de-indexed in December, and despite going through the appeals process, fixing a few things on our end to do with sitemaps, and clearing out some older "low quality" sources that we were tracking, we received no clarity into what our crime was.
Matt, I'd like to raise this issue with you - both as it relates to us, but also as a general industry question - are all aggregators going to be de-indexed? And if not, which aggregators are and which aren't? What is the criteria, and who decides? If its algorithmic, then I am very curious to know what on our sites triggered the de-indexing? And even more curious to know why some of our sites got de-indexed, and some didn't.
I have great respect for Google's efforts to clean up spam and low-quality content - and would always expect to see original content ranking higher than aggregated content. But to completely de-index an established aggregator site and strip it of its PageRank seems very draconian.
I would love to hear your/Google's position, and look forward to some more clarity on both our situation, and the future of news/content aggregators.
Respectfully yours, Niko
Presumably, aggregator sites by themselves also help in content discovery. I find a lot of content through Hacker News. But they should do so by being good enough to be a destination in themselves. An aggregator that needs to be found by search engine isn't doing users any favors.
Google News provides snippets of content, and helps people discover the news providing direct links to the original source of news. For Google to deindex a site like celebrifi and while running a competing product (Google News) smells a bit of monopolistic behavior. It's suspect it's unintentional, but Google is going to have to walk a very very fine line as you start deindexing certain sites.
Google News itself falls into the second category - an aggregator that stands on its own, as a destination. Much like Hacker News. I personally don't use it, but the people that do go because they find it lets them discover a bunch of content that they otherwise wouldn't know about.
Google News does stand alone as an aggregator, but you have to admit that it is promoted heavily by Google search. If GOOG keeps doing stuff like that, I suspect there are going to be a lot more companies that start to take umbrage, and start challenging this behavior in court claiming that it's anti-competitive behavior.
I go to Google because it provides value to me. Aggregator / republishing sites trick you into visiting them with the lure of what looks like actual, original content. This makes the internet less useful because you end up reading the same content over and over.
Of course original content has priority, the original poster explicitly agreed with that and there are some site who do produce no value at all, but saying every site who purely uses other sites content are all ruining the internet is obviously very wrong, considering this is google we are talking about.
Example would be my wife's company, which is a large non-profit science center that helps educate kids and adults about science and has a wonderful website that they pay a lot of money to maintain and support. However, Google doesn't see any problem with the fact that they've scrapped my wife's company website to get their contact data, hours, and a few other points that they then put on their own page (Google Places - whatever that is), along with their own Google ads, as well as link to my wife's company competitors all for the purpose of keeping visitors on their site so that they can get the ad dollars while adding no additional value of their own.
To me this is the biggest scam in the world and makes them look like giant lying crooks on the web because they steal content from others, while banning competitors that do the same thing and all the while telling people that they doing it for the good of the web.
Yeah right?!? Google is doing it for the good of your pocket books. Here's a thought, why doesn't Google start paying sites, or splitting profits with the sites that they scrap data from and then make money off of. They are stealing visitors, traffic, and ad dollars while providing no additional content or benefit to the reader. I believe Google is getting really close to anti-trust issues and should be looked at very closely by the US Gov. They broke Ma Bell up in the 80's because they got too big and I see something like this coming down the line for Google.
People are starting to get concerned that their hands are dappling into many areas that cross-support each other, putting a lot of control and power into one company's hands. Google is not the Internet, they make no original content of their own, they regularly police the web and try to tell us what is relevant or not, similar to a dictator in a closed society, and they make money off all of this so of course it's in their best interest to keep it going and make us think we need them to show us the web. Truth be told, I can find anything I'm looking for on any engine, and often I find good and bad results on all of them so to me Google is just another search engine and if they keep going down their slippery slope they will wind up like Alta Vista.
Here's a suggestion. Drop your bunk PR and link structured algo and develop a new tool that doesn't support spamming the web with paid links. Google created this monster when they put so much emphasis on links and people all over the web buy and sell links to push their rankings up. This model is so old and out-dated that it doesn't make sense. Now Google is putting band-aids all over their old bunk algo trying to keep their out-dated ways going instead of putting in the time, dollars, and effort on making a new model that will actually produced good nature results instead of manipulated results that have to be regularly policed and manipulated to keep it going. The fact that Google has to regular update their algo shows that they know there are problems with it, but they just keep sticking more patches and band-aids on it instead of building a new model that will not support the spam techniques on the web.
All I can say is CONGRATS on putting another band-aid on the OLD ALGO. Maybe this one will only effect a small number of legit sites and, heck, those are casualties of the war on spam, right?
Good luck to all the legit sites that are trying to jump through Google's hoops! Just remember, if you don't make it through today you will be gone tomorrow.
Also, we do actually produce some original content, we have a small staff of writers who create "featured posts" every day - sold ourselves a little short there.
You guys have it, but it's buried under a pile of spam. Lose the spam and you'll recover your integrity.
However, the debate is whether or not these aggregators sites should show up in search results instead of the original source.
I use Techmeme all day long, but don't expect it to ever show up in a search result when I am looking for a specific topic. I want to the original site. I can always go to Techmeme directly to get the "value-added" context on my own.
Aggregator sites are great destinations. They shouldn't be in search results (at least, not above the original sites).
are you talking to the parent post or matt?
/sarcasm
Aggregators, it's time to wake up. Google is not your friend, Google is your competitor.
We can agree that Bing and Google are competitors, right? And we can agree that Bing labels themselves a "Decision Engine" and not a "Search Engine" right?
The line between a Search->Decision->Topical Aggregation Engine is so blurry I'm not sure it exists anymore. Google's stated mission is to "organize the world's information." - they are going to add social and they are going to use your preferences to figure out which stories/pages are interesting to you.
Looking at your site, it's easy to see why Google bumped you down:
http://technifi.com/
http://technifi.com/news/Egypt-Leaves-the-Internet-3841691.h...
It looks like your service is nothing like Techmeme, adding ad-related pages prior to accessing significant content. And the context-aware parts seems to lack any form of editorial choice, such as the Google-image Vodafone photos. In short, you have no insight, let alone opinion, and thus add nothing of value to the information.
Now aware of your site, I don't find any benefit and would ignore it or add it to my block list in my custom Google search.
1. We don't expect our SEO to outrank the original source, I was very clear about that in my original post. SEO is a tool to be used among many other tools to ensure that content is properly "classified", nothing more than that. When a search engine visits, you want that search engine to immediately know what any given page is about, and we do that very well.
2. The Vodafone image concern I don't understand - we identified Vodafone as one of the main entities in the story (Vodafone shut off service to Egypt), and that is why the image is showing up there. If you visit the Renesys blog (the original source) you will see that Vodafone is mentioned there. The Vodafone image is not an ad, it is there to identify what the story is mainly about - context.
3. Regarding adding insight, opinion or value - we believe that the role of algorithmic news aggregation is not to have an opinion, but instead to uncover news that you may not otherwise have been aware of, give you context in the form of links to the main entities in that story, or other stories on the same topic. We believe we do that very accurately, and are always working on making it better and smarter.
4. Regarding the comparison to Techmeme - everyone has their favorite aggregator, and I am a big fan of Techmeme as well. I would, however, point out that Techmeme also displays ads in order to make a living, and they also have "content pages" that you can find through Google search. Indeed, virtually all aggregators run ads against the aggregated content they display on their sites, and all aggregators have "content pages" above and beyond a homepage.
The issue I am trying to uncover is not whether you like our sites or not, but rather what exactly we have done wrong, when compared with other aggregators, that caused us to not just have our pages bumped down in ranking, but to be totally de-indexed and have our PageRank stripped away (from a respectable 5 on Celebrifi). That, and to understand better what the future of content or news aggregation might be - should we expect all aggregators to become de-indexed? Are there guidelines that should be followed, or changes that should be made, to not fall afoul of Google? Are all aggregators on a level playing field, or will the lesser-known ones be shut off while the chosen few with established brand names survive?
These are valid concerns, not just for us but for many others in this space, and we are more than prepared to put in whatever changes might be needed to get back into Google's good books.
The big aggregators will win and rise to the top. The knock-offs will be de-indexed. There's only so much room for news and gossip sites that don't add value, don't take it personally just move on.
On the other hand, if I wanted to see an aggregator site like Huffingtonpost or Digg, then I could Google for a site like that, or even Google for a keyword about a discussion that happens on the aggregator site. That seems legitimate to me, but showing up when I'm googling for something related to the original source is obviously not.
It's unfair to be in the results page with somebody else's content when the people searching for that content are actually targeting the original source, not yours.
That said, I'd be careful of analogizing aggregators to wire services. The AP actually employs their own reporters.
IMO.You are kind of hiding the original source and do make it hard to find. Yes you show the source and your call to action is "Read More" but your site does not give the appearance of a news aggregator.
There's no specific queries/websites talked about there (I might be wrong but I think Webmaster World has some rules preventing specific discussion of websites/queries), although I thought I'd flag it up since some webmasters have noticed adverse affects from a Jan 26th algo change; and it sounds like this might be the cause.
Anywhoo, that being said: it's great to see Google continuing to be on the ball and responding to the recent feedback from various blogs and other sources (e.g. here).
This is great news, but I have to wonder: How do you figure out which site wrote the original content?
I'm wondering based on what happened to Tarsnap last week, where hackzq8search.appspot.com outranked tarsnap.com on a search for "tarsnap".
mysql spatial index example lft rgt
I see the following order:
1: http://planet.mysql.com/entry/?id=23512
2: http://efreedom.com/Question/1-1743894/Mysql-Optimizing-Find...
3: http://explainextended.com/2009/09/29/adjacency-list-vs-nest...
4: http://stackoverflow.com/questions/1743894/mysql-optimizing-...
If the algorithm properly detected site relevance, importance and viewer satisfaction, those copycat sites should never have ranked higher in the first place. In a way this is admitting that it is impossible to stop the gaming of search engine optimization, and that the only way to deal with it is to "win" in some special cases.
That being said I provide no real solution, this is a huge problem with millions behind each side.
Although this is good news it is also gloom news.
Stack overflow is an obvious benefactor of this new change, I'm just wondering if smaller content providers might benefit as well?
Now I am always on alert when clicking on a link in the results to avoid the spam pages like efreedom, expertexchange and whatnot. Why not just remove them from search results?
They are causing enough bad publicity to Google.
To clarify, this change measures unique content site-wide, not just for a particular web page/url?
I may be too focused on semantics, but it would be important if you are trying to maximize visibility for a single page in Google's search results and your other pages have a significant amount of duplicate content.
For instance, one page on your site is about a WordPress plugin you've created (it's totally unique), but on most of your other pages, you've copied and organized relevant sections of the WordPress Codex so that your users can easily find the documentation needed to customize the plugin. Is your unique webpage about the plugin safe? I was always under the impression that search rankings were determined on a page-by-page basis rather than site-wide.
P.S. Sorry if this was already asked and I missed it.
What if you have a blog with lots of quotes from other sites; will that hurt your rankings because Google sees "unoriginal" content?
Is there some ratio of original to unoriginal content that must be met to keep from being flagged as a scraper?
for me, the 10th result is:
www.questionhub.com/StackOverflow/3910933 - Cached
I think it is safe to say they are scraping, and, while there are some SO answers, that particular answer does correctly interpret what my intention was.
In fact, the SO page scraped shows up as the 12th entry.
Examples: Demand Media in general. Or say w3schoolS.com for JavaScript/CSS/HTML.
I can understand that there would likely be a deluge of legal action against Google if this was done in a heavy handed way.
Or is it a principled thing where everyone should be treated equal? If so isn't that a lot of algorithm ideology. In practice hard rules without human judgment leads to ridiculous edge cases or bypassing the intent of the rules.
I always assumed you curated in a way via having teams that specialized in creating topic specific search techniques which then all get combined together via topic detection or some other meta algorithm. This would be a good balance between manual curation and having it be a scalable approach.
http://www.bay12forums.com/smf/index.php?topic=73045.0
Apparently, for a short time (until the admins fixed it), a wiki for a game I play was hijacked to spam Google. Actually, said wiki (magmawiki.com) is under attack right now from spam user accounts. I don't run the place, but I thought I'd pass the information along because it might give you something to examine and because I have to imagine that anything that helps Google identify spam-hijacked wikis would be helpful to everyone.
I've seen many sites use this method of adding text to an image heavy site and have watched one particular site that uses this method of adding text, drop dramatically out of the search results since October 28th and again around Dec 28th.
Would you advise to discontinue the use of wikipedia content even though the targeted keywords differ from those of the actual wikipedia page?
http://goo.gl/vVs8A
Some sites, including this one: http://www.boonebank.com/brc/SBR_template.cfm?Document=headl...
... which links back to the original post (is that not supposed to acknowladge the original source of the article?) and quotes Matt ranks higher in my SERP results
Shouldn't that be more important than whether it has original content or not?
Maybe an aggregator displays the content in a more useful way than the originator so it gets linked to more than the originator.
If the content creator doesn't like his content copied he can take it up with the copier. It's not Googles job to get involved in that.
Google's job is to give the searcher a list of the sites that matches his keywords in an unbiased way. They should do that mostly on what the internet thinks is the best site, not what Google thinks is the best site.
Publicly discussing algorithms changes like this seems like a potential PR problem.
I mention this because Google often talks publicly about being entirely algorithmic, and elements of this narrative feel human.
I'd recommend 'Seeks', (http://www.seeks-project.info/). It requires you run it on your machine, or use a public node. Though while on your machine, it 'learns' from your navigation and re-ranks the results based on these local data. Additionally I use regexps to remove websites I don't want to hear about, like expertexchange.com.
Try something like: http://www.google.com/search?q=hidfind.exe
Another similar search area (Windows drivers for various hardware) has improved a lot judging from the few examples I just tried - actual hardware manufacturers are now amongst the top results for a change.
So that's quite a step forward...
http://www.google.co.nz/search?sourceid=chrome&ie=UTF-8&...
Mailing list entries from http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-... show up from old.nabble.com before the original source.
From a quick look there is significant improvement.
So ideally, perhaps m.google.com would be able to sort through this and not penalize the duplicated-nature of the mobile version.... Anyway, something to think about if you haven't already.
Who'm gets penalised? The higher ranking, more frequently indexed property portals or my local website.
a question that I always pondered, what is googles approach on more clever forms of rehashed content that involve photoshopping / cropping an original image, is this something that google looks at?
Example source photo: (http://shoes.n-sb.org/img/thumbs/472153d5fc674fa1c685f3c7814...)
rehashed image: (http://images.sneakernews.com/wp-content/uploads/2011/01/nik...)
if you have a blog about wines, then what do you need to write - in order to be "original" blog about wines - does it mean mostly:
1) express original opinions about wines, 2) original structure of sentence and original wording - but the same opinions that 10 other sites write about, 3) original brands of wine you talk about - original names mentioned, 4) original content in the sense that you have not copied the text 100% from someone else - or what 5) combination of the factors mentioned above 6) something else that makes your content original ..
Internet marketers use software to "spin" text and make it appear unique. Sometimes the text is annoying for humans to read, but search engines eat it up. You can take one article and turn it into 50 with the click of a couple of buttons.
To be an "original" wine blog, write your own blog entries. Don't copy someone else's.
Hit http://duckduckgo.com with your programming queries.
No - I am not affiliated and am not getting paid to write this.