This still seems to be missing the point.

As I understand it, sites that are not doing anything in the realm of SEO can get penalized because of the actions of people outside of their control.

If the wrong sorts of sites (however defined), sites outside my control, are linking to my site, why am I penalized, and why should the burden be placed on me to go fix it?

Check the people who try to drop profile links & comment spam links on your blog and such. Are there any websites which have as many spammy inbound links as YouTube does? How many people does Google allocate to cleaning up YouTube's spammy link profile?

There's another factor in that Google may shift what is considered reasonable over time. A buddy of mine got an unnatural link warning where the link cited was a link that has been in place LONGER THAN GOOGLE HAS EXISTED.

Now consider that some websites are bought & sold, change ownership, etc. ... does it make sense to penalize today for something which happened 10 or 15 years ago?

Another factor is that just by ranking you will pick up some scraper site inbound links you do not want. Many of those sites in various forms or fashion are monetized via Google ads. Google continues to run their ads on many of those aggregation sites, yet you may get an unnatural link penalty for having links from the same sites.

Good enough for Google, NOT good enough for you TM

Very well put.

Oh he gets it, it's his job to act like he doesn't and to change the subject "...Ummm...Bing does it too..look I made a video"

Well, if a website was link spamming then it seems appropriate that the site should attempt to clean up the spam they made before they can rank well in Google again. Otherwise it's not fair to the other websites that have been trying to rank fairly and would like a level playing ground.

In general, if you weren't trying to create spam links, then it's very unlikely that any of this is an issue for you and the burden isn't on you to fix it. I made a video about the "what if the wrong sorts of sites link to my site" question here: http://www.youtube.com/watch?v=HWJUU-g5U_I

So if Bing wants to rank Bing.com higher than Google.com on the keyword "Search Engine", all they need to do is to pay link farms to link to Google?

Like everything this could go both ways. If you penalize sites for having links from link farms, what is stopping your competitors from paying link farms to link to your website? or worse yet, some link farm owner comes up with an ingenious idea to hold other websites hostage until they pay up?

It would be more understanding if Google din't have any other resources on hand to make smart decisions but heck you guys have hands on realtime experience with most traffics via Google Analytics, Google Adsense and even DNS. All these resources and can't even distinguish a link farm from a legitimate site? Doesn't sound believable.

The Google webspam team seems to prefer psychology over technology to solve the problem, especially recently. Nearly everything that's come out of Matt Cutt's mouth in the last 18 months or so has been a scare tactic.

IMO all this does is further encourage the development of "churn and burn" websites from blackhats who have being penalized in their business plan. So why should I risk all the time and effort it takes to generate quality web content when it could all come crashing down because an imperfect and overzealous algorithm thinks it's spam? Or worse, some intern or non-google employee doing a manual review wrongly decides the site violates webmaster guidelines?

yup.

>Nearly everything that's come out of Matt Cutt's mouth in the last 18 months or so has been a scare tactic.<

the bizarre thing is the gap in perceptions internal to Google versus external. https://twitter.com/mattcutts/status/449008706655510528 https://polldaddy.com/poll/7916680/

>IMO all this does is further encourage the development of "churn and burn" websites from blackhats who have being penalized in their business plan.<

Absolutely. churn & burn sites, and then some mixed in parasitic hosting on sites which are already highly trusted.

During this "crackdown campaign" Google has been on, I have sat in on meetings at large clients where they've pulled sites that are supposedly link farms (according to google) and put together an ad buy on behalf of competitors meant to cause these penalties to be levied and WMT messages to be sent.

They needed to provide a useful service to begin with to get the traction (good search algorithms for the web-that-is), but now that they are the incumbent, it is more in their interests to penalise website owners and force them to buy ads.

It's sad, but it's normal monopolist behaviour.

The cost of throwing scummy links at a website is under $100.

And if their first $5, $10, or $20 test doesn't work? Then so what for them. They put it on cron job and throw another $20 at it again and again. http://www.seobook.com/images/sape-sape-sape.gif

The key point here is that there's a 100X or 1000X gap between 4 figures & 6 to 7 figures. The cost of burning things to a crisp is much lower than building them up. Whenever incentive structures have a 1000-fold difference in outcomes the path of least resistance becomes a popular path.

After a few cases like this, a long list of multiple disavowed domains/networks/fingerprints, where are these negative links going to come from? This anti-SEO tactic you outline can only be used a handful of times before it becomes useless.

Matt Cutts' job at Google involves him making tons of videos explaining how the Google algorithm works. It isn't omniscient, and the guidelines show how to work within its limitations. If you don't care about optimizing your Google presence (as we don't) then you shouldn't care about Google's guidelines either. But if you do comply with some guidelines, then this will probably help your rank in other search engines which also take similar measures to TRY and detect linkspam.

Google's customers are its users. It tries hard to have the most relevant results (or provide value some other way, or just maintain a brand image so people keep coming back). The advertisers come because the users find them and spend real money. And the users want relevant search results, not spam. That's why Google spends a ton of resources on combating the spam.

How do you propose they magically "know" when a link is natural?

No, Google's customers are its advertisers. They're the ones who pay money to Google.

I agree with your general point though, the algorithm isn't perfect and sometimes it goofs. What I think people are having a negative reaction to is Matt Cutts' seemingly obtuse responses, as though he doesn't even understand the issue at hand (Google's actions distorting natural behavior), so he just keeps explaining the anti-spam policies in general terms. It's not a conversation, it's people talking past one another.

Think of some of the stereotypes of engineers. Many can code but struggle to communicate. Matt can code, has his name on many patents, leads a team, makes hundreds or thousands of videos, regularly keynotes at conferences & interacts with thousands of people, was GoogleGuy for years, participates in the comments here, was an expert in some of the past lawsuits against Google ... he has basically had a near infinite number of opportunities to put his foot in his mouth & yet how many times has he ever done it? Almost never, if ever.

About the only times I think he has potentially missed at all was the "breaking their spirits" bit on TWIG, and then two minor bits in a few YouTube videos

http://www.youtube.com/watch?v=muZuX9OaMLo&feature=youtu.be&... "We got less spam and so it looks like people don't like the new algorithms as much."

https://www.youtube.com/watch?v=2jap11IIxfw "the gap between what you can get away with and what google says you can get away with is getting smaller all the time"

---

Who do you know who has as much media exposure as he has had who hasn't repeatedly put their foot in their mouth? For me, that answer is nobody.

---

What makes Matt's performance even more impressive is how scummy some of Google's policies have been, even as he came off smelling like roses. There are endless debates on how reasonable it is for Google to infer intent on links & so on, yet at the same time some of the past Google executive emails have quotes in them like:

http://www.washingtonpost.com/politics/google-faces-new-pres... "As with all of our policies, we do not verify what these sites actually do, only what they claim to do." (for AdWords advertisers)

http://www.theverge.com/2013/1/23/3906310/the-no-hire-paper-... "I would prefer that Omid do it verbally since I don't want to create a paper trail over which we can be sued later? Not sure about this. thanks Eric"

No, Google's customers are primarily its advertisers. Optimizing advertising revenue has some overlap with pleasing users, as you note, but if a decision needed to be made between pleasing users and increasing revenue, there's no doubt about which way it would go.

I suppose any company that works out how to increase revenue while upsetting and alienating it's users could possibly implement said ideas. But if you're smart enough to do that, I would guess you would also be pretty successful at the traditional approach of increasing revenue by pleasing your users?

Rather than these fear mongering tactics, why can't Google just ignore spammy links instead of penalizing the whole websites and all the websites that link to them?

But! You may have hit the nail on the head. if i had to guess i would say that storing exceptions means that suddenly the algorithm had to check part of the corpus for these exceptions. at the scale at which google operates this could probably lead to rather big size and speed implications.

But maybe i'm just fantasizing.

Ask Google engineers how they feel about centralization of power in telecom or finance or politics & they will tell you it is the worst thing ever http://www.youtube.com/watch?v=Ik1AK56FtVc

Yet that same sort of centralization is fine when it is Google.

Those same engineers will admit that any algorithm has some level of false positives & false negatives. They may try to minimize these, but they can't make them zero.

The other thing which gets very little coverage, but is crucially important is that in spite of already having a monopoly market position Google keeps buying search marketshare with: their secret Android partner contracts, Firefox default search placement, iOS default search placement, Flash embedded into Chrome (so security updates happen in the background without sending you to a download site where trash can be bundled in with the update) & Flash security updates that hit all other web browsers coming bundled with Chrome bundleware which sets Chrome as your default browser, etc.

If Google decides they don't like you (for any reason), then you need searchers to use a different search engine in a browser which isn't paid off by Google for default placement. And you need to hope that users are savvy enough to repeatedly say "no thank you" to the automatically bundled Chrome install with their frequent Flash updates.

Google mentions how anything is only a click away or similar, but most people tend to use defaults.

When someone changes those defaults on Chrome users, that's a horrible user experience & the "hijacked" settings must be reset. http://chrome.blogspot.com/2014/01/clean-up-your-hijacked-se... But when those defaults are paid for by Google or changed through Google bundleware it is "a great user experience."

If Google believes their marketshare is a reflection of their superior search offering, they are welcome to stop buying default search placement in other web browsers & stop bundling Chrome installs on Flash security updates. But currently they are spending north of a billion Dollars a year on these activities, which indicates they clearly feel there is significant value to them.

You are suggesting an approach where there are no downsides to link spam. Obviously, players of this game would then ramp up their link spam bots, and run them non-stop. Because it causes no damage to the site they want to rank.

Link spam isn't just a search engine indexing problem, it is also a detriment to the general Web user too. It makes the Web worse, not just search.

Don't like YouTube's copyright claim issues on video game playthroughs? Use Twitch. Oh wait, what's that? Google is trying to buy Twitch. http://kotaku.com/report-youtube-is-buying-twitch-for-1-bill... Oooops.

So then what's the difference between disavows and having the links removed? Getting the links removed costs far more time & money. And it gives Google a data mining stream of feedback they can leverage to dish out further penalties.

Google requires some of the links to be removed not because it improves the web, but rather their goals & interests are aligned with punishment. They want to add cost & uncertainty to SEO in order to discourage investment in SEO from entities not formally connected to Google.

That's a good thing right? Pushing sources of bad links down through the basement, effectively nuking the source/network from the link graph.

You know Google's ranking system is based on information, and adding more information leads to better results for the Web user.

That really depends on your perspective.

If it were impossible for others to throw trashy links at your site in bulk for cheap it could perhaps be a good thing.

If Google didn't change policies overnight & then retroactively penalize you for things which were fine in the past http://searchengineland.com/guest-post-google-penalty-187707 then it could perhaps be a good thing.

Unfortunately, in reality, both of those ifs are hypothetical & untrue. Which means it is absolutely not a good thing.

>You know Google's ranking system is based on information, and adding more information leads to better results for the Web user.<

Quality of information matters as much as the volume of information. Look at Demand Media's current stock price for an example of this.

But in terms of links, you have to think through the impacts here...

Yes if Google freezes some activities they dislike then perhaps that is a net positive for relevancy, however the more they fear-monger about links the less natural linking goes on. Most of the major social sites put nofollow on almost all external links. And with the sea of fear approach to relevancy, in some cases magazines or newspapers will profile a person & not link to the source in spite of the entire article being about that source. I've spent many hours being cited by journalists where in spite of being good enough to be the primary source for their content, there was no link citation.

paraphrasing David Naylor's excellent recent video http://youtu.be/UdflGrCJxYc?t=1m50s "More and more high profile websites are having a no linking policy. Which really is kind of weird isn't it, because if that is a journalistic website and they are writing about your website surely those are the kinds of links Google would want to see. ... It seems kind of weird that the links Google actually wants you to get are the first links to dry up on the web." - David Naylor

The other important factor is the quality of the information created by those who are trying to do disavows and link removals. When people are irrationally driven by fear while in a harmed state, they are NOT acting rationally. Business owners who are selling off assets, firing employees, stalling with creditors, aligning predatory lending to try to keep things afloat, etc. are stressed out & are likely to make many poor decisions in that rushed & panicked state.

Thus some of that more information which is created is junk misinformation.

Consider this email: http://www.seobook.com/images/link-unremoval.png They asked the website to remove the link claiming it was unnatural, and then they emailed again to ask the website to ignore the removal request.

They don't know which links are "unnatural" and so they are using automation to try to sort through it all. Are their automated quick guesses (which are often driven by tools) more useful than all of Google's internal reviews & ratings data which has been built up for over a decade? Color me skeptical.

There's another factor with the disavow data as well. Look at this removal request http://www.seobook.com/images/trampolines.png Our site has about 20,000 unique linking domains referencing it. However over the years we have had well over a million registered user profiles. If only 2% of the registered user profiles were ignorant spammers who spammed our profile pages and then later added our site to a disavow file, we would have more people voting against our site than we have voting for it. And those profile pages were already not indexed & the links were nofollowed anyhow. Those pages are effectively outside the search game, yet those ignorant spammers can still create negative votes against our site based on the fear-mongering.

And as bad as that sounds, there's no reward for actually removing the bad links either. At some SEO conferences numerous SEO experts have gave the advice to "just disavow them anyhow" even if the links are removed. Thus this deluge of email spam offers publishers no value whatsoever, just sunk cost & wasted time - time which could have been spent creating useful information.

In such scenario, how does Google determines if I created the links? Does Google have precognitive abilities or it can spy on my communication channels (Gmail, Google Voice, Hangouts etc.) to figure out if I am innocent?

A model is only as good as the information in it. In this case providing details of these spammy links helps you, and every other site targeted through the same sites/networks/fingerprints.

Consider it a reverse no-follow mechanism.

It appears after years and millions spent it is impossible to get the perfect search algorithm.

That is an absolutely ridiculous rule of thumb to follow. What exactly is wrong about doing something like:

1) Having a good article on your site about a certain topic 2) Reaching out to other site owners who have similar content and asking them to link to you as an additional resource 3) Getting that link.

Can you honestly, with a straight face, tell me that's BAD!?

The situation you mentioned can start out as innocuous but quickly become downright shady. Hmm, you're asking them to link to you, but what do they get in return? Oh, here's a thought, you tell them you'll link to them as well if they link to you, that way you'll both get a ranking bump! Totally fair and win-win for everyone except for the people who are trying to get quality information.

That might be the ideal result, but it's hard to see how anyone could achieve it with a purely automated system. To make the kinds of distinction you're describing, in general you probably need some form of manual curation by genuine experts with uncontroversial opinions, assuming such people even exist in the field of interest.

How do you feel about Google sponsoring educational seminars for market regulators where they do not disclose their sponsorship, actively conceal it, and request the entity putting on the sponsorship do the same thing?

http://www.washingtonpost.com/politics/how-google-is-transfo...

It is also worth mentioning that when Google was lobbying against regulation they indeed DID control & manipulate the placement of favorable media masquerading as regular content:

"the staff and professors at GMU’s law center were in regular contact with Google executives, who supplied them with the company’s arguments against antitrust action and helped them get favorable op-ed pieces published, according to the documents obtained by The Post."

---

Another thing with the line of thinking based on pureness...should companies which have repeatedly been caught rigging actual physical markets (e.g. municipal bonds, California energy, LIBOR, interest rate swaps, forex, etc.) in the real world get a penalty by Google for it?

Or is it reasonable that they take their outsized gains from their market rigging behavior & invest some small portion of it into buying out smaller competitors, lobbying & writing regulations to harm smaller competitors, buy some feel good brand ads & do various charity donations to paint a picture of themselves?

---

I also want to quote the following past statement from a Google executive about the AdWords ads Google puts directly in the search results:

"As with all of our policies, we do not verify what these sites actually do, only what they claim to do."

http://www.washingtonpost.com/politics/google-faces-new-pres...

---

and, some icing on the cake, ... ;)

check out the keyword rich deep links to some of Google's affiliate offers on BeatThatQuote.com (for credit cards, car insurance, etc.)

Google bought BTQ & then put a bunch of cross-site keyword rich deep links on it. is that natural? how is that any different than buying links?

"While important as a vote of confidence for the content they point to, there is simply so much link spam these days that it’s tough to know where to turn. Obviously buying links is a dead end, and it doesn’t matter how you split this hair: sharing, encouraging, incentivizing, buying – it’s all the same. You want links to surprise you. You should never know in advance a link is coming, or where it’s coming from. If you do, that’s the wrong path. Links are part of the bigger picture. You want them, but you want them to be natural. If an engine sees you growing tem naturally, you’re rewarded with rankings. If they see you growing them unnaturally, you’re rewarded with penalties."

If you disagree with that, you'd need to take it up with Bing's webmaster outreach team.

Your statement implies both Google and Bing share the same sentiment.

So now, my question still stands: What about that scenario that I laid out is actually bad/spammy/evil? Why should anyone ever be punished for doing that?

This practice has existing long before Internet, and internet has made it easier than ever. Google is essentially telling us not to ask our customers to evangelize for us.

Why such hypocrisy? How about the invites system Google uses for many products including Gmail where users are encouraged and incentivized to invite friends.

According to Google's terms if I link to my Dropbox referral link on my blog, Dropbox should be penalized for it because they incentivize users to do that.

The internet should not work according to Google's algorithm and policies, it should be the other way around.

Every marketing seminar I have ever been to has emphasised that as much as you can turn your clients into fans, in the end you still need to ask them for a testimonial and that you need to tell them that you love referrals.

Under that same logic, Google Adwords should never know in advance where their ad clients are coming from, and if you have someone out there trying to on-board big clients (or increase their spend) that's the wrong path.

You say "if they get caught" when sometimes, there is nothing to catch. You're dinging people who have grinded to forge relationships, in the name of trying to keep spammers out of the index.

It makes it too easy for me to hire a spam company to blast an overwhelming amount of crappy links at my competitors who may or may not be well equipped to disavow those links.

Which, if my company has the budget to hire someone to monitor & disavow, and my competition does not, well... somebody's still gaming the system...just not in the same way...

That's what's wrong with this picture. The link was perfectly valid and yet Google recommended that site's admin to disavow the natural link, simply because Google's algorithm isn't good enough to see what's natural and what isn't. So his suggestion is: if it's not good enough, then don't recommend the wrong stuff to people.

This is pretty typical of 2014 Google.

To be honest these days I pretty much regard Google as one of the banes of my existence, you guys hold so much power and influence so much of the internet the rest of us have to work around you, I'm currently pulling all our customers off Google Apps for Enterprise for a multitude of reasons (buggy code, broken imap implementations, impossible to get an actual human answer) as well as moving myself away from all Google services (including Android).

I remember when you guys where the good guys.

People have not only been complaining about this since Penguin came out (around two years now), before that they were complaining about the opposite: that Google's arbitrary rules were creating spammy links all over the internet. It's completely ignoring the past to pretend this is some new thing to fit a preconceived narrative.

Most of the SEO folks that seem to come out of the woodwork for these threads would agree with me that "This is pretty typical of 2014 Google" ignores years of them complaining about the exact same thing, except they're just happy to have more people on their side.

Then we stared seeing the first indications that real "negative" SEO was possible in certain circumstances and we complained even further. Stop counting links you don't like. Now we are at a point where we see old websites being told to remove or disavow links that may be as many as 10 years old.

Can Matt or Google really stand there and say that it took them 10 YEARS to figure out that not only was that a link that they didn't like, but it was so evil that now it needed to be removed from the internet? The solution has been, and always will be, stop letting things you don't like count as a plus in your algorithm. When in doubt, don't count it.

Instead, we have Google adding thousands of webmasters and site owners as a living part of the algorithm and getting them to remove all those links, plus many they can't see or understand the value of, by the use of fear and scare tactics. Remove these links are we will ruin your website rankings. It doesn't matter if you created or asked for them or not, remove them.

Yes, it most certainly did!

From the article:

> Apparently Google convinced them, via their [read: Google's] Webmaster Tools portal, that the link looked “unnatural”, and that they should use the [Google] Link Disavow Tool to discredit the link.

(emphasis added)

> It's quite likely they noticed a penalty...

Yes, and where would they have noticed that penalty?

In a drop in incoming traffic, so with whatever they're using for analytics.

In all honesty it's probably just a new SEO consultant trying to clean up past sins and move on with whatever the latest grey hat strategy is.

IMO it's a gross overreaction to a form letter that a high percentage of webmasters have received.

The whole point is, Google spreads FUD to paper over its inadequate algorithm. People believe and look up to you as a source of truth. However, you take that trust, and you ask people to do your dirty work. That means more work for them, and not every small business has all sorts extra time money to spend on stuff like that. This is not to mention that there are probably a bunch of other more important things they could be doing. (and I know you know this, your friendly face is the head of FUD over at Google)

With all of your power and authority, you have a responsibility be more honest and straightforward. Not everyone knows better, and when people end up listening to your Google/self serving advice, they are ruining the open and free nature of the internet.

This wouldn't make me so salty, but the complete BS and favoritism exhibited by you and Google is obnoxious. So, sometimes, it might be better if you just shut up.

sorry if there are any typos etc, my (virtual) keyboard is spazzing out.