News for hackers. Stuff that matters.
Re:How Google uses blacklists, algorithm tweaks and contract... by creaghpatr on Friday November 15, 2019 @10:28AM (#21545572)
Here are the findings of the investigation, according to the article:

>More than 100 interviews and the Journal’s own testing of Google’s search results reveal:

• Google made algorithmic changes to its search results that favor big businesses over smaller ones, and in at least one case made changes on behalf of a major advertiser, eBay Inc., contrary to its public position that it never takes that type of action. The company also boosts some major websites, such as Amazon.com Inc. and Facebook Inc., according to people familiar with the matter.

• Google engineers regularly make behind-the-scenes adjustments to other information the company is increasingly layering on top of its basic search results. These features include auto-complete suggestions, boxes called “knowledge panels” and “featured snippets,” and news results, which aren’t subject to the same company policies limiting what engineers can remove or change.

• Despite publicly denying doing so, Google keeps blacklists to remove certain sites or prevent others from surfacing in certain types of results. These moves are separate from those that block sites as required by U.S. or foreign law, such as those featuring child abuse or with copyright infringement, and from changes designed to demote spam sites, which attempt to game the system to appear higher in results.

• In auto-complete, the feature that predicts search terms as the user types a query, Google’s engineers have created algorithms and blacklists to weed out more-incendiary suggestions for controversial subjects, such as abortion or immigration, in effect filtering out inflammatory results on high-profile topics.

• Google employees and executives, including co-founders Larry Page and Sergey Brin, have disagreed on how much to intervene on search results and to what extent. Employees can push for revisions in specific search results, including on topics such as vaccinations and autism.

• To evaluate its search results, Google employs thousands of low-paid contractors whose purpose the company says is to assess the quality of the algorithms’ rankings. Even so, contractors said Google gave feedback to these workers to convey what it considered to be the correct ranking of results, and they revised their assessments accordingly, according to contractors interviewed by the Journal. The contractors’ collective evaluations are then used to adjust algorithms

Re:How Google uses blacklists, algorithm tweaks and contract... by WalterGR on Friday November 15, 2019 @02:07PM (#21547712)
Despite publicly denying doing so, Google keeps blacklists to remove certain sites or prevent others from surfacing in certain types of results. These moves are separate from those that block sites as required by U.S. or foreign law, such as those featuring child abuse or with copyright infringement, and from changes designed to demote spam sites, which attempt to game the system to appear higher in results.

Google has a permanent demotion applied to a site I've run since 1996: http://onlineslangdictionary.com/ . I estimate that my traffic would be 2.5x - 3x what it is now, were the demotion not in place.

These demotions are hidden, permanent, and cannot be appealed. Moreover, these demotions can be performed by hand internally within Google - for whatever reason they choose, or for no reason at all. That is to say, some demotions are manual and not automated.

I have never been officially notified that the demotion exists, in any of Google's available tools or any other way. However, a Google employee checked the internal status of my website and there is, indeed, a permanent demotion in place.

There is no reason for my site to be demoted. This demotion was put in place while Matt Cutts was the head of the web spam team. I asked him about it here on HN, and he lied about it. (I know he lied because of my communication with the Google employee.) You can read my thread with Matt here: https://news.ycombinator.com/item?id=5408087 .

I'd like to make the email chain between the Google employee and I public. But I don't want to ruin someone's career / life, just because they did the right thing and told me about the penalty.

So... I don't know what to do. Thoughts?

Re:How Google uses blacklists, algorithm tweaks and contract... by tobltobs on Friday November 15, 2019 @02:38PM (#21548016)
A search for "slang dictionary" gives me your site as Nr.1 result. For "slang words" you are somewhere on page 4 or 5.

A Google demotion looks different. What do you expect? Ranking on the first page for every slang word?

Re:How Google uses blacklists, algorithm tweaks and contract... by WalterGR on Friday November 15, 2019 @02:50PM (#21548119)
Fair question. That's not how these demotion work.

More details follow. But whether my metaphor is apt or not doesn't eliminate the fact that a Google employee informed me about the demotion.

The following is a FAQ taken from the page on my website about the demotion. It's written for a general audience.

Q: I just did a Google search and your site appeared in the first few results. Does that mean that the penalty has been removed?

A: No.

"Google Juice" is an informal term for how favorably Google views a website and pages on that website. There are a lot of factors that go into how much juice websites and pages earn.

Every time you do a search, Google's algorithms evaluate every page on the web to decide how relevant each page is to your query - how much Google Juice each of the pages has for your query. Then they show the search results, which are ordered by the amount of Google Juice each page has.

What the penalty does is subtract an amount of Google Juice from every page on this site. But whether that means we appear 1st, 2nd, or 307th in the search results depends on how much juice each of the other pages on the web have for your query.

You could think of it as a foot race. The penalty doesn't work as in: however you finish in the race, Google will drop you down by 9 places. It works as in: Google attaches a 20 pound weight to your foot, and whether that means you finish 1st or 307th depends on how good the other runners are.

So sometimes pages from this site appear towards the top of search results. That just means that those specific pages have enough Google Juice - and other pages on the web have so little Google Juice - that even with the penalty, we can appear towards the top. But overall, the penalty drags our pages down far enough that we would get about three times as many visitors if the penalty weren't in place.

Re:How Google uses blacklists, algorithm tweaks and contract... by rbinv on Friday November 15, 2019 @03:35PM (#21548485)
It's important to differentiate between "link juice" and overall ranking power which I think you mean by "Google juice" (as links are not all there is to ranking well).

That said, I agree that penalties/demotions seem to have a somewhat dampening effect, e.g. resulting in the previously unmodified rank but times 0.2 or whatever.

However, penalties should not last 13 years if the causes have been fixed. What did said Google employee tell you about your case?

edit: also, there is a way to submit a reconsideration request. Have you tried that?

Re:How Google uses blacklists, algorithm tweaks and contract... by WalterGR on Friday November 15, 2019 @03:39PM (#21548533)
It's important to differentiate between "link juice" and overall ranking power which I think you mean by "Google juice" (as links are not all there is to ranking well).

True. I wanted the text to be immediately understandable by non-technical people. I'm on like revision 3,194 of the text. :) Any suggestions for improvement are welcome.

Also, there is a way to submit a reinclusion request. Have you tried that?

Yes, several times.

However, penalties should not last 13 years if the causes have been fixed.

They shouldn't. But in this case, it has lasted that long. It's a manual penalty turned on by a Google employee. The only way to get rid of the penalty is for them to (manually) turn it off.

What did said Google employee tell you about your case?

That there's nothing I can do.

Re:How Google uses blacklists, algorithm tweaks and contract... by google_webspam on Saturday November 16, 2019 @12:06AM (#21551119)
Manual penalties can be appealed via the reconsiderations process. If a webmaster is unable to file a reconsideration request, this usually means that there is no manual penalty on the site.
Re:How Google uses blacklists, algorithm tweaks and contract... by WalterGR on Saturday November 16, 2019 @12:21AM (#21551164)
As I mentioned above, manual penalties can be hidden. I know this because there was no indication from any tool that there was a penalty against my site, and because Matt Cutts claimed that there was no manual penalty against my site, and yet a Google employee verified the penalty internally.

If you're a Google employee, I'd love to discuss this with you! My email address is [email protected] .

Re:How Google uses blacklists, algorithm tweaks and contract... by google_webspam on Saturday November 16, 2019 @01:31AM (#21551334)
Matt's feedback was correct. There's nothing you need to do for your site.
Re:How Google uses blacklists, algorithm tweaks and contract... by WalterGR on Saturday November 16, 2019 @03:14AM (#21551598)
There's nothing you need to do for your site.

I'm a little unclear on that. Matt specifically mentioned two active penalties.

Matt said that there was an automated penalty due to advertising on my site. Following this, I removed all advertising from the site. My site's ranking did not change, nor did traffic referred by Google searches. Why is that? If there's nothing I need to do for the site, why have those metrics not changed?

Matt also said that there was an automated Panda penalty against my site. Following this, I removed all citations from the site. My site's ranking did not change, nor did traffic referred by Google searches. Same questions as above: Why is that? If there's nothing I need to do for the site, why have those metrics not changed?

What did the prior manual penalty against my site that Matt mentioned have to do with Web Build Pages / Jim Boykin? I had never even heard of them. What specifically was that manual penalty? I have seen no evidence in ranking and in traffic referred by Google searches to ever indicate that this penalty existed. Furthermore, I was never informed by Google via any mechanism that there was a manual penalty against my site. Why is that?

Have Google employees ever been able to apply demotions, penalties, or any mechanism whatsoever to drop a website's positions in Google SERPs, in a way that the website owner is never made aware of it?

Has that ever been done to my site, The Online Slang Dictionary?

Thanks very much for your input.

Re:How Google uses blacklists, algorithm tweaks and contract... by WalterGR on Monday November 18, 2019 @08:59PM (#21570121)
Clarification:

> Have Google employees ever been able to apply demotions, penalties, or any mechanism whatsoever to drop a website's positions in Google SERPs, in a way that the website owner is never made aware of it?

should have read, in part,

> Have Google employees ever been able to manually apply...

Re:How Google uses blacklists, algorithm tweaks and contract... by zepto on Friday November 15, 2019 @03:03PM (#21548214)
That thread with Matt Cutts looks terrible for Google.

In it he admits that the site is being penalized for having prominent ads above the fold.

90% or more of Google’s revenue comes from presenting prominent ads above the fold.

How is demoting the organic results of sites that use the same business model that it does not the epitome of anti-competitive behavior?

Re:How Google uses blacklists, algorithm tweaks and contract... by WalterGR on Friday November 15, 2019 @03:37PM (#21548498)
In it he admits that the site is being penalized for having prominent ads above the fold.

Yep. And so I removed all advertising from the site for months.

There was absolutely no change in ranking.

Matt Cutts also wrote, "Your site is also affected by our Panda algorithm..."

My website had citations of slang use. By definition citations are 'duplicate content' since they also exist somewhere else.

So I removed all citations from the site for months.

There was absolutely no change in ranking.

The 3 claims Matt Cutts made were:

1. "...the only manual webspam action I see regarding onlineslangdictionary.com is from several years ago (are you familiar with a company called Web Build Pages or someone named Jim Boykin?)..." Wrong. I've never heard of him or his company, I've never worked with anyone involved with SEO, and there was never a change in ranking / site traffic that would suggest that my site had a penalty. Given the very very dim view of SEO practitioners among technically savvy people, I can only assume that his implication ("are you familiar with") was designed to discredit me here.

2. "You're affected by a couple algorithms in our general web ranking. The first is our page layout algorithm... your site has much more prominent ads above the fold compared to Urban Dictionary." Wrong. (As above, I removed all advertising from the site.)

3. "Your site is also affected by our Panda algorithm." Wrong. (As above, I removed all citations from the site.)

I can't say that anything Matt Cutts told me was truthful.

Re:How Google uses blacklists, algorithm tweaks and contract... by im3w1l on Saturday November 16, 2019 @10:19AM (#21553040)
Putting ads above the fold makes a site worse for users. This doesn't mean it's immoral, and it doesn't mean it's the wrong thing to do. You are trading of the amount of value you provide and ability to capture a portion of that value. Capturing value can be essential to continued existence of the site, and ability to create more content.

However, Google search is acting on behalf of the users, trying to find them the result that brings them the most value. And everything else equal, that is the one without ads above the fold.

Re:How Google uses blacklists, algorithm tweaks and contract... by zepto on Saturday November 16, 2019 @12:49PM (#21553895)
The argument that Google is acting on behalf of users is contradicted by the fact that they put their own ads above the fold.
Re:How Google uses blacklists, algorithm tweaks and contract... by dave_b on Friday November 15, 2019 @06:52PM (#21549888)
Business model has nothing to with anti-competitive behavior, it’s about being in the same industry / providing the same service.
Re:How Google uses blacklists, algorithm tweaks and contract... by zepto on Friday November 15, 2019 @06:55PM (#21549897)
Suppressing sites that display advertising seems like it is anticompetitive behavior against other ad networks, no?
Re:How Google uses blacklists, algorithm tweaks and contract... by strbean on Friday November 15, 2019 @07:42PM (#21550102)
Only if they targeted sites that didn't use AdSense and let sites that did use AdSense place ads wherever they wanted.
Re:How Google uses blacklists, algorithm tweaks and contract... by zepto on Friday November 15, 2019 @10:00PM (#21550700)
Since their own ads on the search results page are not targeted that is a clear yes then.
Re:How Google uses blacklists, algorithm tweaks and contract... by seminatl on Friday November 15, 2019 @02:17PM (#21547824)
Wait, what is the basis of ranking that you expect for your site? Is it based on Planck's constant or ???
Re:How Google uses blacklists, algorithm tweaks and contract... by WalterGR on Friday November 15, 2019 @02:29PM (#21547944)
Via comparison to similar websites. It's an estimation.
Re:How Google uses blacklists, algorithm tweaks and contract... by seminatl on Friday November 15, 2019 @02:46PM (#21548088)
Is it possible that the sites that outrank yours are preferred by search users?
Re:How Google uses blacklists, algorithm tweaks and contract... by WalterGR on Friday November 15, 2019 @02:54PM (#21548152)
Sure. If the demotion were removed and my competitors got more traffic than my site, hey, that's great.

I don't expect special treatment. All I want is for the demotion to be removed so that I can compete with the rest of the web on a level playing field.

Re:How Google uses blacklists, algorithm tweaks and contract... by frenchyatwork on Friday November 15, 2019 @02:23PM (#21547875)
I don't think it would be right to out the whistleblower unless you're going to sue Google for unfair treatment, and you need their testimony.

This sort of thing is why we have courts.

Re:How Google uses blacklists, algorithm tweaks and contract... by WalterGR on Friday November 15, 2019 @02:32PM (#21547973)
This sort of thing is why we have courts.

I wish I had the money to pursue that.

Re:How Google uses blacklists, algorithm tweaks and contract... by GordonS on Friday November 15, 2019 @06:11PM (#21549681)
> Despite publicly denying doing so, Google keeps blacklists to remove certain sites or prevent others from surfacing in certain types of results

Google absolutely does this. Around 6-9 months ago bluelight.org (a somewhat controversial drug harm reduction forum) disappeared from search results overnight.

It will show results if you explicitly use `site:bluelight.org`, otherwise nothing

* I feel the need to add a disclaimer here: I don't abuse drugs, but find blue light an interesting source of information regarding side effects and bioavailability of various prescription medications

Re:How Google uses blacklists, algorithm tweaks and contract... by buboard on Friday November 15, 2019 @10:54AM (#21545808)
I wonder how its founders feel about this culture? Google used to have a specific role, creating a win-win situation for everyone on the public web, but now it has become a greedy monster having lost its mind. I wonder what caused it ? was it reckless hiring or the fact that the founders no longer seem to care or anywhere to be found? It's a shame really, for such a monumental company.
Re:How Google uses blacklists, algorithm tweaks and contract... by creaghpatr on Friday November 15, 2019 @11:08AM (#21545958)
The article (which is really long) talks pretty extensively about founder conflict about manual intervention, this is the passage I feel captured the sentiment:

>Mr. Brin still opposed making large-scale efforts to fight spam, because it involved more human intervention. Mr. Brin, whose parents were Jewish émigrés from the former Soviet Union, even personally decided to allow anti-Semitic sites that were in the results for the query “Jew,” according to people familiar with the decision. Google posted a disclaimer with results for that query saying, “Our search results are generated completely objectively and are independent of the beliefs and preferences of those who work at Google.”

>Finally, in 2004, in the bathroom one day at Google’s headquarters in Mountain View, Calif., Mr. Page approached Ben Gomes, one of Google’s early search executives, to express support for his efforts fighting spam. “Just do what you need to do,” said Mr. Page, according to a person familiar with the conversation. “Sergey is going to ruin this f—ing company.”

Re:How Google uses blacklists, algorithm tweaks and contract... by paganel on Friday November 15, 2019 @11:22AM (#21546109)
> I wonder what caused it ?

I guess money caused it. When you have a market-cap of $900+ billion you’re not going to keep up with promises made back when you were “only” a $90 billion company or a $9 billion company. I remember the pre/IPO days when each index update was followed almost religiously on webmasterworld.com and where an user called Googleguy (supposedly Matt Cutts) was implicitly promising all the website owners in there that Google was our friend, that they were never going to become a portal and that they would never steal anyone’s content.

Re:How Google uses blacklists, algorithm tweaks and contract... by jammygit on Friday November 15, 2019 @12:03PM (#21546493)
You’re right on an observational level, but I don’t think we should normalize breaking promises because of money
Re:How Google uses blacklists, algorithm tweaks and contract... by paganel on Friday November 15, 2019 @12:26PM (#21546728)
Being right on an observational level is generally considered to be, well, right when it comes to perceived reality, what should or should not happen from a moral/ethical point of view is another discussion (the OP asked why did this happen, not if what happened was right or wrong).
Re:How Google uses blacklists, algorithm tweaks and contract... by Spooky23 on Friday November 15, 2019 @11:29AM (#21546183)
What is the public web today? The old web is dead -- most normal humans live in the modern versions of AOL. Twitter, Facebook, Instagram, etc.

Microsoft figured out that this was going to be a problem first -- you may recall that Bing was marketed as providing answers instead of results, which was a nascent threat. Google moved towards providing answers, and found that embedding ads / product placement in those answers was profitable.

Also remember that the users of Google in 2002 or whenever were a different audience. The "win-win" for you is confusion to the average punter.

I don't want to come off as a defender of Google -- but search is an incredibly complex business that means something different to many cohorts of users. I think the surveillance stuff is approaching or crossing a line nowadays, but I think it's fair to say they have done a pretty good job considering how the online and offline world has evolved.

Re:How Google uses blacklists, algorithm tweaks and contract... by buboard on Friday November 15, 2019 @12:10PM (#21546555)
> The old web is dead

I don't think so, and that s thanks to people who keep writing blogs, tutorials, make thoughtful videos, posting thoughtful comments etc. The "Old web" had a high technical barrier to entry and that acted as a selection filter. The "new web" is the former TV audience, which moved to facebook/instagram and mobile apps. It's big audience, and marketers are after it, but that didnt make the "old web" disappear. I ve never read anything interesting inside facebook except from Yann Lecun's posts (which he keeps there out of courtesy to his employer).

I just hope people stop thinking the old web should also move inside those because that's where their crowd is. It's not true, that's a different crowd.