Google search only has 60% of my content from 2006

Re:Google search only has 60% of my content from 2006 by ma2rten on Saturday April 27, 2019 @10:42AM (#19765771)

I work on AdSense at Google. We need to crawl all the pages that serve our ads so that we can show ads that are relevant. We actually store those crawl results in a different place than search to prevent exactly that problem. We could probably save a lot of money if we consolidated those indices, but we don't do that prevent biasing search results. As a result a large percentage of pages that show Google Ads are not in the search index.

If Google was so evil, why would we purposefully send traffic to sites where we only get a percentage cut of the revenue?

EDIT: I realize I needed to explain this statement a little bit more. If we show ads on google.com we get 100% of the revenue. If show ads on reversephonelookup.it they get majority of the revenue. There is a limited amount of advertiser demand. Instead of manipulating the organic search results, it would be more profitable for google to just show more ads on the search page or inflate the ad price or something.

[ Parent | on HN ]

Re:Google search only has 60% of my content from 2006 by rhizome on Saturday April 27, 2019 @10:59AM (#19765840)

If Google was so evil, why would we purposefully send traffic to sites where we only get a percentage cut of the revenue?

I'm pretty sure you didn't mean to open this can of worms, did it have something else printed on its label?

Phone number lookup sites are almost certainly a) easily detectable; and b) low traffic. If Google only gets a percentage cut of this, why index them for fractions of pennies per year?

[ Parent | on HN ]

Re:Google search only has 60% of my content from 2006 by citrablue on Saturday April 27, 2019 @11:19AM (#19765947)

It's because Google's constraint is number of engineer-attention-hours. This kind of thing probably just isn't a priority, when they have a billion other things they could work on.

Keep in mind that Google is not a product company, they are a data company. They work on the biggest most impactful items, and don't have a lot of time for one-offs.

Finally, it might not be clear at all that these sites should be removed from the index.

[ Parent | on HN ]

Re:Google search only has 60% of my content from 2006 by idlewords on Saturday April 27, 2019 @12:10PM (#19766211)

Google has vast resources and could easily get this stuff right, but chooses not to. Their protests about not having the capacity can get pretty comical. Note also that the claim about "working on the things that have the most impact" is tautological.

[ Parent | on HN ]

Re:Google search only has 60% of my content from 2006 by ma2rten on Saturday April 27, 2019 @11:13AM (#19765917)

There may be cases where we drop low traffic sites from our index, which is separate from the search index. I don't how much of that is public information, so can't go into detail.

[ Parent | on HN ]

Re:Google search only has 60% of my content from 2006 by rhizome on Saturday April 27, 2019 @12:50PM (#19766411)

Well but my point is why can't Google recognize these phone-number-lookup sites as the chaff that they are? Nothing should rank lower than them in any search that would return their pages. Said another way: it should be harder for them to clog the top of the SERPs (of course "them" could imply any number of topics).

[ Parent | on HN ]

Re:Google search only has 60% of my content from 2006 by JetSpiegel on Saturday April 27, 2019 @07:44PM (#19768374)

> As a result a large percentage of pages that show Google Ads are not in the search index.

Those results are organic, at least they were not marked as ads. That's even worse, it means the regular crawler is preferring generated phone number sites over blogs. That's the real money waster right there.

> We need to crawl all the pages that serve our ads so that we can show ads that are relevant.

Why? I though Google's business model was tracking users to show relevant ads to them. You say it's more like DuckDuckGo, or the old magazine ads: film magazines get blockbuster trailers, gardening magazines get compost ads.

Either the ads are personalized, so the surrounding content doesn't matter, or the ads are "static", so why track everyone all the time, then?

> If we show ads on google.com we get 100% of the revenue. If show ads on reversephonelookup.it they get majority of the revenue. There is a limited amount of advertiser demand. Instead of manipulating the organic search results, it would be more profitable for google to just show more ads on the search page or inflate the ad price or something.

As your sibling comments imply, this works like a protection racket. The profit is closing the market to webmasters that don't allow ads at all, or that use non-Google networks.

[ Parent | on HN ]

Re:Google search only has 60% of my content from 2006 by rvnx on Saturday April 27, 2019 @10:50AM (#19765803)

Because if you show artificially irrelevant results too often (by biasing search results to prioritise AdSense pages), then users would stop using Google Search at some point. Eventually you would loose the revenues coming from the SERP, which are 10 times higher than AdSense :)

Would you risk loosing 10, to get 1 ?

[ Parent | on HN ]

Re:Google search only has 60% of my content from 2006 by ma2rten on Saturday April 27, 2019 @11:20AM (#19765960)

Let's say the search index is size N and the AdSense index is size M. If we were to join them, we would save the storage space for pages that are in both indices.

Also, the search index would gain all the sites that are in the AdSense index so search results would potentially improve. However, it would give an unfair advantage to google publishers.

[ Parent | on HN ]

Re:Google search only has 60% of my content from 2006 by WaltPurvis on Saturday April 27, 2019 @10:48AM (#19765795)

> If Google was so evil, why would we purposefully send traffic to sites where we only get a percentage cut of the revenue?

Can you explain this? Not sure if it's your phrasing or what, but I'm not getting what you mean.

[ Parent | on HN ]

Re:Google search only has 60% of my content from 2006 by trickstra on Saturday April 27, 2019 @11:28AM (#19766001)

because it's better to have 10% of all visits than 100% of only some visits. Google is not the only way people get to those pages. Yes, maybe phone registries, but since the policy cannot be split, we are talking about the whole internet.

[ Parent | on HN ]