Remember running one of those RSS-to-Wordpress plugins that scraped the post from some blog and just reposted it on mine. I monetized it, and it was shocking how much attention it was getting. Those were the days. I imagine this practice has grown even more prevalent.

Google Says 60% of the Internet Is Duplicate.

I say 50% of the Internet is Duplicate. (N=2)

Google Says 60% of what they scrape needs to be filtered for duplicates?

Google Says 60% of the Internet is Duplicate.

What does this even mean? If I break up all of the data on the internet up into pieces, wouldn't every chunk be a duplicate of another chunk if we exploded the samples into small enough pieces?

Did anyone else not click the link tho?

The light gray text in the slide [1] suggests that this is about multiple URLs returning the same content:

> 1. remove protocol duplicates - favor HTTPS

> 2. remove www/non-www

> 3. remove URLs with useless parameters (sessionID?)

> 4. remove slash/no-slash variant

> 5. [...] [che]cksum dups

I haven't seen the talk, but I guess that the 60% is after taking 1-4 into account and directly comparing checksums of the full response (body?)

[1] https://twitter.com/suzukik/status/1595679289873428480

That just means that Google's index of URLs contain duplicates, not the Internet itself.

Google indexes `/page` and `/page.aspx` as two unique pages with duplicate content. All that happened was that the search spider clicked a link to a server-rendered page that can be rendered with a non-canonical link.

Thanks mostly to Google themselves, with their algorithms rewarding useless copied content.

Yeah, I don't get it. Google happily shows all the useless Stackoverflow copies in the search result and I guess that Google could easily filter such garbage out. But they do not. Why?

Those duplicate sites exist to serve ads, Google profits from selling ad space...

So does Google send users to sites with original content, or to those with Google ads and tracking?

I too have seen googles search results for any technical topic.

Everything is afaict generated content that mostly repeats itself enough times to fit in ads.

Don't forget to also thank the idea of "streaming" instead of caching; so they know exactly what, when, and for how long you listen/watch any content.