If you want to search for the history of pineapples,
  you don't need 100 sites with the same info.

"Pineapples grow from a plant in the tropics."

"Pineapples are grown in equatorial countries."

---

"Pineapples, when ripe, have a very sweet taste."

"Ripe pineapples have a high concentration of sugar."

---

"The etymology of the word for pineapple is 'excellent fruit' in the language of Tupi, although the english English translation is an aberration, with a stem of 'pine' due to it's resemblance to a pine cone."

"Most European languages name the fruit using the root word 'nanas', from the original native language where it was grown. Many languages have adapted the root word, although a few have a large degree of uncertainty, such as the origin of the reference to 'apple' in the English word."

---

It would need a fairly decent AI to recognise the commonality of information in these examples.

I think people often underestimate the power of much simpler methods. You'd probably get substantial improvements over the baseline just by penalizing sites for having the same words too many times. I forget the exact number, but I think it's something like 50% of the words in your average article appear only once. So if two articles have even 5% similarity among the words they only use once, they are probably quite similar.

AI of course can get you much farther still, but you definitely don't need something that powerful for a convincing upgrade from baseline.

Google is way way past that point. It worked that way 10 years ago and was exploited, so they added many more signals, generally far subtler than that.

It's even more odd considering diversity in page ranking and novelty in recommendation are well studied (although not solved) problems in information retrieval, so it's not like there isn't a solution. And it's not like Google doesn't have the human or computer resources to tackle it.

The blackhat SEO would at least be interesting if not terrifying.

I would assume that to counter the junk/spam/wrong information that would develop (even unintentionally) about a topic you would have to combine this with a "trust" ranking like PageRank.

Basically, only more mainstream sites have the right to present different views which leads you back to square one: Repeating information.

If you market it right, I think you could have a wildly successful "counterculture" search engine.

You could also use current search algorithms but add a penalty for redundancy. Or perhaps group similar results. I've always like the idea of column-based web results. I suppose some of what Google does has this flavor.

Even better, add settings the user can tweak for which heuristic setup you want. Put it behind enough layers of dashboards that it doesn't terrify the average user.

The heuristics you're looking for are accuracy and relevance.

The problem is that PageRank is no longer delivering on these.

Certainly it's not _as_ dangerous.