How is this different than regular intelligence? We grow up trained on a polluted dataset, being taught things that are subtly or completely false, and we often struggle to tell which things we have learned are false.

And what was the result of unchecked faulty learning on a corrupt dataset by natural intelligence?

Dark ages where millions died in pointless wars over who's imaginary diety is the true one. A third of europe dying of preventable plague. It took a rigorous system of citation and a publically funded system of education to crawl out of the drakness.

Should we try AI powred dark ages but with resource shortage and nuclear weapons?

Once in a while somebody does some actual on-the-ground research and people are convinced to change their minds through evidence and reasoning. So far AI can't do that, it'll just parrot whatever it hears more often, (apparently) with some degree of randomness.

Maybe it'll learn to adopt a new view, but for now it won't be through actual reasoning.

GPT has had up/downvotes for years. They were used to train ChatGPT. It’s not just “what’s the most likely answer”.

(The most likely answer is the letter “e”.)

It is true but then again, we constantly ajust the trust in sources based on what works and what doesn't.

If I watch a youtube tutorial teaching me how to bake pizza and it comes out totally wrong and a watch another that produces exelent pizza, I disregard the first channel.

And there are many more ways where humans check the reputation of sources automatically.

GPT currently lacks this possibility.

Well this is unfair because we haven’t given natural intelligence a chance yet.

What if we invented an algebraic notation for search indexing and allowed people to index and classify documents in their own way. Then a search engine could do something special with all that.

Human intelligence cannot progress without a notation to capture our thoughts in.

Are you aware of https://commoncrawl.org/ ?