My God, I'm so tired of this response.

It's not a person. It's software. It shouldn't be wrong just like humans.

Imagine opening your banking app and your total savings is wrong. Would you say, "It's okay, people make mistakes, too!" or would you be pissed that your bank's software is incorrect? Why are we pretending like this is any different?

Because LLMs aren't doing calculations that result in precise mathematical answers like your bank should be doing. All they are is very advanced pattern-matching machines. They match patterns, but those patterns don't always relate to accurate information.

I'm not pissed when google doesn't have a good result for my search on the first page, no.

There you go. The difference is that a human can transparently explain to you step by step how they have concluded to a given answer. Humans have always used multiple methods to explain their decisions transparently when questioned to.

AI machines and especially neural networks which LLMs are based on still cannot explain themselves and are as transparent as a black-box.

A real person would say "I don't know" occasionally.

And a really intelligent person would say "I don't know" a lot.

I don’t usually ask a random person about some topic I want sound information on. I ask an expert on that topic. If ChatGPT-like AI can’t fulfill that role (and ChatGPT usually can’t), then they’re not very useful for that.

And yet, when I google something, am I landing on an experts page, or a seoptimizer who's day job it is to write expert like looking content?

Search is absolutely like asking a random person right now. Whether I get raw hyperlinks back on a page or a chat window, the results are as good as their source data. Garbage in, garbage out.

Google is just as confidently wrong giving me garbage links as chat is as giving me garbage recommendations.

Why are we ok with “AI is equally as bad as the current bad thing?” Even then, it falls flat because the average person doesn’t understand this as well as you do. There will be some severe consequences because of this.

Just because it doesnt solve the garbage in problem, doesnt mean it doesnt solve other problems in the realm of usability.

This is why I am very concerned about people ascribing attributes to LLMs that they simply don't have. The real danger with LLMs, it seems to me, is that people seem to be viewing them as intelligent or as some sort of "truth machines". They are neither. Saying that isn't saying they aren't powerful tools, of course, but a misunderstood tool can be a very dangerous thing.

The difference is that with search you get a whole range of linked resources, and up to now (it may change with AI-generated content going forward) the style of the content usually gives enough signal to make an assessment, as well as comparing content between sites, and you learn which sites to trust.

With ChatGPT, correct and incorrect information is presented identically. You have nothing to go on except falling back to web search to fact-check ChatGPT’s output. This can still be useful as a starting point, but a world with only ChatGPT and no web search would be horrible, would overall be a step back compared to a world with only web search and no ChatGPT.

Yes, but at least with a web search I can click on a different link. If chatGPT gives you a wrong answer, where is another answer to compare it to? It would be like if wikipedia replaced web search. It's usually a good source for summarizing things, but if you rely on it as The Source, then it becomes a problem.

Not only that, but if it's a topic you don't know much about and you aren't seeing information from a variety of different sources, how would you even be able to know it's a wrong answer?

Yes, that also. Or it could be an unresolved topic with different possible answers, but how would you know if it just gives you one? If you ask about the measurement problem and it says decoherence has solved it because MWI, then you're getting short changed. There's a lot more to the debate.