ChatGPT is optimized to seem to make sense, it’s like talking to a world class con artist you won’t notice it’s talking nonsense. Thus is subtly dangerous in ways that aren’t obvious in the beginning.

Sure it seems great at writing boilerplate code, unfortunately it’s really good and writing the kind of bugs that you don’t notice. This is maddening in a completely novel way, like working with an actively malicious autocomplete.

You can verify it’s accuracy. The metaphor of “con artist” is a poor one. It has no adversarial interest. A con artist does. Furthermore, con stands for confidence. A con artist manipulates confidence. Gpt and other language models are “what it says on the tin.”

Not understanding the fallibility of LLM is not the fault of the technology. It’s the user’s fault.

Not at all like a con artist. More like a tool that can be dangerous in the hands of the uneducated. That is a very large set of human tools, from nuclear reaction to pharmaceuticals to the automobile. Perhaps we just need “drivers ed” for large language models.

The problem is needing to verify accuracy makes it a really bad search engine replacement. A good con artist is generally truthful, they just know how to seem truthful even when they lie which exactly the issue with trusting ChatGPT etc. You can’t tell if the model is telling bullshit unless you already know the truth.

This doesn’t make them useless in general, just a poor fit if you can’t easily fact check the results.

Good point. I don’t think they will be a good search engine replacement, but maybe a powerful supplement for some users. And honestly with SEO and other issues even search engines can be manipulated in ways that an average user might have difficulty detecting.

Easy, you use a second source and double check.

In google you would go to another search result.

For ChatGPT you actually need to verify via Google or actually try it when it is code.

What’s the point of using ChatGPT when searching if you still need to verify the information through another search?

Conversely I asked it some simple questions about some songs' harmonic structures, and music theory, and it got things quite wrong.

There's already an example of it in the article where it incorrectly stated the tritone of F was Bb.