Sometimes it will, sometimes it won't. The point is that it's "random", it has no way to tell truth from falsity.

Language models are unsuitable for anything where the output needs to be "correct" for some definition of "correct" (code, math, legal advice, medical advice).

This is a well-known limitation that doesn't make those systems any less impressive from a technical point of view.

How can this interface be useful as a search engine replacement if the answers are often incorrect?

Can we fix it?

Because earlier today it told me that George VI was currently king of England. And I asked it a simple arithmetic question, which it got subtly wrong. And it told my friend there were a handful of primes less than 1000.

Everyone’s talking about it being a Google replacement. What’s the idea? That we train it over time by telling it when things are wrong? Or is the reality that these types of language models will only be useful for generating creative output?

there are plenty of google queries that return incorrect answers, and they've been operating for decades

It's not the same.

If you ask a chat interface a question and it says "this is true", that's very different from a search engine containing a list of results where one of them might be untrue.

For one thing, you can look at all the queries and take a majority vote etc. Second, you can look at the source to see if it's trustworthy.

Doctors are often not totally correct, but they're useful.