No, you are probably right. It has no guardrails for generating a true citation. When you ask it for citations, its programming generates the form of citation. As the authors of the article discuss, it has no model of reality to understand that citations are not just a form of words and symbols put together.

Indeed and it gave several other actual citation with seemingly correct summaries.

And it's definitely less prone to obvious bs than just a few weeks ago. For a lot of general things, it gives a summary paragraph that's seems a distillation of standard information - the main problems seem to appear when these "scripts" lack consistency.

Edit: Once it "learns" to get citations right, you'll have to dig into those to see if it's right. It's easy to see that arms not looking good.

> Once it "learns" to get citations right, you'll have to dig into those to see if it's right. It's easy to see that arms not looking good.

But that's fundamentally not possible for the model as is, not in the sense you would normally think of citations at least. If it were to give an honest answer on "why did you say that Obama was born in Chicago", the model would generate training set example with propositions like Obama was born in ____, ___ was born in Chicago, ____ was ____ in ____ and so on - because that is how it learns.

Instead, to generate what you would actually want as citations, it would have to have a way to look up the statement it just produced in a corpus of "citeable materials" and find the closest match, based on some entirely different ML model most likely (presumably something similar to Google's search algos).

> As the authors of the article discuss, it has no model of reality to understand that citations are not just a form of words and symbols put together.

Which is the exact same issue that plagues it for the answer it’s generating a citation for.

I believe that it can be trained to improve this, and I believe it can be nearly solvable but I don’t believe it’s a solvable problem.

It can be trained to get facts mostly right using old-school academic tests and adversarial/supervised training, like an elementary school child. Today, it (supposedly?) punts math and some other logix to a separate function, and it just compiled the prompt. That can work for validating citations and “known facts” in many cases.

But again, it can’t “think” and understand when it returns something illogical (regardless of fact correctness).

The problem is mostly solved though. ChatGPT doesn't have the feature to cite its sources, but some other models do https://www.deepmind.com/blog/building-safer-dialogue-agents

I honestly doubt very much that a reinforcement learning based approach can scale to the breadth of facts that such a model would need to know about. Is there any reason that using a rewards-based approach to cover one set of facts would in any way generalize to a completely different area of expertise (say, rewards for true facts in mathematics generalizing to questions about literary criticism, or new world cultures), or would you have to include (manually curated) rewards for all areas?

It’s really good at showing what types of scientific books and papers should exist. GPT5 will just write them, so it isn’t lying.

It has no concept of lying.

GPT5 will.