> The issue I think is that it's pulling from too many sources. There are plenty of sources that are pretty machine readable that will give it good answers. There's a lot of training that can be eked out from the legal databases that already exist that could make it a lot better. If it takes in too much information from too many sources, it tends to get garbled.

No, this is a common misunderstanding about the way these things work. A LLM is not really pulling from any sources specifically. It has no concept of a source. It has a bunch of weights that were trained to predict the next likely word, and those weights were tuned by feeding in a large amount of text from the internet.

Improving the quality of the sources used to train the weights would likely help, but would not solve the fundamental problem that this isn't actually a lossless knowledge compression algorithm. It's a statistical machine designed to guess the next word. That makes it fundamentally non-deterministic and unsuitable for any task where factual correctness matters (and there's no knowledgeable human in the loop to issue corrections).

If that's true, then it'll never get to the point that it would need to, and its reliability would always be too low to use.

Exactly. The paradigm is wrong, and in order to get past the problems we need a new paradigm, not incremental improvement on LLMs.

If it is never pulling from a source, then why is it able to provide citations?

check the chat bot on you.com:

https://you.com/search?q=list%20top%205%20best%20selling%20c...

Asking "Can you cite some legal precedence for lemon law cases?" gives an answer containing

"In California, for example, the California Supreme Court in the case of Lemon v. Kurtzman (1941) held that a vehicle which did not meet the manufacturer's express warranty was a "lemon" and the manufacturer was liable for damages."

I dont think that case exist, there is a first amendment case Lemon v. Kurtzman, 403 U.S. 602 (1971) though.

I can't find any reference to Kurtzman or 1941 in any of the references. I think the answer is that the AI generating the text, and the code supplying the references are distinct and do not interact.

You.com is hugged to death right now, but from what I can see it's a different kind of chatbot. It looks closer to Google's featured snippets than it is to ChatGPT.

That kind of chatbot has different limitations that would make it unsuitable to be an unsupervised legal advice generator.

One useful way to think of language models is that they are statistical completion engines. It attempts to create a completion to the prompt, and then evaluates the likelihood, in a statistical sense, that the completion would follow the prompt, based on the patterns in the training data.

A citation in legalese is very common. A citation that is similar or identical to actual citations, in similar contexts, is therefore an excellent candidate for the completion. A fake citation that looks like a real citation is also a rather good candidate, and will sometimes squeak past the "is this real or fake?" metric used to evaluate generated potential responses.

This may seem like "pulling from a source" but there is no token, semantic information, or even any information in the model about where and when the citation was encountered. There is no identifiable structure or object (so far as anyone can tell anyway) in the model that is a token related to and containing the citation. It just learns to create fake citations so convincingly, that most of the time they're actually real citations.

This explains some of the particular errors that I've seen when poking and prodding it on complex legal questions and in trying to get it to brief cases.

ChatGPT can provide correct citations because somewhere deep in its weights it does lossily encode real texts and citations to real texts. That makes real citations in some cases be its most confident guess for what is supposed to come next in the sentence. But when there isn't a real text it is confident about giving a citation about, but it still feels like a citation should be next in the output, it will happily invent realistic looking citations to texts that have never existed and it has never seen in any sources. On the outside, as readers, it's hard to tell when this occurs without getting an outside confirmation. I'm assuming though that to some degree it is itself aware that a linked citation doesn't refer to anything

A few days ago on HN there was a short story of five paragraphs that started badly & finished OK, and I wondered if some operations research tricks could be applied. One is forward-backward-forward planning, this produces better schedules, and if applied would create a better opening, and perhaps ending (i.e. model is run three times).

In the case of citations, you really need a language model & a fact model. The language model then passes over to the fact model, then back to the language model. This means double(+) training.

I suppose the fact model could include things like Wolfram (also discussed on HN).

The example you give isn't necessarily a valid one. You're asking for a specific piece of knowable, measurable data -- one that has a single right answer and many wrong answers. Legal questions may have conflicting answers, they may have answers that are correct in one venue but not in another, etc., I have't yet seen any examples of an AI drawing the distinctions necessary for those situations.

thanks very much for this insightful response. i haven't heard this perspective before but it makes sense.

ChatGPT has a lot of trouble understanding jurisdiction and what constitutes controlling precedent over what in which jurisdictions. As in it has no conception of it at all and gets it really badly mixed up. It doesn't understand the hierarchy of any court system so there are some questions that it will just always get wrong.

> If it is never pulling from a source, then why is it able to provide citations?

If you have the training set, and models that summarize text and/or assess similarity, and some basic search engine style tools to reduce or prioritize the problem space, it seems intuitively possible to synthesize probably-credible citations from a draft response without the response being drawn from citations the way a human author would.

Kind of a variant of how plagiarism detection works.

What if we prompt the LLM to generate a response with citations, and then we have program which looks up the citations in a citation database to validate their correctness? Responses with hallucinated citations are thrown away, and the LLM is asked to try again. Then, we could retrieve the text of the cited article, and get another LLM to opine on whether the article text supports the point it is being cited in favour of. I think a lot of these problems with LLMs could be worked around with a few more moving parts.

Definitely, no one is arguing that an AI lawyer will be the near future, but I can totally see it being good enough for the vast majority of small scale lawsuits within 10-20 years.