They couldn't find a more apt demnonstration of what an LLM is and does if they tried.

An LLM doesn't know more than what's in the training data.

In Michael Crichton's The Great Train Robbery (published in 1975, about events that happened in 1855) the perpetrator, having been caught, explains to a baffled court that he was able to walk on top of a running train "because of the Bernoulli effect", that he misspells and completely misunderstands. I don't remember if this argument helps him get away with the crime? Maybe it does, I'm not sure.

This is another attempt at a Great Robbery.

For those who want to read about the "Baroni" effect in the book: https://bookreadfree.com/361033/8879470

It goes on:

> At this point, the prosecutor asked for further elucidation, which Pierce gave in garbled form. The summary of this portion of the trial, as reported in the Times, was garbled still further. The general idea was that Pierce--- by now almost revered in the press as a master criminal--- possessed some knowledge of a scientific principle that had aided him.

How apropos to modern science reporting and LLMs.

> An LLM doesn't know more than what's in the training data.

Post-training for an LLM isn't "data" anymore, it's also verifier programs, so it can in fact be more correct than the data. As long as search finds LLM weights that produce more verifiably correct answers.

Please demonstrate that you know anything more than what was in your training data.

I know that some specific parts of what's in my training data is false, even though it was in there often. I am not just the average-by-volume of everything I've read.

I doubt that their training data is internally consistent. I am sure there are plenty of conflicting statements that it gets trained on.

It's a good question, but there are things I figured out by myself, that weren't in my training data, some, even, where my training data said the exact opposite.

IIRC I was required to regurgitate this wrong answer to pass my FAA pilot exam.

Yeah me too, so it's found in many authoritative places.

And I might be wrong but my understanding is that it's not wrong per-se, it's just wildly incomplete. Which, is kind of like the same as wrong. But I believe the airfoil design does indeed have the effect described which does contribute to lift somewhat right? Or am I just a victim of the misconception.

I've always wondered if the acrobatic exams require repeating the same wrong answer. Obviously planes can't fly upside down.

Yeah, it's like asking a car driver (even a professional driver) to explain the Otto cycle. Enduser vs. engineer.

And your suspicion is right. The sad reality is that it's just a stochastic parrot, that can produce really good answers in certain occasions.

This honestly mirrors many of my interactions with credentialed professionals too. I am not claiming LLMs shouldn't be held to a higher standard, but we are already living in a society built on varying degrees of blind trust.

Majority of us are prone to believe whatever comes our way, and it takes painstaking science to debunk much of that. In spite of the debunking, many of us continue to believe whatever we wish, and now LLMs will average all of that and present it in a nice sounding capsule.