They have a very complex multidimensional "probability table" (more correctly a compressed geometric representation of token relationships) that they use to string together tokens (which have no semantic meaning), which then get converted to words that have semantic meaning to US, but not to the machine.
The presence or absence of understanding can't be proven by mere association of with a "probability table", especially if such probability table is exactly expected from the perspective of physics, and if the models have continuously gained better and better performance by training them directly on human expressions!
On the other hand, until it's proven it'd likely be considered a hallucination. You need to test something before you can dismiss it. (They did burn witches for discoveries back in the day, deemed witchcraft). We also reduce randomness and pre-train to avoid overfitting.
Day to day human creative outputs as humans are actually less exciting when you think about it further, we build on pre-existing knowledge. No different to good prompt output with the right input. Humans are just more knowledgeable & smarter at the moment.
The same goes for humans. Most awards are built on novel research built on pre-existing works. This a LLM is capable of doing.
You're focusing too close, abstract up a level. Your point relates to the "micro" system functioning, not the wider "macro" result (think emergent capabilities).
An apple falls to the ground because of? gravity.
In real life this is the answer, I'm very sure the pre-carved channel will also lead to gravity.
Most of the time that data is already available to it and they merely need to a prove a thereom using existing historic data points and math.
For instance the Black-Scholes-Merton equation which won the Nobel economics prize was derived using preexisting mathematical concepts and mathematical principles. The application and validation relied on existing data.
LLMs only rearrange descriptions of discoveries. They can't recognize when their model contradicts reality because they never touch reality. That's not a solvable limitation. It's definitional.
We're clearly operating from different premises about what constitutes discovery versus recombination. I've made my case; you're welcome to the last word
LLM's these days have reasoning and can learn in context. They do touch reality, your feedback. It's also proven mathematically. Other people's scientific papers are critiqued and corrected as new feedback arrives.
This is no different to claude code bash testing and fixing it's own output errors recursively until the code works.
They already deal with unknown combinations all day, our prompting.
Yes it is brittle though. They are also not very intelligent yet.