Yeah I was going to call out MIT for pointing out the obvious, but there's enough noise/misunderstanding out there, that this kind of article can lead to the 'I get it' moment for someone.

A map, if it is useful (which the subjective human experience of reality tends to be, for most people most of the time,) is by definition a coherent view of the territory. Coherent doesn't imply perfect objective accuracy.

By definition, you say? By what definition?

I don’t see why something would need to be entirely coherent to be useful. If you have a program which, when given a statement, assigns a value between 0 and 1, to be interpreted as if it were a probability of the statement being true, it needn’t define an entirely coherent probability distribution.

The coherence comes from the relationship between the virtual (mapped) data and the real data it represents (territory), and the ability of the map to correctly predict the attributes of the territory.

Human perception is coherent because it tends to accurately model the physical world from which it's constructed, by means of sensory input and memory, and if this were not the case, we would have gone extinct as a species ages ago. While it is true that the model of human perception isn't perfect, it would be incorrect to call it incoherent to the same or similar degree as the incoherence demostrated by LLMs, as jmole's earlier comment implies.

Even if one must consider the model of human perception incoherent, one must also consider it vastly more coherent than whatever LLMs model reality on, if anything.

That is emphatically not true - animals and small children that can't speak yet know about object persistence. If something has come from over there and is now here then it's no longer there.

LLM's do not have that concept, and you'll notice very quickly if you ask chemistry questions. Atoms appear twice, and the LLM just won't notice. The approach has to be changed for AI to be useful in the physical sciences.

Perhaps you are using the wrong LLMs that are not fine-tuned for chemistry.

https://www.science.org/doi/10.1126/science.adg9774

https://www.technologyreview.com/2024/10/18/1105880/the-race...

The argument was that with current word-based methods an LLM can puzzle out mechanistic problems better than a human. Turns out it can't at all, it makes errors that no human would. Anyone who looks at such attempts recognizes them as blather that is not rooted in reality.

The parent poster posted a paper where an AI guesses the Hamiltonian (nice, but with iterative methods you at least get an idea of the error associated with your numbers, not sure how far anyone should trust AI's there), maybe methods that do guesswork on topological networks would help. But I haven't seen those yet.

And yet that's exactly how the world actually behaves - see quantum mechanics. Object persistence is just a heuristic that we develop learning on our training data (our interactions with the macroscopic world).

You're inability to recognize high dimension relations between word tokens is no less evidence you lack coherent understanding.

Emmy Noether, paging Emmy Noether, Emmy Noether to the red courtesy phone please!

You can't say that given <conditions>, 1-phenylpropane rearranges to 1,2-diphenylpropane, you cannot pull whole atoms out of thin air for an indefinite amount of time. No human would make such a mistake, but AI does, and what's worse, it is impossible to talk the AI out of its strange reasoning.

We build approximations: we develop world models. That suffices. That the product is incomplete is just a matter of finiteness.

We build them as we do because we grow up in them first.

AI gets trained on a lot of different worlds (books, stories, science, etc. )

No, that is irrelevant and false. False, because we also access sub-worlds - the world model is made of contexts -, like LLMs (not «AI»).

Irrelevant, because for the intended purpose (reasoning over a reliable representation of the world) there is no need for an interactive stream - the salient examples of interaction, cause and effect, are provided by the dataset.

A large perception model "LPM" will not be superior to a large language model "LLM" if its underlying system is not superior.