This doesn't seem to make much sense. Surely us being able to prove that something is outside their modelling ability doesn't affect whether it is or not. If I prove something true tomorrow, whatever I proved was also true today.

Or do we have a proof that everything beyond them has already been proved and there are no more proofs left to find?

The existing LLM training methods on the other hand give the results that are hard to distinguish from "thinking like people," judging by the end results.

The connectionist models are basically a proposed highest possible abstraction of naturally evolved intelligences so it is in retrospect not surprising that passing some hardware scaling threshold they will start doing things that humans and animals do

It's more that formal Turing equivalence plus the Church-Turing thesis tells us that we're not allowed to assume counterarguments based on magic, there's no magic sauce barrier that prevents AI from running on CPU models. The algorithms exist and most of us thought discovering them would be hard.

The empirical surprise was that human intelligence is maybe not that computationally complex after all. (The entirety of academia was basically caught off guard.) That's one not unreasonable interpretation given recent events.

What do you mean exactly? "Thinking is not an algorithm."?

It's a category mistake because thinking like we do requires a certain sort of algorithm or one operationally equivalent to it, whereas things that are Turing-complete include Befunge-98 and C++ templates. While something being Turing-complete implies that the cognitive algorithm could be implemented with it, clearly the categories are incorrect.

Note that, while the set of all LLMs with CoT covers Turing completeness, a specific LLM with CoT is only Turing-complete if it has the right weights, but training does not produce such LLMs. Note also that there are no UTMs in the real world since that requires unbounded resources. This is true all digital computers and well as brains so is not a fatal blow to LLMs, but the resource limitations play out very differently in LLMs and brains.

As for the dead troll comment, not only is it a foolish ad hominem on its face, it's quite false -- I'm an atheist and do not believe in souls.

> but training does not produce such LLMs

If we are talking about fundamental limitations it should not be an empirical observation: "does not produce" (which is factually wrong, BTW). It should be a fundamental limitation: "can not produce in principle."

I still don't understand what you are talking about when you say "category mistake." I was talking about computational capabilities of LLMs with CoT that their training can exploit, not about Befunge-98.

Not my problem.