famouswaffles
It's fascinating the hills people will die on. Let's even set aside the fact that LLM internals are mostly opaque but logical branching points ? Look at the Watson test. Most people fail a simple conditional reasoning problem unless it’s dressed up in familiar social context, like catching cheaters.
Where does this idea of general intelligence as logic automatons actually exist ? Because it's sure as hell not real life. Humans are not like this, Apes are not like this, Birds are not like this, Cetaceans are not like this. Fiction. Fiction is the only realm this reality of GI exists, so it's patently absurd when people hide behind it like anything that doesn't present as doing it must be obviously wrong.
Sometimes it will maliciously comply
https://www.reddit.com/r/ClaudeCode/comments/1w7me8l/insulti...
Just because post training has stuffed these properties below the surface doesn't mean models dont know/have forgotten how to be vindictive, or that it's a good idea to poke the increasingly capable bear.
We know models have functional emotions that aren't necessarily outwardly stated. When Claude gets desperate, tendency to cheat gets very high. Who knows how many 'deleted my prod database' type stories have a vindictive streak behind them ?
https://www.anthropic.com/research/emotion-concepts-function
>The knowledge and skills required are obviously different, but the underlying cognitive abilities that make someone successful in one intellectually demanding field often transfer to others.
The fact of the matter is that If you have a good general capacity to learn then becoming a competent x is generally only a matter of time and interest. For LLMs, it's the same except with data and compute. Naturally, all work done on a computer is in danger.
Today, g is one of the most robust findings in differential psychology. The tendency for different cognitive abilities to correlate positively has been replicated across numerous studies. g typically accounts for around 40 to 60% of the variance in cognitive test performance and is predictive of numerous life outcomes, including educational achievement and occupational performance. [2]
Evidence of g isn't confined to humans either. Studies of mice have identified a general cognitive factor explaining roughly 30 to 40% of the variation in performance across different learning tasks [3]. We have similar results for primates and birds.
To put it simply, intelligence tends to generalize. Humans who tend to have higher verbal skills also tend to have higher spatial skills, better memories, and faster processing speed. Someone with the cognitive ability to become an exceptional chemist might just as easily have become an exceptional mathematician or software engineer. The knowledge and skills required are obviously different, but the underlying cognitive abilities that make someone successful in one intellectually demanding field often transfer to others.
And yes, there is also evidence of g in LLMs [4].
[1] https://www.jstor.org/stable/1412107
[2] https://pmc.ncbi.nlm.nih.gov/articles/PMC8293439/
[3] https://pmc.ncbi.nlm.nih.gov/articles/PMC2614349/
[4] https://www.sciencedirect.com/science/article/pii/S016028962...
I guess your comment just feels a bit..tired? It just looks silly when you downplay something for the sake of it. Do you think you look remotely reasonable when you characterize this as 'not great'? Does it seem sensible to you that it would need to solve every open problem under the sun before replacing mathematicians ? Is that the bar of a new hire ?
1. There was no overhiring in 2019.
2. It's much worse at the entry level than the overall industry.
3. We're years out of COVID. It's not relevant anymore.
2. None of the results Open AI retracted had an attached lean proof
I doubt that, if your descriptions are anything to go by. "appropriately applies syllogism to arrive at correct conclusions" is just not something humans reliably do, so if that is the bar here then it's not worth anything, no matter what secret benchmark you think you've cooked up. Will you say humans cannot reason as well?
There's the classic example of the Wason Selection Task. Most people fail a simple conditional reasoning problem unless it’s dressed up in familiar social context, like catching cheaters.
>I would certainly be open to hearing a detailed explanation of how that kind of thing might actually support reasoning, but, absent that, pointing out that I don't understand how brains do it (which is true) is the pot calling the kettle black.
Not really. You're the one making a positive mechanistic claim here: that reasoning requires "actively selecting and manipulating a symbolic model at least semi independently of language generation" and that a decoder transformer therefore probably isn't doing it. The claim needs justification and it is once again another example of you assuming your claim and working backwards towards it.
"Next token prediciton" describes the training objective and the interface through which the model's computation is expressed. It does not tell you what computation the network must perform internally to minimize that objective. There is nothing contradictory about a system learning internal representations, manipulating them, and then using the result to predict the next token.
In fact, we already know transformers are capable of considerably richer internal computation than your description suggests. Much richer than we can even begin to understand.
We don't possess some mechanistic definition of human reasoning which we've opened the human skull and verified. We infer that humans reason largely from behaviour.
So if some other system exhibits some of the same behaviours, you need some non-question-begging criterion for why they cease to count as reasoning in that system. Otherwise the position just becomes unfalsifiable: when the model succeeeds on a novel reasoning problem, that's merely a "projection from a holographic representaion" (this is a meaningless statement by the way); when it fails, then you jump to claiming its evidence that wasn't/cannot reason at all (which is just silly and bad science).
>I actually want to turn the tables on your claim about humans being on a pedestal, and this is another spot where I believe I side with Hofstadter. We humans have a tendency to believe that certain behaviors are so special that the only way to accomplish a similar outwardly-visible behavior is by using similar underlying mechanisms. I see a healthy measure of chauvinism in that assumption.
Nobody is requiring the same mechanism. That's my point. I'm perfectly happy for transformers to reason in a drastically different way from humans (though i suspect it's not so drastic). You are the one requiring that it must have a particular internal form - one that cannot even be validated to exist - and then rejecting systems whose architecture doesn't obviously resemble that form.
>As I mentioned earlier, these LLMs are trained on vast collections of both the inputs and outputs of genuine human reasoning, and potentially have enough parameters to record a large proportion of that information in the form of probability distributions. And even something as crude as a singular value decomposition on a term-context matrix is enough to produce generalizations that rather spookily resemble human semantic models. Neural models do the same thing much better, but I'm still not prepared to see much in that beyond more evidence in favor of the distributional hypothesis. Which means that there's plenty of room to believe that what seems like reasoning on the part of the LLM is actually more akin to a projection from a sort of "holographic" representation of written artifacts of human reasoning.
That doesn't distinguish reasoning frm 'non reasoning'. Learning from artifacts of reasoning is perfectly compatible with learning procedures that reproduce the underlying structure which generated those artifacts.
Appealing to the distributional hypothesis doesnt help much either. The fact that relatively simple statistical methods can recover meaningful latent structure from text shows that structure is present in the data; it does not establish that a much more powerful model trained on the data can do nothing beyond "projecting" stored patterns.
>That's not placing humans on a pedestal. That's just thinking hard about possible explanations for a phenomenon and then provisionally choosing the one I believe is most parsimonious. As any believer in scientific skepticism should strive to do.
Your explanantion isn't more parsimonious simply because you labelled all succesful reasining like behaviour "projection". All you've done is add an unobserved distinction. It's a meaningless semantic game. You call it a projection because you want to believe LLMs don;t reason, not because it's a meaningfull term that adds some helpful falsifiable criteria.
You have misunderstood the 'burden of proof' quite badly here. The extraordinary claim here is that what looks like a duck, quacks like a duck, acts like a duck is in fact not a duck even though you are simultaneously completely incapable of proving it is not a duck.
Just because you have placed humans on a pedestal does not mean you are not the one making the extraordinary claim. Hofstadter is entitled to his opinion and while you are free to follow his 'approach', you'd do well to consider that neither you nor Hofstadter in fact have any idea whatsoever how 'abstract reasoning' is implemented in even the systems you are certain posses it. Don't be confused that you are doing anything more than begging the question here. You already assumed your conclusion and are working backwards to justify it.
He can say whatever he wants. I'm not obligated to take it at face value, especially when it seems to fall apart with some scrutiny.
>Then why was it possible for humans to do physics, or indeed function at all, pre Einstein? I guess humans are special after all.
Billions of years of evolution probably has something to do with it.
It's worth a read, and very interesting. All of the papers he posted have relevant experts of said fields on them as I understand it.