If I am repeating this back correctly, the argument is that the process itself looks nothing like human reasoning and has a number of technical limitations and even hacks that are in no way attributes or qualities of reasoning. Therefore, it clearly cannot be in any way considered reasoning. Temperature is one element of this, but there are others which you could continue to enumerate beyond even what's written above.
I can get behind part of that argument, certainly, and I appreciate you elaborating on it. I think is what I was trying to say with the part about me believing that it's not useful to think of it as reasoning. This is very different from what we might consider reasoning in very meaningful ways.
I also agree with you also that parts of this is just loaded language, as it is anthropomorphizing what is fundamentally just a bunch of matrices and non-linear functions.
I think where we differ is probably on that "when it's not even really close" part of it, at least in what I mean is "close" versus what I think you mean.
While I (think) we agree that obviously it's a different process, I do think that the input->outputs and the different qualities of input->outputs (like the so-called reasoning tokens) above can often seem quite close to the different inputs and outputs of some human reasoning. That's why I was saying that didn't see how the process works, like temperature, is relevant. Putting the processes aside, if you black box a human and a language model and put us head to head on reasoning tasks, sometimes you're going to get quite similar results.
I'm basically saying that, sure, an LLM or foundation model is clearly a Chinese room, without any understanding. What are we comparing it to, though?
Now, I don't have any kind of training in biology, but I have been led to understand that our brains are quite complex and that how their function arises from the underlying biological processes. is still fairly poorly understood. Given that, I tend to discount the degree of difference between the processes themselves and just look at the inputs and outputs. It's not obvious to me that we aren't ourselves Chinese rooms, at least to some significant degree.
So _maybe_ it's fair to try to compare what the outputs of these Transformers are to what our outputs would be. If it walks like a duck, and talks like a duck, does it matter?
Obviously, that's not fully correct -- how the output arises _has_ to matter somewhat. The fact I am sitting here writing this, and not an AI, refutes that point to some degree. And if I am understanding your thoughts correctly, I fully agree that the process really is nothing close. I just don't see how it can be a clear-cut issue on the basis of analyzing the Transformer algorithm itself.
Depends on what your goals are. LLMs can get to a state where they contain a lot of human knowledge, with a lot of detail, to answer a lot of questions, and be used in many different ways. If your idea of intelligence is akin to having a bunch of experts on tap in all the different areas, then LLMS are totally fine.
I personally want something that can solve problems, not just answer questions. For example, lets say I want to build a flying car, quadcopter style, in my garage. Given the information that exists on the internet and availability of parts, this is a deterministic problem. Given that prompt, I want a set of specific instructions like "buy this part from here", "send this cad model to sendcutsend.com here and select these options", all the way down to "here is a binary file to load on the controller". And along the same lines, the AI should be able to build a full simulator application Flight Sim style, where I can load the file and play with controls to see how the thing behaves, including in less than optimal conditions.
Whatever that model does under the hood, that is called reasoning, and it certainly won't be structured like an LLM.
It seems like you want LLMs to be able to use tools (which some of them do. For instance, see the search engine chat bots, which can do searches) and make independent decisions (Search term here is "agent", I don't know how well they work, but I wouldn't personally let my computer do things like that unsupervised). However, I personally wouldn't consider these things to be a prerequisite to reasoning.
I would consider being able to solve a large range of problems that a human could solve with just pencil and paper to be reasoning. LLMs don't really seem to be as good as humans, but the certainly CAN solve these types of problems.
I cannot believe this is true. LLMs are awful at whatever problems are not present in the dataset used for training. They are very bad at planning problems for example, because they cannot possibly memorize every single instance, and they cannot reason to reach a solution, but a black-boxed human of course it can.
So what? Can you propose another method to make a computing device understand language? The method of the creation of the output does not stipulate anything about the nature of the thing creating it. If someone could map out a human brain and tell you how thoughts are made and added a 'all that is being done is' in front of it, does that make your thought creation trivial?
> An LLM on top of an LLM (i.e using context to generate inputs to an LLM) is just a fancier LLM.
This is called a tautology. You have not given any compelling reasons why an LLM cannot do anything, so calling something another LLM is not compelling either.
> To really understand all of this, all you need to do is look at how Transformer works, namely the attention block. There is no such thing as Query, Key, and Value in the sense of how they are implied to be used. The may as well be called A,B,C, as they are all learned in training, and can be freely interchanged in naming. All you do for inference is multiply the output vector by A,B,C to get 3 matrices, then multiply them together (technically with a scaling factor for 2 of them, but again, doesn't matter for which 2, and the scaling factor can be built into the matrix itself)
Here is how it works, so therefore it must meet some criteria I have imposed arbitrarily.
> So we have to rely on hacks like temperature to make it appear like reasoning, when its really not even close.
You still haven't produced any valid argument at all, for why one thing would be evidence of the other.
It should be pretty clear to anyone that human brains aren't just one giant compute functions with a limited set of outputs. There is no concept in your or my brain what 12074389762193867*2398720876324 is, but we can certainly figure it out, some even with good memory with complete sensory depravation.
If you disagree with this, you are entitled to your opinion, but your comments on the state of AI are just irrelevant.
My post was pointing out basic flaws in reasoning and trying to provoke some thought about something where it appeared to be lacking. Unfortunately I did not succeed, since a myopic view made you hallucinate that I was saying something definitive about something I was not.
Irrelevant, indeed.
This is the problem with using loaded language like "reason" and "interpret". The model is not interpreting anything. All that is being done is a multdimentional map lookup with statistics.
> also don't see how that idea would fit in with the o1 models, which explicitly have "reasoning" tokens.
An LLM on top of an LLM (i.e using context to generate inputs to an LLM) is just a fancier LLM.
To really understand all of this, all you need to do is look at how Transformer works, namely the attention block. There is no such thing as Query, Key, and Value in the sense of how they are implied to be used. The may as well be called A,B,C, as they are all learned in training, and can be freely interchanged in naming. All you do for inference is multiply the output vector by A,B,C to get 3 matrices, then multiply them together (technically with a scaling factor for 2 of them, but again, doesn't matter for which 2, and the scaling factor can be built into the matrix itself)
And because you can unroll matrix multiplication into a 2 layer neural network, that means that any LLM in its current form today can be represented as a set of linear layers. And we know that a set of linear layers is simply a function. And every function has a finite range for a finite domain. And the inability to expand that range given a finite domain means its not reasoning.
So we have to rely on hacks like temperature to make it appear like reasoning, when its really not even close.