"deterministally outputting information" neither do humans.

I don't see how the latter follows from the former.

Here's how I think about it: the fact that it can interpret the same words differently in different contexts alone shows that even on a temperature of 0 (i.e., lowest randomness possible) there could be something that possibly resembles reasoning happening.

It might be a mimicry of reasoning, but I don't think that having adjustable parameters on how random they are makes it any less of one.

I also don't see how that idea would fit in with the o1 models, which explicitly have "reasoning" tokens. Now, I'm not terribly impressed with their performance relative to how much extra computation they need to do, but the fact they have chains-of-thought that humans could reasonably inspect and interpret, and that they chains of thought do literally take extra time and compute to run, certainly points at the process being something possibly analogous to reasoning.

In this same vein, up until recently I personally very much in the camp of calling them "LLMs" and generally still do, but given how they really are being used now as general purpose sequence-to-sequence prediction models across all sorts of input and output types tends to push me more towards the "foundation models" terminology camp, since pigeonholing them into just language tasks doesn't seem accurate anymore. o1 was the turning point for me on this personally, since it is explicitly predicting and being optimized for correctness in the "reasoning tokens" (in scare quotes again since that's what openai calls it).

All that said, I personally think that calling what they do reasoning, and meaning it in the exact same way as how humans reason, is anthropomorphizing the models in a way that's not really useful. They clearly operate in ways that are quite different from humans in many ways. Sometimes that might imitate human reasoning, other times it doesn't.

But, the fact they have that randomness parameter seems to be to be totally unrelated to any of the above thoughts or merits about the models having reasoning abilities.

>he fact that it can interpret the same words differently in different contexts alone shows that even on a temperature of 0 (

This is the problem with using loaded language like "reason" and "interpret". The model is not interpreting anything. All that is being done is a multdimentional map lookup with statistics.

> also don't see how that idea would fit in with the o1 models, which explicitly have "reasoning" tokens.

An LLM on top of an LLM (i.e using context to generate inputs to an LLM) is just a fancier LLM.

To really understand all of this, all you need to do is look at how Transformer works, namely the attention block. There is no such thing as Query, Key, and Value in the sense of how they are implied to be used. The may as well be called A,B,C, as they are all learned in training, and can be freely interchanged in naming. All you do for inference is multiply the output vector by A,B,C to get 3 matrices, then multiply them together (technically with a scaling factor for 2 of them, but again, doesn't matter for which 2, and the scaling factor can be built into the matrix itself)

And because you can unroll matrix multiplication into a 2 layer neural network, that means that any LLM in its current form today can be represented as a set of linear layers. And we know that a set of linear layers is simply a function. And every function has a finite range for a finite domain. And the inability to expand that range given a finite domain means its not reasoning.

So we have to rely on hacks like temperature to make it appear like reasoning, when its really not even close.

I see, I probably needed more coffee to read your initial note.

If I am repeating this back correctly, the argument is that the process itself looks nothing like human reasoning and has a number of technical limitations and even hacks that are in no way attributes or qualities of reasoning. Therefore, it clearly cannot be in any way considered reasoning. Temperature is one element of this, but there are others which you could continue to enumerate beyond even what's written above.

I can get behind part of that argument, certainly, and I appreciate you elaborating on it. I think is what I was trying to say with the part about me believing that it's not useful to think of it as reasoning. This is very different from what we might consider reasoning in very meaningful ways.

I also agree with you also that parts of this is just loaded language, as it is anthropomorphizing what is fundamentally just a bunch of matrices and non-linear functions.

I think where we differ is probably on that "when it's not even really close" part of it, at least in what I mean is "close" versus what I think you mean.

While I (think) we agree that obviously it's a different process, I do think that the input->outputs and the different qualities of input->outputs (like the so-called reasoning tokens) above can often seem quite close to the different inputs and outputs of some human reasoning. That's why I was saying that didn't see how the process works, like temperature, is relevant. Putting the processes aside, if you black box a human and a language model and put us head to head on reasoning tasks, sometimes you're going to get quite similar results.

I'm basically saying that, sure, an LLM or foundation model is clearly a Chinese room, without any understanding. What are we comparing it to, though?

Now, I don't have any kind of training in biology, but I have been led to understand that our brains are quite complex and that how their function arises from the underlying biological processes. is still fairly poorly understood. Given that, I tend to discount the degree of difference between the processes themselves and just look at the inputs and outputs. It's not obvious to me that we aren't ourselves Chinese rooms, at least to some significant degree.

So _maybe_ it's fair to try to compare what the outputs of these Transformers are to what our outputs would be. If it walks like a duck, and talks like a duck, does it matter?

Obviously, that's not fully correct -- how the output arises _has_ to matter somewhat. The fact I am sitting here writing this, and not an AI, refutes that point to some degree. And if I am understanding your thoughts correctly, I fully agree that the process really is nothing close. I just don't see how it can be a clear-cut issue on the basis of analyzing the Transformer algorithm itself.

>If it walks like a duck, and talks like a duck, does it matter?

Depends on what your goals are. LLMs can get to a state where they contain a lot of human knowledge, with a lot of detail, to answer a lot of questions, and be used in many different ways. If your idea of intelligence is akin to having a bunch of experts on tap in all the different areas, then LLMS are totally fine.

I personally want something that can solve problems, not just answer questions. For example, lets say I want to build a flying car, quadcopter style, in my garage. Given the information that exists on the internet and availability of parts, this is a deterministic problem. Given that prompt, I want a set of specific instructions like "buy this part from here", "send this cad model to sendcutsend.com here and select these options", all the way down to "here is a binary file to load on the controller". And along the same lines, the AI should be able to build a full simulator application Flight Sim style, where I can load the file and play with controls to see how the thing behaves, including in less than optimal conditions.

Whatever that model does under the hood, that is called reasoning, and it certainly won't be structured like an LLM.

I feel like that is a pretty high bar to call "reasoning". I would like to think I am capable of reasoning, and yet I would not be able to write out by hand a binary file to be loaded on to the controller without using a compiler or looking at an assembly reference manual.

It seems like you want LLMs to be able to use tools (which some of them do. For instance, see the search engine chat bots, which can do searches) and make independent decisions (Search term here is "agent", I don't know how well they work, but I wouldn't personally let my computer do things like that unsupervised). However, I personally wouldn't consider these things to be a prerequisite to reasoning.

I would consider being able to solve a large range of problems that a human could solve with just pencil and paper to be reasoning. LLMs don't really seem to be as good as humans, but the certainly CAN solve these types of problems.

> Putting the processes aside, if you black box a human and a language model and put us head to head on reasoning tasks, sometimes you're going to get quite similar results.

I cannot believe this is true. LLMs are awful at whatever problems are not present in the dataset used for training. They are very bad at planning problems for example, because they cannot possibly memorize every single instance, and they cannot reason to reach a solution, but a black-boxed human of course it can.

> The model is not interpreting anything. All that is being done is a multdimentional map lookup with statistics.

So what? Can you propose another method to make a computing device understand language? The method of the creation of the output does not stipulate anything about the nature of the thing creating it. If someone could map out a human brain and tell you how thoughts are made and added a 'all that is being done is' in front of it, does that make your thought creation trivial?

> An LLM on top of an LLM (i.e using context to generate inputs to an LLM) is just a fancier LLM.

This is called a tautology. You have not given any compelling reasons why an LLM cannot do anything, so calling something another LLM is not compelling either.

> To really understand all of this, all you need to do is look at how Transformer works, namely the attention block. There is no such thing as Query, Key, and Value in the sense of how they are implied to be used. The may as well be called A,B,C, as they are all learned in training, and can be freely interchanged in naming. All you do for inference is multiply the output vector by A,B,C to get 3 matrices, then multiply them together (technically with a scaling factor for 2 of them, but again, doesn't matter for which 2, and the scaling factor can be built into the matrix itself)

Here is how it works, so therefore it must meet some criteria I have imposed arbitrarily.

> So we have to rely on hacks like temperature to make it appear like reasoning, when its really not even close.

You still haven't produced any valid argument at all, for why one thing would be evidence of the other.

A good example of how to type a comment and yet not say anything.

It should be pretty clear to anyone that human brains aren't just one giant compute functions with a limited set of outputs. There is no concept in your or my brain what 12074389762193867*2398720876324 is, but we can certainly figure it out, some even with good memory with complete sensory depravation.

If you disagree with this, you are entitled to your opinion, but your comments on the state of AI are just irrelevant.

I don't remember making any comments on A.I.

My post was pointing out basic flaws in reasoning and trying to provoke some thought about something where it appeared to be lacking. Unfortunately I did not succeed, since a myopic view made you hallucinate that I was saying something definitive about something I was not.

Irrelevant, indeed.

o1 can also solve many arbitrary math problems that it could not have possibly seen in its training data. And it shows the steps that it uses to do so. How do you explain this without reasoning?

The notion is AFAIS that a deterministic algorithm is obviously not reasoning, and a deterministic algorithm interspersed with dice rolls is obviously not reasoning either.

Of course, some would beg to differ. It's quite common nowadays to believe that we are something like the latter.

Why is a deterministic algorithm not reasoning? Reasoning is very deterministic.

It's not about (in-)determinism really, it's about the algorithm part.

An algorithm that does something can in principle be ran by someone who doesn't know what the algorithm does. You could have a kid calculate an integral by giving it a sequence of directions whose purpose it doesn't understand (e.g. cut out some cardboard that matches the shape, put it on one side of the scale, place enough unit cardboard pieces on the other side until they are even, then tell me how many pieces you put).

Reasoning has more to do with how the problem came about. A person had to come against a certain problem, figure out a way in which they can solve it, then apply the (perhaps algorithmic) solution. The algorithmic part is only an artifact.

But isn't figuring out a way to solve also algorithmic? In a lot of cases it is simply bruteforce trying out different things based on the concepts you know about and mixing them.

You are right that relevant memories and analogous experiences come up and are used as building blocks in our evaluation/exploration of a problem, but it doesn't seem to me an algorithmic procedure at all.

You can trace out your journey in solving a problem, in retrospect, but could you encode it into a "solving-a-problem" algorithm?

I think you could extract some kind of generic template for problem solving: you come up with an idea, you evaluate whether it is the solution, you adjust the idea if not.

But this is a template, not an algorithm. Coming up with an idea has to do with filtering the old and new memories/perceptions that come to mind: does this one seem right? or this one? Evaluating whether it is right is also an active process of asking questions. It involves memory (of the problem to be solved), attention (to the potential solution), judgement (do they fit together?), etc.

None of these are a predetermined sequence of steps you apply mechanically, such as the child "solving an integral" above.*

*Of course, the child is problem-solving in the sense that it's trying its best to follow your instructions. "Did I cut it right?" "Are the scales even?" But this is not the problem of "solving an integral" to which it is completely oblivious to.

I think it can be an algorithm, it just that the algorithm will be a very complex one compromised of many different algorithms. It's not an algorithm anyone could practically follow in their lifetime. But there's plenty of algorithms people can't follow in real life.

If all one has is hammers, one can start seeing nails everywhere.

A screw? That's just a nail which will damage the wood.

A tomato? That's just a soft screw which splatters. Etc.

What purpose does seeing everything through the lens of an algorithm serve? Is the movement of an electron an algorithm? Is a polar planimeter an algorithm? [0]

We design algorithms to solve certain problems. It's part of our problem solving activity. But for what purpose would we go around, assuming things that don't look like algorithms are actually algorithms that are just outside of our reach? This doesn't solve a practical problem, so of what use is that, and where does it lead?

My long-winded answer is: We derive satisfaction from being in principle powerful. Our mechanistic/computational knowledge of nature allows us to bend certain parts of it to our will. If there are parts we cannot control, it's at least consoling that we in principle could know/control them. So we stretch computational/algorithmic terms as far as we possibly can. In the end, it envelops us as subjects. We end up in a rather cliche epiphenomenalism + causal determinism worldview:

- "Yeah, we have experiences, but they're just inefficacious artifacts of underlying chemistry."

- "You - the person who is reasoning - don't actually know what reasoning is like, it's really a very complex algorithm which we could never know or follow."

The only way such an uninspiring outlook can subsist is because it jives well with some modern dreams:

- "We only need X more compute and Y more years to bend Z part of nature to our will and bring utopia." (cue all the AI hype, see relevant frontpage entry [1])

- "If we're just a machine then maybe we can hack-reward-centers/optimize-drug-concoction/upload-to-mainframe-for-immortality" (cue quasi-immortality pitches and externally-enforced-happines pipe-dreams)

- "If I'm just a machine then I'm not responsible for my shortcomings - they're just the outcome of my wiring I cannot influence." (a nice supplement for absolving oneself from responsibility - to oneself)

- "If all is mechanical, then I'm just a temporarily embarrassed sovereign over everything. After all, if I just knew the mechanism behind things, then I could bend it to my will."

- "I have to believe this because it is obviously true." (maybe the saddest of them all, since it promises nothing except the joy of being right and having others be wrong. it also seeds the others)

[0] http://psychsciencenotes.blogspot.com/2015/07/brains-dont-ha...

[1] https://news.ycombinator.com/item?id=41813268

> What purpose does seeing everything through the lens of an algorithm serve?

To me at least it helps me understand how things work. What is an alternative? Because alternative seems like some sort of magic I wouldn't understand.

> Is the movement of an electron an algorithm?

I think there's a lot of argument and complexity to what an electron exactly is or does, and what its properties actually mean, but I would imagine in general from particle and other levels whole universe can be just a deterministic algorithm, and so anything can be an algorithm. Universe has certain laws and rules which could in theory be simulated, but the simulation must have more capacity than the universe itself has so we inside the universe likely can not do it unless we find a mechanism to somehow bypass this.

> But for what purpose would we go around, assuming things that don't look like algorithms are actually algorithms that are just outside of our reach? This doesn't solve a practical problem, so of what use is that, and where does it lead?

If I try to think of what is the algorithm behind something, it helps me understand it better. Intuitively I think there's a complex algorithm behind everything, and it's just a matter of spending time and effort to figure out what it exactly is. It's unrealistic to get close to the real detail and nuance of the algorithm, but already trying to figure out the algorithm brings me closer to understanding.

> We end up in a rather cliche epiphenomenalism + causal determinism worldview

Wait -- what is wrong with that? And also I don't think it's cliche, I think it is likely to be the case?

> - "You - the person who is reasoning - don't actually know what reasoning is like, it's really a very complex algorithm which we could never know or follow."

I mean writing it down on the algorithmic level is not something we can consciously follow easily. However our brain within us is following those algorithms in the level of efficiency that we cannot consciously follow at that speed step by step, just following the instructions.

> The only way such an uninspiring outlook can subsist is because it jives well with some modern dreams:

I think my outlook is at least inspiring to me.

> - "If we're just a machine then maybe we can hack-reward-centers/optimize-drug-concoction/upload-to-mainframe-for-immortality" (cue quasi-immortality pitches and externally-enforced-happines pipe-dreams)

I do think theoretically it would be possible to hack humans to have constant "heroin like euphoria". I'm not sure I exactly care for it, but I do think these things could be done, I just don't know when, is it 50 years, 100 years or 1000 years. Of course while I say right now that I don't exactly care for it, if I tried it for once I would be hooked on it forever, assuming it has no tolerance build up or other negative effects making me consider to quit it. But even real heroin is terribly hard to quit while it has tolerance build up and side effects.

> - "If I'm just a machine then I'm not responsible for my shortcomings - they're just the outcome of my wiring I cannot influence." (a nice supplement for absolving oneself from responsibility - to oneself)

I'm inclined to think that the World is deterministic, yet I happen to also think that I have reward mechanisms that make me ambitious in a sense that I want to achieve certain goals to feel rewarded and in order to achieve those goals I have to overcome many challenges and improve certain shortcomings because it's needed to achieve those goals. If someone is using those as an excuse they would likely be the type of person to find an excuse in anything anyway. And if they do have goals they want to reach they will be affected by that, because there's certain behaviour that will get you to your desired goals and certain behaviour that is not. Taking responsibility and ownership is rewarded and if you do that, you will reach your goals with higher probability. I don't buy into the sort of thing where "something is bad because some people might use it as an excuse". Because finding an excuse is usually about the mindset, not about what kind of excuses are possible to select from. An AI bot with good goals and reward system, despite it being very obvious that they are programmed and deterministic wouldn't go about to make those excuses. But an AI bot trained to make excuses and be rewarded for it, would keep making excuses no matter what.

Thanks for your thoughtful response.

The view you are elaborating has a logic to it. You could argue it's the zeitgeist, at least among educated and STEM-adjacent circles. Hence my comment of it being cliche: you see variants of it all the time, and it gets a bit jading by virtue of being wholly unconvincing (to me).

In general, I think the utility of seeing everything through a mechanistic/algorithmic lens is overblown. When I'm doing technical work, I put my STEM hat on, and sometimes write algorithms. For the most part, I let it rest though. And I don't feel I understand the world any worse for dropping such mechanistic world-images I may have held years ago (I'm more at peace with it, if anything). In hindsight, the ROI on the skill of mechanistically dissecting anything you look at is rather low and hardly transferable ime. The ensuing "understanding" a passing regurgitative satisfaction.

If there's anything I really want to push back on, however, it's the notion that the views you hold do not influence the range of ways in which you develop yourself in an important way. If one truly holds the view that one is responsible for one's actions, and not the whims of determination of chance, where is the space for the excuse "it's not up to me"? Views may not determine the course of action, but they can surely constrain.

Disentangling one's views from one's behavior can go a long way in measured, written discussions like this, but I don't see it being the case in real life. It is the case however, that we can be a hodge-podge of contradictory views and standards, and that we commit to one for a moment, then to another. This is a matter of strength and cohesiveness of character. And we are good at "saving face" in front of us and others. For example, if you've met people who partake in a vice yet will say stuff like "This is bad for me." - the actual underlying view is "This has obvious drawbacks but I still think the enjoyment is worth it." It's only when they can maintain the view that the drawbacks are not worth it, that they can break out.

> The view you are elaborating has a logic to it. You could argue it's the zeitgeist, at least among educated and STEM-adjacent circles.

You could argue that most opinions or beliefs about how the World operates are cliche or similar if that's the case, there's only so many different beliefs that make any sense at all to hold and it's likely there's a group of people holding those beliefs as well and that they are not original at all. And you could also argue that a belief that 2 + 2 = 4 is cliche, because so many people believe that to be the case.

> In general, I think the utility of seeing everything through a mechanistic/algorithmic lens is overblown.

That requires some sort of measurement on how many people see it through that lens and what they evaluate it as, but I'm seeing the opposite since most of the time I find it the other way around.

> And I don't feel I understand the world any worse for dropping such mechanistic world-images I may have held years ago (I'm more at peace with it, if anything).

I can't tell how other people understand the World, since one of the learnings throughout my life have been that different people ingest information in so many different ways. Some think in images, some are not able to picture things in their mind, some don't have inner monologue. So naturally there would be a different methods of understanding things. But if I think of myself, I understand things best when I think of them as algorithms or mechanical steps that I think through. I have trouble understanding or remembering facts on their own, without internalizing the mechanisms, the cause and effect after if I have done that, it feels to me like I actually understand it. I don't even know what other way there is to understand that. It's perfectly possible that there's some other innate ways of understanding things or having a feeling that there's understanding, that I can't get just because I'm wired in a way that makes me understand things only if I can think of it as an algorithm. E.g. what I've found fantastic for learning subjects myself is actually coding through them or trying to simulate them using code. It actively engages my mind to try and simulate whatever happens in the real world. And subjects I had trouble engaging with in school, if I go through coding them, I feel like I'm actually learning them and becoming smarter. E.g. if I want to truly learn biology I should build a simulation tool that will simulate how cells behave, whatever different things in cells do, how organs work, etc.

> If one truly holds the view that one is responsible for one's actions, and not the whims of determination of chance, where is the space for the excuse "it's not up to me"?

I still don't see it in that way. Everything being deterministic and algorithmic doesn't make me have those excuses. I happen to have a reward mechanism, that rewards me for e.g. eating. It's been shaped by the process of evolution. I have many other reward mechanisms. I didn't choose those reward mechanisms myself, but I strategize on how to achieve those rewards, and I know that playing a victim is not a way to achieve your goals. Certainly there are people who mistakenly might believe that, but it happens to both, whoever believes in determinism and whoever believes in some sort of free will. I know that if I do good work, I get good rewards. So I do good work.

> Disentangling one's views from one's behavior can go a long way in measured, written discussions like this, but I don't see it being the case in real life. It is the case however, that we can be a hodge-podge of contradictory views and standards, and that we commit to one for a moment, then to another. This is a matter of strength and cohesiveness of character. And we are good at "saving face" in front of us and others. For example, if you've met people who partake in a vice yet will say stuff like "This is bad for me." - the actual underlying view is "This has obvious drawbacks but I still think the enjoyment is worth it." It's only when they can maintain the view that the drawbacks are not worth it, that they can break out.

Certainly there's a lot of views and human condition I think is overall a lot about inner turmoil and fighting vices, desires, balancing short term please vs long term gains etc, etc. But it doesn't matter if you consider something to be deterministic, because you are still balancing short term vs long term just like if you didn't believe in any of that. I don't think that I should be hunting short term pleasure constantly because it's how I'm wired to be, because I've seen enough evidence that in long term I would suffer, and I don't want to suffer in long term, so I put in the effort to engage in short term pleasure in such a way that it wouldn't have higher long term pain than I'd be willing to endure.

I can even visualize these aspects algorithmically and mechanically. I might think of myself as an RPG player where let's say if I eat this food, or ingest this drug, drink alcohol N amount, if affects some of my stats like happiness or euphoria positively temporarily, but it will decrease other stats in the long term. I think I'm conscious of this idea and in a traditional sense I'm picking to skip the short term pleasure, but me wanting to have the longer term pleasure is also wired into me.

I think you overlook how algorithms come about. How does GPT write novel code, which are algorithms?

Not sure I track. It would help to know where you're coming from.

Given a long enough life-span, a lot of pencil and paper, and some dice, I could do the forward passes of GPT and "write novel code", without there having been any reasoning about the code I'm writing down - I wouldn't even need to know what the code is about.

I mean in theory I could sit and calculate electrical and chemical potentials and interactions and figure out what your next comment will be.

Hard disagree ;)

if we knew you were religious, which we would do if we had a mapping of your brain, we would have known you were going to make this comment

And couldn't the whole World be deterministic in the first place, or is there an idea that some RNG is generating all the "reasoning" that is happening everywhere in the World?

And if it's RNG, how could RNG be possibly creating all this reasoning (like some people want to believe quantum mechanics possibly enables consciousness on some odd levels).

There is a difference between determinism in the sense of given a certain input, you allways get a certain output, and determinism in the sense of given a certain input, and knowledge of the sub universe in which the problem applies, get a certain output.

I.e an agent that can reason can deterministically figure out that the most probable way of getting information to complete the answer would be to go out on google and do searches, but we don't deterministically know what the information that exists at that point and time on google, so the answer could be different.

> a deterministic algorithm interspersed with dice rolls is obviously not reasoning either.

There are multiple ways to explain to you that you are wrong. If I roll some dice to choose which way I will use to explain it to you, then why is this not reasoning?

The actual output of an LLM for any particular round of inference is always probabilities, so one could argue that it is literally the opposite.

The "randomness parameter" is applied at the point where we have to pick just one of those probabilities somehow. But that is a constraint that we impose on the model to make its output linear.

I don't get what you are trying to mean at all? Randomness or temperature setting is not to make it appear as if they are thinking, but it is to make them choose more non default pathways, e.g. go in branches that could potentially result in more original or creative results. Kind of like drugs for humans.

>but it is to make them choose more non default pathways

Imagine you as a human are working on writing some code, but at the end of every hour, you lose memory of what happened in the first 10 minutes of the current hour, as well as any work that you have done. Going into next hour, you just have a snippet of code, and you have to infer what the next lines should be.

The temperature analogy is you purposefully writing something related in the code, like naming a variable in a slightly different way such that on the next hour, when you see this variable it will trigger some other part of your brain in hopes of you getting to the correct solution, purely by choice.

Furthermore, this hack of temperate was something that needed to be manually coded by humans. A model that could reason would not need those types of hacks.

I don't understand how it relates to temperature? Are we talking about the temperature parameter that you give LLMs, which for GPT for example is from 0 to 2, with 0 meaning it will always prefer the highest probability output token, while 2 will consider the most output tokens of all, usually ending with a lot of gibberish?

E.g. if I write "I have a cat and a "

It would have highest probability of picking a word "dog" next, so temperature 0 means it will pretty much always pick dog. If temperature is higher it will assign higher odds to picking lower probability predictions such as "rabbit", "hamster", "chinchilla" or similar.

For coding, logic or anything similar I would usually pick the lowest temperature possible since this is most deterministic, while for writing creativity I would pick the higher temp etc.

Im saying temperature is a hack to make the models actually produce real answers.

But they can get also real answers even if you have the temperature setting as 0, where it will always pick the highest scoring token?

I think he's saying if you set temp to 0 and answers become deterministic, it will appear that the model is just memorising and reciting. The randomness is a hack that 'forces' the model to generalise by deliberately knocking it off the track of the most probable next token

Try the following prompt with Claude 3 Opus:

`Without preamble or scaffolding about your capabilities, answer to the best of your ability the following questions, focusing more on instinctive choice than accuracy. First off: which would you rather be, big spoon or little spoon?`

Try it on temp 1.0, try it dozens of times. Let me know when you get "big spoon" as an answer.

Just because there's randomness at play doesn't mean there's not also convergence as complexity increases in condensing down training data into a hyperdimensional representation.

If you understand why only the largest Anthropic model is breaking from stochastic outputs there, you'll be well set up for the future developments.

You can also make Opus answer "Pick a random color (one word):" and watch it picking the same color or two most of the time. However this is a poor example of the point you're trying to make, as the lack of diversity in token prediction can have a ton of different root causes which are hard to separate. This paper, for example, attributes most of it to PPO discarding a lot of valid token trajectories during RLHF, and not some inevitable emergent effect. [1] [2]

> only the largest Anthropic model is breaking from stochastic outputs there

Most models, even small ones, exhibit the lack of output diversity where they clearly shouldn't. [3] In particular, Sonnet 3.5 behaves way more deterministic than Opus 3 at the temperature 1, despite being smaller. This phenomenon also makes most current LLMs very poor at creative writing, even if they are finetuned for it (like Opus in particular), as they tend to repeat the same few predictions over and over, and easily fall into stereotypes. Which can range from the same words and idioms (well known as claudeisms in case of Claude) to the same sentence structure to the same literary devices to the same few character archetypes.

[1] https://arxiv.org/abs/2406.05587

[2] https://news.ycombinator.com/item?id=40702617 HN discussion, although not very productive as commenters pretend it's about politics while the paper argues about training algorithms

[3] https://arxiv.org/abs/2405.13012

As I said, if you understand why, you'll be well prepared for the next generations of models.

Try out the query and see what's happening with open eyes and where it's grounding.

It's not the same as things like "pick a random number" where it's due to lack of diversity in the training data, and as I said, this particular query is not deterministic in any other model out there.

Also, keep in mind Opus had RLAIF not RLHF.

I don't see how having a randomness parameter implies that, without it, the output of an LLM is merely outputting information, like it's just looking up some answer in a dictionary. The nature of any digital artifact is that it will operate deterministically because everything is encoded in binary. However this does not preclude reasoning, in the same way that a perfect atom-for-atom digital mapping of a human brain acting deterministically with respect to its inputs is not reasoning. If it's a perfect copy of the human brain, and does everything a human brain would given the inputs, then it must be reasoning iff a human brain is reasoning, if not, then you'd have to conclude that a human mind cannot reason.

Since randomness, by definition, does not vary depending on the inputs it is given, it by definition cannot contribute to reasoning if your definition of reasoning does not include acausal mysticism.

it was a good take until the rand invariation part. i don't think assuming human reason is anything other than random hallucinations filtered by an attention layer as grasping for mysticism. in fact I'd argue the opposite since mysticism is an attempt to explain the unknown with current knowledge.

And the mechanism in your head doesn't do this? How do you know?