I don't know what the exact definition of "thinking" is. But if a definition of thinking rejects the possibility of that current LLMs think, I'd consider that definition useless.

Generally thinking has been used to describe the process human follow in their brains when problem solving.

If the Palms do not follow that process, they are not thinking.

That doesn't mean they cannot solve problems using other mechanisms, they do, and we understand those mechanisms much better than we do human thinking.

The term “thinking” is rather ill-defined, too bound to how we perceive our own wakeful thinking.

When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.

I would agree that LLMs reason (well, the reasoning models). But “thinking”? I don’t know. There is something missing.

“Let me think about this.” “I have to think on it.”

My brain regulates all sorts of processes unconsciously, like breathing, for example. I don’t treat those as “thinking,” so I don’t know why other unconscious brain activity would be either.

"Thinking" to me is very much NOT just conscious reasoning. So much of what I think is not done consciously.

Indeed "let me think about it" is often simply giving my brain time to "sit on it", for another expression - only after which will I have enough mind time on the various alternatives for a worthwhile conscious decision.

Updates your models for the next morning, which is why the answer is there when it wasn’t before.

Yes I would.

The current crop of models do not "sleep" in any way. The associated limitations on long term task adaptation are obvious barriers to their general utility.

> When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.

One of the key functions of REM sleep seems to be the ability to generalize concepts and make connections between "distant" ideas in latent space [1].

I would argue that the current crop of LLMs are overfit on recall ability, particularly on their training corpus. The inherent trade-off is that they are underfit on "conceptual" intelligence. The ability to make connections between these ideas.

As a result, you often get "thinking shaped objects", to paraphrase Janelle Shane [2]. It does feel like the primordial ooze of intelligence, but it is clear we still have several transformer-shaped breakthroughs before actual (human comparable) intelligence.

1. https://en.wikipedia.org/wiki/Why_We_Sleep 2. https://www.aiweirdness.com/

The continuity is currently an illusion.

Much like speaking to a less experienced colleague, no?

They say things that contain the right ideas, but arrange it unconvincingly. Still useful to have though.

Just now in an debugging session with claude code: * let me read this file... * let me read this file... * I think there's a caching issue with the model after dropping the module. Let me check if there's a save or reload needed after DROP MODULE. First, let me verify something: * creates a bash/javascript script to verify its assumption * runs the script (after review and approval) * Aha! I found the problem! Look at the output

Whereas a gorilla who learns sign language to communicate and use that communication to achieve aims which have direct correlation with its sense of self - that's thought in the Cogito, Ergo Sum sense of the word.

Thought as commonly concieved by the layman is a sort of isolated phenomenon that is mechanical in nature and can be judged by its outward effects; whereas in the philosophical tradition defining thought is known to be one of the hard questions for its mysterious qualia of being interconnected with will and being as described above.

Guess I gave you the long answer. (though, really, it could be much longer than this.) The Turing Test touches on this distinction between the appearance of thought and actual thought.

The question goes all the way down to metaphysics; some (such as myself) would say that one must be able to define awareness (what some call consciousness - though I think that term is too loaded) before you can define thought. In fact that is at the heart of the western philosophical tradition; and the jury consensus remains elusive after all these thousands of years.

Even just within humans - many of the least intellectually capable humans seem to have a richer supply of the traits associated with awareness/being than some of the allegedly highest-functioning.

On average you're far more likely to get a sincere hug from someone with Down's syndrome than from a multi-millionaire.

But I'm more interested in this when it comes to the animal kingdom, because while ChatGPT is certainly more useful than my cat, I'm also pretty certain that it's a lot less aware. Meaningful awareness - feelings - seems to be an evolutionary adaptation possessed by k-strategy reproducing vertebrates. Having a small number of kids and being biologically wired to care for them has huge implications for your motivation as an animal, and it's reasonable to think that a lot of our higher emotions are built on hardware originally evolved for that purpose.

(Albeit the evolutionary origins of that are somewhat murky - to what extent mammals/birds reuse capabilities that were developed by a much earlier common ancestor, or whether it's entirely parallel evolution, isn't known afaik - but birds seem to exhibit a similar set of emotional states to mammals, that much is true).

Most people would accept that being able to work out 686799 x 849367 is a form of thinking, albeit an extremely limited one.

First flight simulators, then chess computers, then go computers, then LLMs are the same principle extended to much higher levels of applicability and complexity.

Thinking in itself doesn't require mysterious qualia. It doesn't require self-awareness. It only requires a successful mapping between an input domain and an output domain. And it can be extended with meta-thinking where a process can make decisions and explore possible solutions in a bounded space - starting with if statements, ending (currently) with agentic feedback loops.

Sentience and self-awareness are completely different problems.

In fact it's likely with LLMs that we have off-loaded some of our cognitive techniques to external hardware. With writing, we off-loaded memory, with computing we off-loaded basic algorithmic operations, and now with LLMs we have off-loaded some basic elements of synthetic exploratory intelligence.

These machines are clearly useful, but so far the only reason they're useful is because they do the symbol crunching, we supply the meaning.

From that point of view, nothing has changed. A calculator doesn't know the meaning of addition, an LLM doesn't need to know the meaning of "You're perfectly right." As long as they juggle symbols in ways we can bring meaning to - the core definition of machine thinking - they're still "thinking machines."

It's possible - I suspect likely - they're only three steps away from mimicking sentience. What's needed is a long-term memory, dynamic training so the model is constantly updated and self-corrected in real time, and inputs from a wide range of physical sensors.

At some point fairly soon robotics and LLMs will converge, and then things will get interesting.

Whether or not they'll have human-like qualia will remain an unknowable problem. They'll behave and "reason" as if they do, and we'll have to decide how to handle that. (Although more likely they'll decide that for us.)

That's not of the same nature as reasoning your way to an answer.

But the post we are responding to is directly posing the philosophical question of whether the activity of LLM agents constitutes thought.

1. It has subjective experience (unprovable but assumed)

2. It possesses free will (compatibilist definitions need not apply)

3. It has attended at least one philosophy seminar"[1]

[1]https://ai.vixra.org/pdf/2506.0065v1.pdf

Something doesn't need to learn to think. I think all the time without learning.

There's also an argument for machines already starting to crack learning with literal reinforcement training and feedback loops.

Your language game was when you said the 'cognition ends...', as cognition is just a synonym for thinking. "The thinking ends when the inference cycle ends. It's not thinking'" becomes a clear contradiction.

As for "the process by which it does that is wholly unrelated", buddy it's modelled on human neuron behaviour. That's how we've had this generative AI breakthrough. We've replicated human mental cognition as closely as we can with current technology and the output bears striking resemblance to our own generative capabilities (thoughts).

Happy to admit it's not identical, but it's damn well inside the definition of thinking, may also cover learning. It may be better to take a second look at human thinking and wonder if it's as cryptic and deep as we thought ten, twenty years ago.

People said the same thing about ELIZA

> Consciousness or self awareness is of course a different question,

Then how do you define thinking if not a process that requires consciousness?

LLMs are little more than RNGs. They are the tea leaves and you read whatever you want into them.

Generalizing your experience to everyone else's betrays a lack of imagination.

Any time I try to get a novel insight, it flails wildly, and nothing of value comes out. And yes, I am prompting incrementally and building up slowly.

One guy is generalizing from "they don't work for me" to "they don't work for anyone."

The other one is saying "they do work for me, therefore they do work for some people."

Note that the second of these is a logically valid generalization. Note also that it agrees with folks such as Tim Gowers, who work on novel and hard problems.

One is saying "I've seen an LLM spectacularly fail at basic reasoning enough times to know that LLMs don't have a general ability to think" (but they can sometimes reproduce the appearance of doing so).

The other is trying to generalize "I've seen LLMs produce convincing thought processes therefore LLMs have the general ability to think" (and not just occasionally reproduce the appearance of doing so).

And indeed, only one of these is a valid generalization.

> Having seen LLMs so many times produce incoherent, nonsensical and invalid chains of reasoning... LLMs are little more than RNGs. They are the tea leaves and you read whatever you want into them.

Of course LLMs are capable of generating solutions that aren't in their training data sets but they don't arrive at those solutions through any sort of rigorous reasoning. This means that while their solutions can be impressive at times they're not reliable, they go down wrong paths that they can never get out of and they become less reliable the more autonomy they're given.

I don’t think there’s any point in comparing to human intelligence when assessing machine intelligence, there’s zero reason to think it would have similar qualities. It’s quite clear for the foreseeable future it will be far below human intelligence in many areas, while already exceeding humans in some areas that we regard as signs of intelligence.

Even when mathematicians do in fact do rigorous reasoning, they use years to "train" first, to get experiences to pattern match from.

So far, all I have gotten is data that is outside the knowledge cutoff (this is by far the most common) and technicality wrong information (Hawsmer House instead of Hosmer House) kind of fails.

I thought maybe I hit on something with the recent BBC study about not trusting LLM output, but they used 2nd shelf/old mid-tier models to do their tests. Top LLMs correctly answered their test prompts.

I'm still holding out for one of those totally off the rails Google AI overviews hallucinations showing up in a top shelf model.

I can't say I remember a single coworker that would fit this description though many were frustrating to deal with for other reasons, of course.

I feel like so little is TRUELY novel. Almost everything is built on older concepts and to some degree expertise can be applied or repurposed.

Godot for example in ChatGPT.

It may no longer still be the case, but the documentation for GoDot was lacking and often samples written by others didn't have a version number associated with it. So samples it would suggest would never work, and even when you told it the version number it failed to generate workable code.

The other stuff I've noticed is custom systems. One I work with is a variation of Java, but LLMs were treating it as javascript. I had to create a LoRA just to get the model from not trying to write javascript answer. Even then it could never work, because it had never been trained on real world examples.

Example prompt (paraphrasing and dumbed down, but not a ton): Some users across the country can't get to some fileshares. I know networking, but I'm not on the networking team so I don't have full access to switch, router, and firewall logs/configurations. It looks kind of random, but there must be a root cause, let's find it.

I can't use Python(security team says so) and I don't have access to a Linux box that's joined to the domain and has access the shares.

We are on a Windows domain controller. Write me a PowerShell 5.1 compatible script to be run remotely on devices. Use AD Sites and Services to find groups of random workstations and users at each office and tries to connect to all shares at each other site. Show me progress in the terminal and output an Excel file and Dot file that clearly illustrates successful and failed connections.

---

And it works. Ok, I can see the issue is from certain sites that use x AND y VPN ipsec tunnels to get to particular cloud resources. I give this info to networking and they fix it right away. Problem resolved in less than an hour.

First of all, a couple years ago, I wouldn't have been able to justify writing something like this while an outage is occuring. Could I do it myself? Sure, but I'm going to have to look up the specifics of syntax and certain commands and modules. I don't write PowerShell for a living or fun, but I do need to use it. I am familiar and know how to write it. But I sure as fuck couldn't sit down and spend an hour or two screwing around working on building a goddamn Dot file generator. Yes, years ago I had a whole pile of little utility modules I could use. But that's a far cry from what I can do now to fit the exact situation < 15 minutes while I do other things like pick up the phone, message coworkers, etc.

Secondly, rather than building little custom tools to hook together as I need, I can just ask for the whole thing. I don't need to save any of that stuff anymore and re-figure out what CheckADFSConns(v2).PS1 that I wrote 8 months ago does and how to use it. "Oh, that's not the one, what the did I name that? Where did I put it?"

I work in an environment that is decades old, the company is over 100 years old, I didn't build any of it myself, is not a tech company, and has tons of tech debt and weird shit. AI is insanely useful. For any given problem, there are dozens of different rabbit holes I could go down because of decades of complete system overhaul changes. Today, I can toss a variety of logs at AI and if nothing else, get a sense of direction of why a handful of PCs are rejecting some web certificates. (Combination of a new security policy and their times mismatching the domain controller, because it was new, and NTP wasn't configured properly. I wasn't even looking for timestamps, but it noticed event offsets and pointed it out).

I feel like this community isn't very familiar with what that's like. We aren't all working on self driving cars or whatever seems hard at a brand new company with new everything and no budget. Some of us need to keep the systems running that help people to make actual things. These environments are far from pristine and are held together by underpaid and underappreciated normies through sheer willpower.

Is this kind of work breaking technical frontiers? No. But it's complicated, difficult, and unpredictable. Is it novel? The problems are, sometimes.

Generalizing your experience to everyone else's betrays your lack of self-awareness, sir.

A secretary who works for an inventor is still thinking.

[1] https://mathstodon.xyz/@tao/115420236285085121 [2] https://xcancel.com/wtgowers/status/1984340182351634571

[1] That does not mean that they can never produce texts which describes a valid reasoning process, it means that they can't do so reliably. Sometimes their output can be genius and other times you're left questioning if they even have the reasoning skills of a 1st grader.

Humans sometimes make mistakes in reasoning, too; sometimes they come up with conclusions that leave me completely bewildered (like somehow reasoning that the Earth is flat).

I think we can all agree that humans are significantly better and more consistently good at reasoning than even the best LLM models, but the argument that LLMs cannot reliably reason doesn't seem to match the evidence.

Yeah, I know, yet here we are and it saves me boatloads of time.

Thing is, you wouldn't need to learn to use the tools if the tool was able to think. A thinking entity is able to adapt to other parties who lack learnings. This confirms that LLMs are little more than fancy RNGs.

> I use it daily and get meaningful, quality results.

That's what the tea leaf readers say too, funnily enough.

(And I would hazard a guess that they are a primarily learning chess from the many games that are posted, as opposed to working things out from the rules. Indeed, if you make up a game and tell chatGPT the rules, it tends to be even worse at following them, let alone figuring out optimal play. But again, it will do so significantly better than random chance, so it's doing something with the information you give it, even if it's not doing so very well. I think it's reasonable to call this thinking, or reasoning, but this mostly becomes an argument of semantics. either way they do it significantly better than random chance but still not tremendously well. If your expectation is that they cannot work with anything novel then you're going to be continually surprised, but if your expectation is that they're as good as a human that has 'learned' from all the material its been given, especially material that's in-context and not in the training data, then you're also going to be disappointed.)

What we really mean in both cases is "computing," no?

But if you define thinking as the mechanism and process we mentally undergo and follow mentally... I don't think we have any clue if that's the same. Do we also just vector-map attention tokens and predict the next with a softmax? I doubt, and I don't think we have any proof that we do.

I don’t know what LLMs are doing, but only a little experimentation with getting it to describe its own process shows that it CAN’T describe its own process.

You can call what a TI calculator does “thinking” if you want. But what people are interested in is human-like thinking. We have no reason to believe that the “thinking” of LLMs is human-like.

It's funny that you think people don't also do that. We even have a term, "post hoc rationalization", and theories of mind suggest that our conscious control is a complete illusion, we just construct stories for decisions our subconscious has already made.

But; they don't learn. You can add stuff to their context, but they never get better at doing things, don't really understand feedback. An LLM given a task a thousand times will produce similar results a thousand times; it won't get better at it, or even quicker at it.

And you can't ask them to explain their thinking. If they are thinking, and I agree they might, they don't have any awareness of that process (like we do).

I think if we crack both of those then we'd be a lot closer to something I can recognise as actually thinking.

If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?

Do amnesiacs who are incapable of laying down long-term memories not think?

I personally believe that memory formation and learning are one of the biggest cruces for general intelligence, but I can easily imagine thinking occurring without memory. (Yes, this is potentially ethically very worrying.)

Perhaps this is already known, but I would think there is a high chance that our brains require "write access" to function. That is, the very process of neural activity inherently makes modifications to the underlying structure.

There are multiple learning mechanisms that happen on different time-frames, eg neural plasticity, hippocampus are both longer-term processes for memory consolidation. Whereas the content of “RAM” might be better modeled as a set of fast dynamic weights representing ions and neurotransmitter concentrations.

My hunch is that you could model the latter fast weights in volatile memory, but I wouldn’t count these as “modifying the structure”.

Do you have any particular systems in mind where you have reason to believe that permanent modification is required for brains to function?

it wouldn't work probably, brains constantly alter themselves by forming new connections. Learning is inseparable from our intelligence.

So I guess I’d just clarify “read only” to be a little more specific - I think you could run multiple experiments where you vary the line of what’s modeled in volatile memory at runtime, and what’s immutable. I buy that you need to model STP for thought, but also suspect at this timescale you can keep everything slower immutable and keep the second-scale processes like thought working.

My original point still stands - your subjective experience in this scenario would be thought without long-term memory.

1: https://www.frontiersin.org/journals/computational-neuroscie...

I was using Claude Code today and it was absolutely capable of taking feedback to change behavior?

Start a new session with a blank context and it will not have learned anything.

Why would I use a blank context with Claude Code? I have it dump the stuff I want to save in CLAUDE.md and it reloads it in future sessions.

If one could write a quadrillion-line python script of nothing but if/elif/else statements nested 1 million blocks deep that seemingly parsed your questions and produced seemingly coherent, sensible, valid "chains of reasoning"... would that software be thinking?

And if you don't like the answer, how is the LLM fundamentally different from the software I describe?

>Knee jerk dismissing the evidence in front of your eyes because

There is no evidence here. On the very remote possibility that LLMs are at some level doing what humans are doing, I would then feel really pathetic that humans are as non-sapient as the LLMs. The same way that there is a hole in your vision because of a defective retina, there is a hole in your cognition that blinds you to how cognition works. Because of this, you and all the other humans are stumbling around in the dark, trying to invent intelligence by accident, rather than just introspecting and writing it out from scratch. While our species might someday eventually brute force AGI, it would be many thousands of years before we get there.

> to move through water by moving the body or parts of the body.

Anthropomorphizing cars doesn't make them humans either.

Is this an intellectual exercise for you or have you ever in your life heard someone say something like “the submarine swam through the water”? It’s so ridiculous I would be shocked to see it outside of a story intended for children or an obvious nonnative speaker of English.

That's a choice to limit the meaning of the term to the rigid/immobile parts of the external boundary of an object. It's not obviously the correct choice. Presumably you don't take issue with people saying planes fly. The issue of submarines swimming seems analogous.

>Is this an intellectual exercise for you or have you ever in your life heard someone say something like “the submarine swam through the water”?

I don't think I've ever had a discussion about submarines with anyone, outside of the OceanGate disaster. But this whole approach to the issue seems misguided. With terms like this we should ask what the purpose behind the term is, i.e. it's intension (the concept), not the incidental extension of the term (the collection of things it applies to at some point in time). When we refer to something swimming, we mean that it is moving through water under its own power. The reference to "body" is incidental.

The point that "swim" is not very consistent with "fly" is true enough but not really helpful. It doesn't change the commonly understood meaning of "swim" to include spinning a propeller just because "fly" doesn't imply anything about the particular means used to achieve flight.

I meant a collective choice. Words evolve because someone decides to expand their scope and others find it useful. The question here shouldn't be what do other people mean by a term but whether the expanded scope is clarifying or confusing.

The question of whether submarines swim is a trivial verbal dispute, nothing of substance turns on its resolution. But we shouldn't dismiss the question of whether computers think by reference to the triviality of submarines swimming. The question we need to ask is what work does the concept of thinking do and whether that work is or can be applied to computers. This is extremely relevant in the present day.

When we say someone thinks, we are attributing some space of behavioral capacities to that person. That is, a certain competence and robustness with managing complexity to achieve a goal. Such attributions may warrant a level of responsibility and autonomy that would not be warranted without it. A system that thinks can be trusted in a much wider range of circumstances than one that doesn't. That this level of competence has historically been exclusive to humans should not preclude this consideration. When some future AI does reach this level of competence, we should use terms like thinking and understanding as indicating this competence.

To make a long story short, the distinction is that typical programs don't operate on the semantic features of program state, just on the syntactical features. We assign a correspondence with the syntactical program features and their transformations to the real-world semantic features and logical transformations on them. The execution of the program then tells us the outcomes of the logical transformations applied to the relevant semantic features. We get meaning out of programs because of this analogical correspondence.

LLMs are a different computing paradigm because they now operate on semantic features of program state. Embedding vectors assign semantic features to syntactical structures of the vector space. Operations on these syntactical structures allow the program to engage with semantic features of program state directly. LLMs engage with the meaning of program state and alter its execution accordingly. It's still deterministic, but its a fundamentally more rich programming paradigm, one that bridges the gap between program state as syntactical structures and the meaning they represent. This is why I am optimistic that current or future LLMs should be considered properly thinking machines.

>But I think there’s still the question if this process is more similar to thought or a Markov chain.

It's definitely far from a Markov chain. Markov chains treat the past context as a single unit, an N-tuple that has no internal structure. The next state is indexed by this tuple. LLMs leverage the internal structure of the context which allows a large class of generalization that Markov chains necessarily miss.

How much resemblance does the information in the conditionals need to have with the actual input, or can they immediately be transformed to a completely separate 'language' which simply uses the string object as its conduit? Can the 10^15 lines of code be generated with an external algorithm, or is it assumed that I'd written it by hand given an infinitely long lifespan?

You see how doesn’t make sense what you saying?

The proper category error in the context of the discussion would be to say the photocopier is drawing a picture.

It doesn't matter how well or not the photocopier recreates an image. To say the photocopier is drawing a picture is just nonsense and has no meaning.

The same category error as to say the LLM is "thinking".

Of course, the category error could be well exploited for marketing purposes if you are in the business of selling photocopiers or language models.

But "self-awareness," as in the ability to explicitly describe implicit, inner cognitive processes? That has some very strong evidence for it: https://www.anthropic.com/research/introspection

Try to ask something no one ever came up with a solution so far.

https://chatgpt.com/share/6909b7d2-20bc-8011-95b6-8a36f332ac...

YOU came up with it dawg

Consciousness and self-awareness are a distraction.

Consider that for the exact same prompt and instructions, small variations in wording or spelling change its output significantly. If it thought and reasoned, it would know to ignore those and focus on the variables and input at hand to produce deterministic and consistent output. However, it only computes in terms of tokens, so when a token changes, the probability of what a correct response would look like changes, so it adapts.

It does not actually add 1+2 when you ask it to do so. it does not distinguish 1 from 2 as discrete units in an addition operation. but it uses descriptions of the operation to approximate a result. and even for something so simple, some phrasings and wordings might not result in 3 as a result.

Isn't that still "not thinking"?

It's still just information retrieval. You're just dividing it into internal information (the compressed representation of the training data) and external information (web search, API calls to systems, etc). There is a lot of hidden knowledge embedded in language and LLMs do a good job of teasing it out that resembles reasoning/thinking but really isn't.

Given the fact that "thinking" still hasn't been defined rigourously, I don't understand how people are so confident in claiming they don't think.

[1] https://en.wikipedia.org/wiki/I_know_it_when_I_see_it

* Humans that make mistakes are still considered to be reasoning.

* Deterministic algorithms have limitations, like Goedel incompleteness, which humans seem able to overcome, so presumably, we expect reasoning to also be able to overcome such challenges.

2) Yes, to reason does imply to be infallible. The deterministic algorithms we follow are usually flawed.

3) I can't speak much to that, but I speculate that if "AI" can do reasoning, it would be a much more complex construct that uses LLMs (among other tools) as tools and variables like we do.

LLM's be like "The dumb humans can't even see the dots"[1]

[1]https://compote.slate.com/images/bdbaa19e-2c8f-435e-95ca-a93...

How about non-determinism (i.e. hallucinations)? Ask a human ANY question 3 times and they will give you the same answer, every time, unless you prod them or rephrase the question. Sure the answer might be wrong 3 times, but at least you have consistency. Then again, maybe that's a disadvantage for humans!

Because it's hard to imagine the sheer volume of data it's been trained on.

All marketing.

Auto completion just means predicting the next thing in a sequence. This does not preclude reasoning.

> I don't get why you would say that.

Because I see them solve real debugging problems talking through the impact of code changes or lines all the time to find non-obvious errors with ordering and timing conditions on code they’ve never seen before.

It does not imply it either. to claim reasoning you need evidence. it needs to reliably NOT hallucinate results for simple conversations for example (if it has basic reasoning).

> Because I see them solve real debugging problems talking through the impact of code changes or lines all the time to find non-obvious errors with ordering and timing conditions on code they’ve never seen before.

Programming languages and how programs work are extensively and abundantly documented, solutions to problems and how to approach them,etc.. have been documented on the internet extensively. It takes all of that data and it completes the right text by taking the most correct path way based on your input. it does not actually take your code and debug it. it is the sheer volume of data it uses and the computational resources behind it that are making it hard to wrap your head around the difference between guessing and understanding. You too can look at enough stack overflow and (poorly) guess answers for questions without understanding anything about the topic and if you guess enough you'll get some right. LLMs are just optimized to get the amount of correct responses to be high.

Right, it's irrelevant to the question of whether they can reason.

> to claim reasoning you need evidence

Frankly I have no idea what most people are talking about when they use the term and say these models can't do it. It seems to be a similarly hand-wavey exercise as when people talk about thinking or understanding.

> it needs to reliably NOT hallucinate results for simple conversations for example (if it has basic reasoning).

That's not something I commonly see in frontier models.

Again this doesn't seem related to reasoning. What we call hallucinations would be seen in something that could reason but had a fallible memory. I remember things incorrectly and I can reason.

> it does not actually take your code and debug it

It talks through the code (which it has not seen) and process step by step, can choose to add logging, run it, go through the logs, change what it thinks is happening and repeat. It can do this until it explains what is happening, creates test cases to show the problem and what triggers it, fixes it and shows the tests pass.

If that's not debugging the code I really don't know what to call it.

When was the last time you sat down and solved an original problem for which you had no prior context to "complete" an approximated solution with? When has that ever happened in human history? All the great invention-moment stories that come to mind seem to have exactly that going on in the background: Prior context being auto-completed in an Eureka! moment.

Let's say your visual system auto-completes some pattern and detects a snake while you're walking, that part is auto-completion. You will probably react by freezing or panicing, that part is not auto-compleition, it is a deterministic algorithm. But then you process the detected object, auto-compleiting again to identify it as just a long cucumber. But again, the classification part is auto-completion. What will you do next? "Hmm, free cucumber, i can cook with it for a meal" and you pick it up. auto-completion is all over that simple decision, but you're using results of auto-completion to derive association (food), check your hunger level (not auto-completion), determine that the food is desirable and safe to eat (some auto-compleition), evalute what other options you have for food (evaluate auto-complete outputs), and then instruct your nervous system to pick it up.

We use auto-compleition all the time as an input, we don't reason using auto-compleition in other words. You can argue that if all your input is from auto-completion (it isn't) then it makes no difference. But we have deterministic reasoning logical systems that evaluate auto-completion outputs. if your cucumber detection identified it as rotten cucumber, then decision that it is not safe to eat is not done by auto-completion but a reasoning logic that is using auto-completion output. You can approximate the level of rot but once you recognize it as rotten, you make decision based on that information. You're not approximating a decision, you're evaluating a simple logic of: if(safe()){eat();}.

Now amp that up to solving very complex problems. what experiments will you run, what theories will you develop, what R&D is required for a solution,etc.. these too are not auto-completions. an LLM would auto complete these and might arrive at the same conclusion most of the time. but our brains are following algorithms we developed and learned over time where an LLM is just expanding on auto-completion but with a lot more data. In contrast, our brains are not trained on all the knowledge available on the public internet, we retain a tiny miniscule of that. we can arrive at similar conclusions as the LLM because we are reasoning and following algorithms matured and perfected over time.

The big take away should be that, as powerful as LLMs are now, if they could reason like we do, they'd dominate us and become unstoppable. Because their auto-completion is many magnitudes better than ours, if they can write new and original code based on an understanding of problem solving algorithms, that would be gen ai.

We can not just add 1 + 1 but prove that the addition operation is correct mathematically. and understand that when you add to a set one more object, the addition operation always increments. We don't approximate that, we always, every single time , increment because we are following an algorithm instead of choosing the most likely correct answer.

Any thinking that happens with words is fundamentally no different to what LLMs do, and everything you say applies to human lexical reasoning.

One plus one equals two. Do you have a concept of one-ness, or two-ness, beyond symbolic assignment? Does a cashier possess number theory? Or are these just syntactical stochastic rules?

I think the problem here is the definition of “thinking”.

You can point to non-verbal models, like vision models - but again, these aren’t hugely different from how we parse non-lexical information.

This is such a wildly simplified and naive claim. "Thinking with words" happens inside a brain, not inside a silicon circuit with artificial neurons bolted in place. The brain is plastic, it is never the same from one moment to the next. It does not require structured input, labeled data, or predefined objectives in order to learn "thinking with words." The brain performs continuous, unsupervised learning from chaotic sensory input to do what it does. Its complexity and efficiency are orders of magnitude beyond that of LLM inference. Current models barely scratch the surface of that level of complexity and efficiency.

> Do you have a concept of one-ness, or two-ness, beyond symbolic assignment?

Obviously we do. The human brain's idea of "one-ness" or "two-ness" is grounded in sensory experience — seeing one object, then two, and abstracting the difference. That grounding gives meaning to the symbol, something LLMs don't have.

Models see when photons hit camera sensors, you see when photons hit your retina. Both of them are some kind of sight.

When photons hit the retina, the same kind of photochemical transduction happens — but the signal does not stop at measurement. It flows through a living system that integrates it with memory, emotion, context, and self-awareness. The brain does not just register and store the light, it constructs an experience of seeing, a subjective phenomenon — qualia.

Once models start continuously learning from visual subjective experience, hit me up – and I'll tell you the models "see objects" now. Until direct raw photovoltaic information stream about the world around them without any labelling can actually make model to learn anything, they are not even close to "seeing".

We are no more in touch with physical reality than an LLM, unless you are in the habit of pressing your brain against things. Everything is interpreted through a symbolic map.

See also: feral children.

"dog": [object of class Canine]

They infer meaning from noisy, ambiguous sensory streams. The labels are not explicit, they are discovered through correlation, context, and feedback.

So although caregivers sometimes point and name things, that is a tiny fraction of linguistic input, and it is inconsistent. Children generalize far beyond that.

Real linguistic input to a child is incomplete, fragmented, error-filled, and dependens on context. It is full of interruptions, mispronunciations, and slang. The brain extracts structure from that chaos. Calling that "structured input" confuses the output - inherent structure of language - with the raw input, noisy speech and gestures.

The brain has drives: social bonding, curiosity, pattern-seeking. But it doesn't have a single optimisation target like "expand." Objectives are not hardcoded loss functions, they are emergent and changing.

You're right that lack of linguistic input prevents full language development, but that is not evidence of supervised learning. It just shows that exposure to any language stream is needed to trigger the innate capacity.

Both complexity and efficiency of the human learning is just on another level. Transformers are child's play compared to that level. They are not going to gain consciousness, and no AGI will happen in the foreseeable future, it is all just marketing crap, and it's becoming more and more obvious as the dust settles.

If you train an animal to type the right keys on a keyboard that generates a hello world program, you didn't just teach them how to code. they just memorized the right keys that lead to their reward. a human programmer understands the components of the code, the intent and expectations behind it, and can reason about how changes would affect outcomes. the animal just knows how the reward can be obtained most reliably.

Neither can humans. We also just brute force "autocompletion" with our learned knowledge and combine it to new parts, which we then add to our learned knowledge to deepen the process. We are just much, much better at this than AI, after some decades of training.

And I'm not saying that AI is fully there yet and has solved "thinking". IMHO it's more "pre-thinking" or proto-intelligence.. The picture is there, but the dots are not merging yet to form the real picture.

> It does not actually add 1+2 when you ask it to do so. it does not distinguish 1 from 2 as discrete units in an addition operation.

Neither can a toddler nor an animal. The level of ability is irrelevant for evaluating its foundation.

Its foundation of rational logical thought that can't process basic math? Even a toddler understands 2 is more than 1.

Look at it this way, two students get 100% on an exam. One learned the probability of which multiple choice options have the likelihood of being most correct based on how the question is worded, they have no understanding of the topics at hand, and they're not performing any sort of topic-specific reasoning. They're just good at guessing the right option. The second student actually understood the topics, reasoned, calculated and that's how they aced the exam.

I recently read about a 3-4 year old that impressed their teacher by reading perfectly a story book like an adult. it turns out, their parent read it to them so much, they can predict based on page turns and timing the exact words that need to be spoken. The child didn't know what an alphabet, word,etc.. was they just got so good at predicting the next sequence.

That's the difference here.

The actual work is in gaining the knowledge and programs, not in accessing and executing them. And how they operate, on which data, variables, objects, worldview or whatever you call it, this might make a difference in quality and building speed, but not for the process in general.

no, LLMs can have that contextualized information. understanding in a reasoning sense means classifying the thing and developing a deterministic algorithm to process it. If you don't have a deterministic algorithm to process it, it isn't understanding. LLMs learn to approximate, we do that too, but then we develop algorithms to process input and generate output using a predefined logical process.

A sorting algorithm is a good example, when you compare that with an LLM sorting a list. they both may have correct outcome, but the sorting algorithm "understood" the logic and will follow that specific logic and have consistent performance.

That's the learning-part I was talking about. Which is mainly supported by humans at the moment, which why I called it proto-intelligence.

> If you don't have a deterministic algorithm to process it, it isn't understanding.

Commercial AIs like ChatGPT do have the ability to call programs and integrate the result in their processing. Those AIs are not really just LLMs. The results are still rough and poor, but the concept is there and growing.

Maybe it's just semantics, but I don't think LLMs even come close to a fruit fly's intelligence. Why can't we recognize and accept them for what they are, really powerful classifiers of data.

> Commercial AIs like ChatGPT do have the ability to call programs and integrate the result in their processing. Those AIs are not really just LLMs. The results are still rough and poor, but the concept is there and growing.

Yeah RAG and all of that, but those programs use deterministic algorithms. Now, if LLMs generated programs they call on as tools, that would be much more like the proto-intelligence you're talking about.

Semantics are boring, but it's important that we're not content or celebrate early by calling it what it isn't.

you know this because you're a cognitive scientist right? or because this is the consensus in the field?

Wouldn't be an A.I. discussion without a bizarre, untrue claim that the human brain works identically.

The reason people give that humans are not auto-complete is "Obviously I am not an autocomplete"

Meanwhile, people are just a black box process that output words into their head, which they then take credit for, and calling it cognition. We have no idea how that black box that serves up a word when I say "Think of a car brand" works.

Which is why if you pick up a neuroscience textbook it's 400 pages of blank white pages, correct?

There are different levels of understanding.

I don't need to know how a TV works to know there aren't little men and women acting out the TV shows when I put them on.

I don't need to know how the brain works in detail to know claims that humans are doing the same things as LLMs to be similarly silly.

Yes you do. It's all computation in the end, and isomorphisms can often be surprising.

I don't think any serious person thinks LLMs work like the human brain.

People claiming this online aren't going around murdering their spouses like you'd delete an old LLama model from your hard drive.

I'm not sure why people keep posting these sorts of claims they can't possibly actually believe if we look at their demonstrable real life behavior.

> People claiming this online aren't going around murdering their spouses like you'd delete an old LLama model from your hard drive.

Not sure what you’re trying to say here.

I'd also say stringing together A.I. buzzwords (input output) to describe humans isn't really an argument so much as what philosophers call a category error.

Input and output are not AI buzzwords, they’re fundamental terms in computation. The argument that human beings are computational has been alive in philosophy since the 1940’s brother…

I recently read a little philosophy on this, to say humans are "computational" you first have to define what you mean by "computational". Depending on the definition you employ, you are either making a wacky claim, a controversial claim, or a noncontroversial claim.

Since you provide no definition, you are saying nothing at all.

The discourse that comes up in AI topics seems more glib language games than actual concern over human nature, computational or otherwise.

>That I wouldn’t treat a human like an LLM is completely irrelevant to the topic.

The fact you wouldn't treat a human like an LLM suggests your constant claim of similarity is not something you actually believe.

Flagrantly, ridiculously untrue. We don't know the precise nuts and bolts regarding the emergence of consciousness and the ability to reason, that's fair, but different structures of the brain have been directly linked to different functions and have been observed in operation on patients being stimulated in various ways with machinery attached to them reading levels of neuro-activity in the brain, and in specific regions. We know which parts handle our visual acuity and sense of hearing, and even cooler, we can watch those same regions light up when we use our "minds eye" to imagine things or engage in self-talk, completely silent speech that nevertheless engages our verbal center, which is also engaged by the act of handwriting and typing.

In short: no, we don't have the WHOLE answer. But to say that we have no idea is categorically ridiculous.

As to the notion of LLMs doing similarly: no. They are trained on millions of texts of various sources of humans doing thinking aloud, and that is what you're seeing: a probabilistic read of millions if not billions of documents, written by humans, selected by the machine to "minimize error." And crucially, it can't minimize it 100%. Whatever philosophical points you'd like to raise about intelligence or thinking, I don't think we would ever be willing to call someone intelligent if they just made something up in response to your query, because they think you really want it to be real, even when it isn't. Which points to the overall charade: it wants to LOOK intelligent, while not BEING intelligent, because that's what the engineers who built it wanted it to do.

I have to disagree here. When you are tasked with dividing 2 big numbers you most certainly don't "autocomplete" (with the sense of finding the most probable next tokens, which is what an LLM does), rather you go through set of steps you have learned. Same as with the strawberry example, you're not throwing guesses until something statistically likely to be correct sticks.

> you're not throwing guesses until something statistically likely to be correct sticks.

What do you mean? That's exactly how many humans are operating with unknown situations/topics. If you don't know, just throw punches and look what works. Of course, not everyone is ignorant enough to be vocal about this in every situation.

Why do you think that this is the part that requires intelligence, rather than a more intuitive process? Because they have had machines that can do this mechanically for well over a hundred years.

There is a whole category of critiques of AI of this type: "Humans don't think this way, they mechanically follow an algorithm/logic", but computers have been able to mechanically follow algorithms and perform logic from the beginning! That isn't thinking!

But would a combination of the 2 then be close to what we define as thinking though?

You did not plan the entire thing, every word, ahead of time.

LLMs do the same thing, so... how is your intelligence any different?

Not that it matters, there's evidence that while LLMs output one word at a time, they've got forward-planning going on, having an idea of the end of a sentence before they get there.

Furthermore, when you ask an LLM to count how many r's are in the word strawberry, it will give you a random answer, "think" about it, and give you another random answer. And I guarantee you out of 3 attempts, including reasoning, it will flip-flop between right and wrong, but unlike a human, it will be random, because, unlike humans who, when asked "how many r's are in the word strawberry" will not be able to tell you the correct answer every. fucking. time.

edit: formatting

> How does Claude write rhyming poetry? Consider this ditty:

> He saw a carrot and had to grab it,

> His hunger was like a starving rabbit

> To write the second line, the model had to satisfy two constraints at the same time: the need to rhyme (with "grab it"), and the need to make sense (why did he grab the carrot?). Our guess was that Claude was writing word-by-word without much forethought until the end of the line, where it would make sure to pick a word that rhymes. We therefore expected to see a circuit with parallel paths, one for ensuring the final word made sense, and one for ensuring it rhymes.

> Instead, we found that Claude plans ahead. Before starting the second line, it began "thinking" of potential on-topic words that would rhyme with "grab it". Then, with these plans in mind, it writes a line to end with the planned word.

[https://www.anthropic.com/research/tracing-thoughts-language...]

[0] https://arxiv.org/pdf/2502.16705

I presume they finally got around to training the internal reasoning loop into predicting when it needs to apply answer predictions character-by-character.

—- Transcripts from Claude Sonnet 4.5, initially with extended thinking:

> How many a’s in the phrase “artichokes and aardvarks”? Be sure to check your answer.

> Let me count the letter ‘a’ in “artichokes and aardvarks”: • artichokes: 1 ‘a’ (at the beginning) • and: 1 ‘a’ (at the beginning) • aardvarks: 3 ‘a’s (two at the beginning: “aa”, and one in the middle: “ardvarks”) Total: 1 + 1 + 3 = 5 ‘a’s Let me verify by marking them: artichokes and aardvarks Yes, there are 5 letter ‘a’s in the phrase “artichokes and aardvarks”.

> Are you sure that’s correct?

> Let me recount more carefully: “artichokes and aardvarks” Going through each word letter by letter: 1. artichokes - has 1 ‘a’ (first letter) 2. and - has 1 ‘a’ (first letter) 3. aardvarks - has 3 ‘a’s (first letter, second letter, and one more in the middle) Total: 1 + 1 + 3 = 5 ‘a’s Yes, I’m confident the answer is 5. Is there something specific you’d like me to recheck?

> How many r’s in strawberry?

> Let me count the letter ‘r’ in “strawberry”: s-t-r-a-w-b-e-r-r-y Going through each letter: • s: not an r • t: not an r • r: 1st r • a: not an r • w: not an r • b: not an r • e: not an r • r: 2nd r • r: 3rd r • y: not an r There are 3 r’s in “strawberry”.

<Without extended thinking> > How many u’s in universalulis?

> Looking at “universalulis” letter by letter: u-n-i-v-e-r-s-a-l-u-l-i-s There are 2 u’s in “universalulis”.

OpenAI is working on ChatGPT the application and ecosystem. They have transitioned from model building to software engineering with RL tuning and integration of various services to solve the problems the model can't do on its own. Make it feel smart rather than be smart.

This means that as soon as you find a problem where you step out of the guided experience you get the raw model again which fails when encountering these "gotchas".

Edit - Here's an example where we see a very tuned RL experience in English where a whole load of context is added on how to solve the problem while the Swedish prompt for the same word fails.

https://imgur.com/a/SlD84Ih

The only thing that this stupid test demonstrates is that LLM metacognitive skills are still lacking. Which shouldn't be a surprise to anyone. The only surprising thing is that they have metacognitive skills, despite the base model training doing very little to encourage their development.

[0] https://arxiv.org/pdf/2502.16705

That trick? The LLM has succeeded by spelling the entire word out letter by letter first.

It's much easier for an LLM to perform "tokenized word -> letters -> letter counts" than it is to perform "tokenized word -> letter counts" in one pass. But it doesn't know that! It copies human behavior from human text, and humans never had to deal with tokenizer issues in text!

You can either teach the LLM that explicitly, or just do RLVR on diverse tasks and hope it learns the tricks like this by itself.

I bet you can't give an example such written problem that a human can easily solve but no LLM can.

https://en.wikipedia.org/wiki/Predictive_coding

> If it thought and reasoned, it would know to ignore those and focus on the variables and input at hand to produce deterministic and consistent output

You only do this because you were trained to do this, eg. to see symmetries and translations.

If you took a Claude session into a time machine to 2019 and called it "rent a programmer buddy," how many people would assume it was a human? The only hint that it wasn't a human programmer would be things where it was clearly better: it types things very fast, and seems to know every language.

You can set expectations in the way you would with a real programmer: "I have this script, it runs like this, please fix it so it does so and so". You can do this without being very precise in your explanation (though it helps) and you can make typos, yet it will still work. You can see it literally doing what you would do yourself: running the program, reading the errors, editing the program, and repeating.

People need to keep in mind two things when they compare LLMs to humans: you don't know the internal process of a human either, he is also just telling you that he ran the program, read the errors, and edited. The other thing is the bar for thinking: a four-year old kid who is incapable of any of these things you would not deny as a thinking person.

Depends on the users. Junior devs might be fooled. Senior devs would quickly understand that something is wrong.

Thinking, like intelligence and many other words designating complex things, isn’t a simple topic. The word and concept developed in a world where it referred to human beings, and in a lesser sense, to animals.

To simply disregard that entire conceptual history and say, “well it’s doing a thing that looks like thinking, ergo it’s thinking” is the lazy move. What’s really needed is an analysis of what thinking actually means, as a word. Unfortunately everyone is loathe to argue about definitions, even when that is fundamentally what this is all about.

Until that conceptual clarification happens, you can expect endless messy debates with no real resolution.

“For every complex problem there is an answer that is clear, simple, and wrong.” - H. L. Mencken

They are doing useful stuff, saving time, etc, which can be measured. Thus also the defintion of AGI has largely become: "can produce or surpass the economic output of a human knowledge worker".

But I think this detracts from the more interesting discussion of what they are more essentially. So, while I agree that we should push on getting our terms defined, I think I'd rather work with a hazy definition, than derail so many AI discussion to mere economic output.

If you’re asking big questions like “can a machine think?” Or “is an AI conscious?” without doing the work of clarifying your concepts, then you’re only going to get vague ideas, sci-fi cultural tropes, and a host of other things.

I think the output question is also interesting enough on its own, because we can talk about the pragmatic effects of ChatGPT on writing without falling into this woo trap of thinking ChatGPT is making the human capacity for expression somehow extinct. But this requires one to cut through the hype and reactionary anti-hype, which is not an easy thing to do.

That is how I myself see AI: immensely useful new tools, but in no way some kind of new entity or consciousness, at least without doing the real philosophical work to figure out what that actually means.

IMO the issue is we won't be able to adequately answer this question before we first clearly describe what we mean of conscious thinking applied to ourselves. First we'd need to clearly define our own consciousness and what we mean by our own "conscious thinking" in a much, much clearer way than we currently do.

If we ever reach that point, I think we'd be able to fruitfully apply it to AI, etc., to assess.

Unfortunately we haven't been obstructed from answering this question about ourselves for centuries or millennia, but have failed to do so, so it's unlikely to happen suddenly now. Unless we use AIs to first solve that problem of defining our own consciousness, before applying it back on them. Which would be a deeply problematic order, since nobody would trust a breakthrough in the understanding of consciousness that came from AI, that is then potentially used to put them in the same class and define them as either thinking things or conscious things.

Kind of a shame we didn't get our own consciousness worked out before AI came along. Then again, wasn't for the lack of trying… Philosophy commanded the attention of great thinkers for a long time.

Do you think someone who has only ever studied pre-calc would be able to work through a calculus book if they had sufficient time? how about a multi-variable calc book? How about grad level mathematics?

IMO intelligence and thinking is strictly about this ratio; what can you extrapolate from the smallest amount of information possible, and why? From this perspective, I dont think any of our LLMs are remotely intelligent despite what our tech leaders say.

I would say a good definition has to, minimally, take on the Turing test (even if you disagree, you should say why). Or in current vibe parlance, it does "feel" intelligent to many people--they see intelligence in it. In my book this allows us to call it intelligent, at least loosely.

I have long thought this, but not had as good way to put it as you did.

If you think about geniuses like Einstein and ramanujen, they understood things before they had the mathematical language to express them. LLMs are the opposite; they fail to understand things after untold effort, training data, and training.

So the question is, how intelligent are LLMs when you reduce their training data and training? Since they rapidly devolve into nonsense, the answer must be that they have no internal intelligence

Ever had the experience of helping someone who's chronically doing the wrong thing, to eventually find they had an incorrect assumption, an incorrect reasoning generating deterministic wrong answers? LLMs dont do that; they just lack understanding. They'll hallucinate unrelated things because they dont know what they're talking about - you may have also had this experience with someone :)

This would be the equivalent of removing all senses of a human from birth and expecting them to somehow learn things. They will not. Therefore humans are not intelligent?

> LLMs dont do that; they just lack understanding.

You have no idea what they are doing. Since they are smaller than the dataset, they must have learned an internal algorithm. This algorithm is drawing patterns from somewhere - those are its internal, incorrect assumptions. It does not operate in the same way that a human does, but it seems ridiculous to say that it lacks intelligence because of that.

It sounds like you've reached a conclusion, that LLMs cannot be intelligent because they have said really weird things before, and are trying to justify it in reverse. Sure, it may not have grasped that particular thing. But are you suggesting that you've never met a human that is feigning understanding in a particular topic say some really weird things akin to an LLM? I'm an educator, and I have heard the strangest things that I just cannot comprehend no matter how much I dig. It really feels like shifting goalposts. We need to do better than that.

In split-brain experiments this is exactly how one half of the brain retroactively justifies the action of the other half. Maybe it is the case in LLMs that an overpowered latent feature sets the overall direction of the "thought" and then inference just has to make the best of it.

Simply put, to compare models, you describe both the model and training data using a code (usual reported as number of bits). The trained model that represents the data within the fewest number of bits is the more powerful model.

This paper [2] from ICML 2021 shows a practical approach for attempting to estimate MDL for NLP models applied to text datasets.

[1]: http://www.modelselection.org/mdl/

[2]: https://proceedings.mlr.press/v139/perez21a.html

A crow bending a piece of wire into a hook to retrieve food demonstrates a novel solution extrapolated from minimal, non-instinctive, environmental input. This kind of zero-shot problem-solving aligns better with your definition of intelligence.

And yes, by this definition, LLMs pass with flying colours.

Firstly humans have not been evolving for “billions” of years.

Homo sapiens have been around for maybe 300’000 years, and the “homo” genus has been 2/3 million years. Before that we were chimps etc and that’s 6/7 million years ago.

If you want to look at the entire brain development, ie from mouse like creatures through to apes and then humans that’s 200M years.

If you want to think about generations it’s only 50/75M generations, ie “training loops”.

That’s really not very many.

Also the bigger point is this, for 99.9999% of that time we had no writing, or any kind of complex thinking required.

So our ability to reason about maths, writing, science etc is only in the last 2000-2500 years! Ie only roughly 200 or so generations.

Our brain was not “evolved” to do science, maths etc.

Most of evolution was us running around just killing stuff and eating and having sex. It’s only a tiny tiny amount of time that we’ve been working on maths, science, literature, philosophy.

So actually, these models have a massive, massive amount of training more than humans had to do roughly the same thing but using insane amounts of computing power and energy.

Our brains were evolved for a completely different world and environment and daily life that the life we lead now.

So yes, LLMs are good, but they have been exposed to more data and training time than any human could have unless we lived for 100000 years and still perform worse than we do in most problems!

Tell Boston Dynamics how to do that.

Mice inherited brain from their ancestors. You might think you don't need a working brain to reason about math, but that's because you don't know how thinking works, it's argument from ignorance.

People argue that humans have had the equivalent of training a frontier LLM for billions of years.

But training a frontier LLM involves taking multiple petabytes of data, effectively all of recorded human knowledge and experience, every book ever written, every scientific publication ever written, all of known maths, science, encylopedias, podcasts, etc. And then training that for millions of years worth of GPU-core time.

You cannot possibly equate human evolution with LLM training, it's ridiculous.

Our "training" time didn't involve any books, maths, science, reading, 99.9999% of our time was just in the physical world. So you can quite rationally argue that our brains ability to learn without training is radically better and more efficient that the training we do for LLMs.

Us running around in the jungle wasn't training our brain to write poetry or compose music.

This is a crux of your argument, you need to justify it. It sounds way off base to me. Kinda reads like an argument from incredulity.

Are you suggesting that somehow there were books in the jungle, or perhaps boardgames? Perhaps there was a computer lab in the jungle?

Were apes learning to conjugate verbs while munching on bananas?

I don't think I'm suggesting anything crazy here... I think people who say LLM training is equivalent to "billions of years of evolution" need to justify that argument far more than I need to justify that running around in the jungle is equivalent to mass processing petabytes of highly rich and complex dense and VARIED information.

One year of running around in the same patch of jungle, eating the same fruit, killing the same insects, and having sex with the same old group of monkeys isn't going to be equal to training with the super varied, complete, entirety of human knowledge, is it?

If you somehow think it is though, I'd love to hear your reasoning.

Language is one mode of expression, and humans have many. This is another factor that makes humans so effective. To be honest, I would say that physical observation is far more powerful than all the bodies of text, because it is comprehensive and can respond to interaction. But that is merely my opinion.

No-one should be arguing that an LLM training corpus is the same as evolution. But information comes in many forms.

I'm saying that LLM training is not equivalent to billions of years of evolution because LLMs aren't trained using evolutionary algorithms; there will always be fundamental differences. However, it seems reasonable to think that the effect of that "training" might be more or less around the same level.

I realise upon reading the OP's comment again that they may have been referring to "extrapolation", which is hugely problematic from the statistical viewpoint when you actually try to break things down.

My argument for compression asserts that LLMs see a lot of knowledge, but are actually quite small themselves. To output a vast amount of information in such a small space requires a large amount of pattern matching and underlying learned algorithms. I was arguing that humans are actually incredible compressors because we have many years of history in our composition. It's a moot point though, because it is the ratio of output to capacity that matters.

They can attempt to mimic the results for small instances of the problem, where there are a lot of worked examples in the dataset, but they will never ever be able to generalize and actually give the correct output for arbitrary sized instances of the problem. Not with current architectures. Some algorithms simply can't be expressed as a fixed-size matrix multiplication.

Were mammals the first thing? No. Earth was a ball of ice for a billion years - all life at that point existed solely around thermal vents at the bottom of the oceans... that's inside of you, too.

Evolution doesn't forget - everything that all life has ever been "taught" (violently had programmed into us over incredible timelines) all that has ever been learned in the chain of DNA from the single cell to human beings - its ALL still there.

When did I say that? Of course you look at a human's experience when you judge the quality of their output. And you also judge their output based on the context they did their work in. Newton wouldn't be Newton if he was the 14th guy to claim that the universe is governed by three laws of motion. Extending the example I used above, I would be more impressed if an art student aced a tough calc test than a math student, given that a math student probably has spent much more time with the material.

"Intelligence and "thinking" are abstract concepts, and I'm simply putting forward a way that I think about them. It works very much outside the context of AI too. The "smartest" colleagues I've worked with are somehow able to solve a problem with less information or time than I need. Its usually not because they have more "training data" than me.

1. A desire to learn calculus 2. A good teacher 3. No mental impairments such as dementia or other major brain drainers

could not learn calculus. Most people don't really care to try or don't get good resources. What you see as an intelligent mathematician is almost always someone born with better resources that was also encouraged to pursue math.

Nevertheless, we don’t have a good conceptual framework for thinking about these things, perhaps because we keep trying to apply human concepts to them.

The way I see it, a LLM crystallises a large (but incomplete and disembodied) slice of human culture, as represented by its training set. The fact that a LLM is able to generate human-sounding language

LLMs aren't Data (Star Trek) or Replicants (Blade Runner). They're not even David or the androids from the movie A.I.

It's general intelligence, not general humanity

The idea that we shouldn't use the word until further clarification is rather hilarious. Let's wait hundred years until somebody defines it?

It's not how words work. People might introduce more specific terms, of course. But the word already means what we think it means.

1. All previous discussion of thinking was in nature to human and animal minds. The reason this is a question in the first place right now is because we ostensibly have a new thing which looks like a human mind but isn’t. That’s the question at hand here.

2. The question in this particular topic is not about technological “progress” or anything like it. It’s about determining whether machines can think, or if they are doing something else.

3. There are absolutely instances in which the previous word doesn’t quite fit the new development. We don’t say that submarines are swimming like a fish or sailing like a boat. To suggest that “no, actually they are just swimming” is pretty inadequate if you’re trying to actually describe the new phenomenon. AIs and thinking seem like an analogous situation to me. They may be moving through the water just like fish or boats, but there is obviously a new phenomenon happening.

> if we could find characters or signs appropriate for expressing all our thoughts as definitely and as exactly as arithmetic expresses numbers or geometric analysis expresses lines, we could in all subjects in so far as they are amenable to reasoning accomplish what is done in arithmetic and geometry

2. You're missing the fact that meaning of words is defined through their use. It's an obvious fact that if people call certain phenomenon "thinking" then they call that "thinking".

3. The normal process is to introduce more specific terms and keep more general terms general. E.g. people doing psychometrics were not satisfied with "thinking", so they introduced e.g. "fluid intelligence" and "crystallized intelligence" as different kinds of abilities. They didn't have to redefine what "thinking" means.

The AI companies obviously want the masses to just assume these are intelligent beings who think like humans and so we can just trust their output as being truthful.

I have an intelligent IT colleague who doesn't follow the AI news at all and who has zero knowledge of LLMs, other than that our company recently allowed us limited Copilot usage (with guidelines as to what data we are allowed to share). I noticed a couple weeks ago that he was asking it various mathematical questions, and I warned him to be wary of the output. He asked why, so I asked him to ask copilot/chatGPT "how many r letters are in the word strawberry". Copilot initially said 2, then said after thinking about it, that actually it was definitely 3, then thought about it some more then said it can't say with reasonable certainty, but it would assume it must be 2. We repeated the experiment with completely different results, but the answer was still wrong. On the 3rd attempt, it got it right, though the "thinking" stages were most definitely bogus. Considering how often this question comes up in various online forums, I would have assumed LLM models would finally get this right but alas, here we are. I really hope the lesson instilled some level of skepticism to just trust the output of AI without first double-checking.

But that word can mean different things to different people. With no definition, how can you even begin to have a discussion around something?

Top-down theory of word definitions is just wrong. People are perfectly capable of using words without any formalities.

In any case, even if there isnt a _single_ definition of a given subject, in order to have a discussion around a given area, both sides need to agree on some shared understanding to even begin to debate in good faith in the first place. It's precisely this lack of definition which causes a breakdown in conversation in a myriad of different areas. A recent obvious (morbid) example would be "genocide".

Wittgenstein, who's considered one of most brilliant philosophers of XX century, in _Philosophical Investigations_ (widely regarded as the most important book of 20th-century philosophy) does not provide definitions, but instead goes through a series of examples, remarks, etc. In preface he notes that this structure is deliberate and he could not write it differently. The topic of the book includes philosophy of language ("the concepts of meaning, of understanding, of a proposition, of logic, the foundations of mathematics, states of consciousness,...").

His earlier book _Tractatus Logico-Philosophicus_ was very definition-heavy. And, obviously, Wittgenstein was well aware of things like dictionaries, and, well, all philosophical works up to that point. He's not the guy who's just slacking.

Another thing to note is that attempts to build AI using definitions of words failed, and not for a lack of trying. (E.g. Cyc project is running since 1980s: https://en.wikipedia.org/wiki/Cyc). OTOH LLMs which derive word meaning from usage rather than definition seems to work quite well.

I propose calling it "thunking"

Humans are special, we emit meaning the way stars emit photons, we are rare in the universe as far as empirical observation has revealed. Even with AGI the existence of each complex meaning generator will be a cosmic rarity.

For some people that seems to be not enough, due to their factually wrong word views they see themselves as common and worthless (when they empirically aren't) and need this little psychological boost of unexaminable metaphysical superiority.

But there is an issue of course, the type of thinking humans do is dangerous but net positive and relatively stable, we have a long history where most instantiations of humans can persist and grow themselves and the species as a whole, we have a track record.

These new models do not, people have brains that as they stop functioning they stop persisting the apparatus that supports the brain and they die, people tend to become less capable and active as their thinking deteriorates and hold less influence ocer others accept in rare cases.

This is not the case for an LLM, they seem to be able to hallucinate endlessly and as they have access to the outside world maintain roughly their same amount of causal leverage, their clarity and accuracy of their thinking isn't tied to their persisting.

Until we have empirical evidence to the contrary we need to preserve our species.

If we discover other smarter species or never do, either way I don't care, it's immaterial to the precautionary principle.

We are fucking awesome and rare, and any other species with our amount of meaning generation or even capability for meaning generation is also fucking awesome.

I would 100% grant that ceatacenas and octopuses have human or higher level intelligence, I don't care, I don't need to put other species capabilities down to highlight my species accomplishment, the simple fact is that we have written more facts about the universe, discovered more, done more, gone further than any species we have empirically observed.

I mean it's incontrovertibly true, maybe dolphins have crazy libraries I'm not aware of, but until we verify that fact we need to preserve ourselves (and afterwards too), and we should preserve them too.

Even with other species, aliens, etc, they all need to be preserved because we can't ex ante predict which entities within a species will be part of the causal chain that solves entropy (if it's even possible.)

That's prima facie absurd on the face of it, so I don't know what it means. You would have to a philosophical zombie to make such an argument.

you guys would have called lightsabers "laser swords" like Lucas originally did before Alec Guinness corrected him

How about Artificial Intelligence?

I still say it needs a new name. If we want to be generous, we could state "the limit as time approaches infinity of thunking, is thinking." (I don't believe we will ever achieve astronomically-superior AGI, and certainly don't believe it will ever have a will of its own that someone else didn't give it- which just makes it a tool.)

also, sorry but you (fellow) nerds are terrible at naming.

while "thunking" possibly name-collides with "thunks" from CS, the key is that it is memorable, 2 syllables, a bit whimsical and just different enough to both indicate its source meaning as well as some possible unstated difference. Plus it reminds me of "clunky" which is exactly what it is - "clunky thinking" aka "thunking".

And frankly, the idea it's naming is far bigger than what a "thunk" is in CS

If you do math in your head or math with a pencil/paper or math with a pocket calculator or with a spreadsheet or in a programming language, it is all the same thing.

The only difference with LLMs is the anthropomorphization of the tool.

Which doesn't really help because you can of course say that when you ask an LLM a question of opinion and it responds, it's having an opinion or that it's just predicting the next token and in fact has no opinions because in a lot of cases you could probably get it to produce the opposite opinion.

Same with the second definition - seems to really hinge on the definition of the word mind. Though I'll note the definitions for that are "The element of a person that enables them to be aware of the world and their experiences, to think, and to feel; the faculty of consciousness and thought." and "A person's intellect." Since those specify person, an LLM wouldn't qualify, though of course dictionaries are descriptive rather than prescriptive, so fully possible that meaning gets updated by the fact that people start speaking about LLMs as though they are thinking and have minds.

Ultimately I think it just... doesn't matter at all. What's interesting is what LLMs are capable of doing (crazy, miraculous things) rather than whether we apply a particular linguistic label to their activity.

It's possible there is much thinking that does not happen with written word. It's also possible we are only thinking the way LLMs do (by chaining together rationalizations from probable words), and we just aren't aware of it until the thought appears, whole cloth, in our "conscious" mind. We don't know. We'll probably never know, not in any real way.

But it sure seems likely to me that we trained a system on the output to circumvent the process/physics because we don't understand that process, just as we always do with ML systems. Never before have we looked at image classifications and decided that's how the eye works, or protein folding and decided that's how biochemistry works. But here we are with LLMs - surely this is how thinking works?

Regardless, I submit that we should always treat human thought/spirit as unknowable and divine and sacred, and that anything that mimics it is a tool, a machine, a deletable and malleable experiment. If we attempt to equivocate human minds and machines there are other problems that arise, and none of them good - either the elevation of computers as some kind of "super", or the degredation of humans as just meat matrix multipliers.

> It may be that this tech produces clear, rational, chain of logic writeups, but it's not clear that just because we also do that after thinking that it is only thinking that produces writeups.

I appreciate the way you describe this idea, I find it likely I'll start describing it the same way. But then you go on to write:

> Regardless, I submit that we should always treat human thought/spirit as unknowable and divine and sacred, and that anything that mimics it is a tool, a machine, a deletable and malleable experiment. If we attempt to equivocate human minds and machines there are other problems that arise, and none of them good - either the elevation of computers as some kind of "super", or the degredation of humans as just meat matrix multipliers.

Which I find to be the exact argument that you started by discarding.

It's not clear that equating organic, and synthetic thought will have any meaningful outcome at all, let alone worthy of baseless anxiety that it must be bad. Equally it seems absolutely insane to claim that anything is unknowable, and that because humanity doesn't have a clear foundational understanding that we should pretend that it's either devine, or sacred. Having spent any time watching the outcome of the thoughts of people, neither devine nor sacred are reasonable attributes to apply, but more importantly, I'd submit that you shouldn't be afraid to explore things you don't know, and you shouldn't advocate for others to adopt your anxieties.

I agree! I'm saying "If we equate them, we shortcut all the good stuff, e.g., understanding", because "it may be that this tech produces what we can, but that doesn't mean we are the same", which is good because it keeps us learning vs reducing all of "thinking" to just "Whatever latest chatgpt does". We have to continue to believe there is more to thinking, if only because it pushes us to make it better and to keep "us" as the benchmark.

Perhaps I chose the wrong words, but in essence what I'm saying is that giving up agency to a machine that was built to mimic our agency (by definition as a ML system) should be avoided at all costs.

Actually we have, several times. But the way we arrived at those conclusions is worth observing:

1. ML people figure out how the ML mechanism works.

2. Neuroscientists independently figure out how brains do it.

3. Observe any analogies that may or may not exist between the two underlying mechanisms.

I can't help but notice how that's a scientific way of doing it. By contrast, the way people tend to arrive at similar conclusions when talking about LLMs tends to consist of observing that two things are cosmetically similar, so they must be the same. That's not just pseudoscientific; it's the mode of reasoning that leads people to believe in sympathetic magic.

We spent decades slowly working towards this most recent sprint towards AI without ever landing on definitions of intelligence, consciousness, or sentience. More importantly, we never agreed on a way to recognize those concepts.

I also see those definitions as impossible to nail down though. At best we can approach it like disease - list a number of measurable traits or symptoms we notice, draw a circle around them, and give that circle a name. Then we can presume to know what may cause that specific list of traits or symptoms, but we really won't ever know as the systems are too complex and can never be isolated in a way that we can test parts without having to test the whole.

At the end of the day all we'll ever be able to say is "well it’s doing a thing that looks like thinking, ergo it’s thinking”. That isn't lazy, its acknowledging the limitations of trying to define or measure something that really is a fundamental unknown to us.

I have had this conversation too many times on HN. What I find astounding is the simultaneous confidence and ignorance on the part of many who claim LLMs are intelligent. That, and the occultism surrounding them. Those who have strong philosophical reasons for thinking otherwise are called "knee-jerk". Ad hominem dominates. Dunning-Kruger strikes again.

So LLMs produce output that looks like it could have been produced by a human being. Why would it therefore follow that it must be intelligent? Behaviorism is a non-starter, as it cannot distinguish between simulation and reality. Materialism [2] is a non-starter, because of crippling deficiencies exposed by such things as the problem of qualia...

Of course - and here is the essential point - you don't even need very strong philosophical chops to see that attributing intelligence to LLMs is simply a category mistake. We know what computers are, because they're defined by a formal model (or many equivalent formal models) of a syntactic nature. We know that human minds display intentionality[0] and a capacity for semantics. Indeed, it is what is most essential to intelligence.

Computation is a formalism defined specifically to omit semantic content from its operations, because it is a formalism of the "effective method", i.e., more or less procedures that can be carried out blindly and without understanding of the content it concerns. That's what formalization allows us to do, to eliminate the semantic and focus purely on the syntactic - what did people think "formalization" means? (The inspiration were the human computers that used to be employed by companies and scientists for carrying out vast but boring calculations. These were not people who understood, e.g., physics, but they were able to blindly follow instructions to produce the results needed by physicists, much like a computer.)

The attribution of intelligence to LLMs comes from an ignorance of such basic things, and often an irrational and superstitious credulity. The claim is made that LLMs are intelligent. When pressed to offer justification for the claim, we get some incoherent, hand-wavy nonsense about evolution or the Turing test or whatever. There is no comprehension visible in the answer. I don't understand the attachment here. Personally, I would find it very noteworthy if some technology were intelligent, but you don't believe that computers are intelligent because you find the notion entertaining.

LLMs do not reason. They do not infer. They do not analyze. They do not know, anymore than a book knows the contents on its pages. The cause of a response and the content of a response is not comprehension, but a production of uncomprehended tokens using uncomprehended rules from a model of highly-calibrated token correlations within the training corpus. It cannot be otherwise.[3]

[0] For the uninitiated, "intentionality" does not specifically mean "intent", but the capacity for "aboutness". It is essential to semantic content. Denying this will lead you immediately into similar paradoxes that skepticism [1] suffers from.

[1] For the uninitiated, "skepticism" here is not a synonym for critical thinking or verifying claims. It is a stance involving the denial of the possibility of knowledge, which is incoherent, as it presupposes that you know that knowledge is impossible.

[2] For the uninitiated, "materialism" is a metaphysical position that claims that of the dualism proposed by Descartes (which itself is a position riddled with serious problems), the res cogitans or "mental substance" does not exist; everything is reducible to res extensa or "extended substance" or "matter" according to a certain definition of matter. The problem of qualia merely points out that the phenomena that Descartes attributes exclusively to the former cannot by definition be accounted for in the latter. That is the whole point of the division! It's this broken view of matter that people sometimes read into scientific results.

[3] And if it wasn't clear, symbolic methods popular in the 80s aren't it either. Again, they're purely formal. You may know what the intended meaning behind and justification for a syntactic rule is - like modus ponens in a purely formal sense - but the computer does not.

Your premise is wrong.

Unless you want to claim that the distant cause by way of the training data is us, but that's exactly the conclusion you're trying to avoid. After all, we put the patterns in the training data, which means we already did the upfront intellectual work for the LLM.

> LLMs do not reason. They do not infer. They do not analyze.

(definitions from Oxford Languages)

reason(v): think, understand, and form judgments by a process of logic.

to avoid being circular, I'm willing to write this one off because of the 'think' and 'understand', as those are the root of the question here. However, forming a judgement by a process of logic is precisely what these LLMs do, and we can see that clearly in chain-of-logic LLM processes.

infer(v): deduce or conclude (information) from evidence and reasoning rather than from explicit statements.

Again, we run the risk of circular logic because of the use of 'reason'. An LLM is for sure using evidence to get to conclusions, however.

analyze(v): examine methodically and in detail the constitution or structure of (something, especially information), typically for purposes of explanation and interpretation.

This one I'm willing to go to bat for completely. I have seen LLM do this, precisely according to the definition above.

For those looking for the link to the above definitions - they're the snippets google provides when searching for "SOMETHING definition". They're a non-paywalled version of OED definitions.

Philosophically I would argue that it's impossible to know what these processes look like in the human mind, and so creating an equivalency (positive or negative) is an exercise in futility. We do not know what a human memory looks like, we do not know what a human thought looks like, we only know what the output of these things looks like. So the only real metric we have for an apples-to-apples comparison is the appearance of thought, not the substance of the thing itself.

That said, there are perceptible differences between the output of a human thought and what is produced by an LLM. These differences are shrinking, and there will come a point where we can no longer distinguish machine thinking and human thinking anymore (perhaps it won't be an LLM doing it, but some model of some kind will). I would argue that at that point the difference is academic at best.

Say we figure out how to have these models teach themselves and glean new information from their interactions. Say we also grant them directives to protect themselves and multiply. At what point do we say that the distinction between the image of man and man itself is moot?

I don't know how you arrived at that conclusion. This is no mystery. LLMs work by making statistical predictions, and even the word "prediction" is loaded here. This is not inference. We cannot clearly see it is doing inference, as inference is not observable. What we observe is the product of a process that has a resemblance to the products of human reasoning. Your claim is effectively behaviorist.

> An LLM is for sure using evidence to get to conclusions, however.

Again, the certainty. No, it isn't "for sure". It is neither using evidence nor reasoning, for the reasons I gave. These presuppose intentionality, which is excluded by Turing machines and equivalent models.

> [w.r.t. "analyze"] I have seen LLM do this, precisely according to the definition above.

Again, you have not seen an LLM do this. You have seen an LLM produce output that might resemble this. Analysis likewise presupposes intentionality, because it involves breaking down concepts, and concepts are the very locus of intentionality. Without concepts, you don't get analysis. I cannot understate the centrality of concepts to intelligence. They're more important than inference and indeed presupposed by inference.

> Philosophically I would argue that it's impossible to know what these processes look like in the human mind, and so creating an equivalency (positive or negative) is an exercise in futility.

That's not a philosophical claim. It's a neuroscientific one that insists that the answer must be phrased in neuroscientific terms. Philosophically, we don't even need to know the mechanisms or processes or causes of human intelligence to know that the heart of human intelligence is intentionality. It's implicit in the definition of what intelligence is! If you deny intentionality, you subject yourself to a dizzying array of incoherence, beginning with the self-refuting consequence that you could not be making this argument against intentionality in the first place without intentionality.

> At what point do we say that the distinction between the image of man and man itself is moot?

Whether something is moot depends on the aim. What is your aim? If you aim is theoretical, which is to say the truth for its own sake, and to know whether something is A or something is B and whether A is B, then it is never moot. If your aim is practical and scoped, if you want some instrument that has utility indistinguishable from or superior to that of a human being in the desired effects that it produces, then sure, maybe the question is moot in that case. I don't care if my computer was fabricated by a machine or a human being. I care about the quality of the computer. But then, in the latter case, you're not really asking whether there is a distinction between man and the image of man (which, btw, already makes the distinction that for some reason you want to forget or deny, as the image of a thing is never the same as the thing). So I don't really understand the question. The use of the word "moot" seems like a category mistake here. Besides, the ability to distinguish two things is an epistemic question, not an ontological one.

I ask the LLM to do some or another assessment. The LLM prints out the chain-of-thought (whether that moniker is accurate is academic - we can read the chain and see that at the very least, it follows a form recognizable as logic). At the end of the chain-of-thought, we are left with a final conclusion that the model has come to - a judgement. Whether the internal state of the machine looks anything like our own is irrelevant to these definitions, much like writing out a formalism (if A then B, if B then C, A implies C). Those symbols do not have any form save for the shape of them, but when used in accordance with the rules we have laid out regarding logic, they have meaning nonetheless.

I'd similarly push back against the idea that the LLM isn't using evidence - I routinely ask my LLMs to do so, and they search on the web, integrating the information gleaned into a cohesive writeup, and provide links so I can check their work. If this doesn't constitute "using evidence" then I don't know what does.

w.r.t. "analyze", I think you're adding some human-sauce to the definition. At least in common usage, we've used the term analyze to refer to algorithmic decoction of data for decades now - systems that we know for a fact have no intentionality other than directed by the user.

I think I can divine the place where our understandings diverge, and where we're actually on the same track. Per Dennet, I would agree with you that the current state of an LLM lacks intrinsic intention and thus certain related aspects of thought. Any intent must be granted by the user, at the moment.

However, it is on this point that I think we're truly diverging - whether it is possible for a machine to ever have intent. To the best of my understanding, animal intent traces it's roots to the biological imperative - and I think it's a bit of hubris to think that we can separate that from human intent. Now, I'm an empiricist before anything else, so I have to qualify this next part by saying it's a guess, but I suppose that all one needs to qualify for intent is a single spark - a directive that lives outside of the cognitive construct. For us, it lives in Maslow's hierarchy - any human intent can be traced back to some directive there. For a machine, perhaps all that's needed is to provide such a spark (along with a loop that would allow the machine to act without the prodding of the enter key).

I should apologize in advance, at this point, because I'm about to get even more pedantic. Still, I feel it relevant so let's soldier on...

As for whether the image of a thing is a thing, I ask this: is the definition of a thing, also that thing? When I use a phrase to define a chair, is the truth of the existence of that collection of atoms and energy contained within the word "chair", or my meaning in uttering it? Any idea that lives in words is constrained by the understanding of the speaker - so when we talk about things like consciousness and intentionality and reasoning we are all necessarily taking shortcuts with the actual Truth. It's for this reason that I'm not quite comfortable with laying out a solid boundary where empirical evidence cannot be built to back it up.

If I seem to be picking at the weeds, here, it's because I see this as an impending ethical issue. From what my meagre understanding can grok, there is a nonzero chance that we are going to be faced with determining the fate of a possibly conscious entity birthed from these machines in our lifetime. If we do not take the time to understand the thing and write it off as "just a machine", we risk doing great harm. I do not mean to say that I believe it is a foregone conclusion, but I think it right and correct that we be careful in examining our own presuppositions regarding the nature and scope of the thing. We have never had to question our understanding of consciousness in this way, so I worry that we are badly in need of practice.

This is a pretty messy argument as computers have been simulating material interactions for quite some time now.

Seems like you have that backwards. If consciousness is from a nonphysical process, like a soul that's only given to humans, then it follows that you can't build consciousness with physical machines. If it's purely physical, it could be built.

Your very idea is also predicated on the idea that it's possible for a real object to exist that isn't physical, and I think most modern philosophers reject the idea of a spiritual particle.

The simpler belief than the idea that just any intelligent process is conscious is that consciousness is some kind of physical process that happens in particular conditions.

I understand why computation doesn't intuitively seem the same as having a subjective conscious experience. But why is it any more intuitive that a bunch of protons, neutrons and electrons pushing and pulling on each other give rise to consciousness, but somehow do it by doing something other than computation?

If I impose a computational interpretation onto inert material after the fact, did that material possess the consciousness? Suppose I overlay a projection of the calculation of an AGI onto a circuit board, is that projection equally conscious to if that circuit board did the computation in the usual way?

No, that is absurd. The reasonable view is that computation is irrelevant and what is relevant is some special physical process.

But the alternative is equally absurd. In essence, you have a bunch protons, neutrons and electrons pushing and pulling on each other and somehow that causes consciousness. There just aren't that many weird properties or particles to appeal to as a cause of consciousness. While there are exotic particles and the quarks and gluons that make up protons/neutrons, these particles are relevant for our study of the universe, but they aren't relevant to understanding the human brain. Abstracting protons as a single particle that push and pull on other particles via forces works fine for explaining everything we know about the brain.

Not that it really changes anything if the particular way quarks interact did affect the brain in a way that couldn't be explained through the simplified view of a proton. It adds a few more particles to consider and the weirdness of quantum chromodynamics, but nothing there explains consciousness either.

So how do you go from particles pushing and pulling on each other to consciousness? It seems to me no matter how you arrange a bunch of particles, there is never any reason to assume that arrangement is conscious. It's just a bunch of points moving according to a few simple rules.

If you're saying there is something more in physics, some other force or particle or currently unknown way particles interact that causes consciousness, that's roughly the soul particle idea.

Saying consciousness is caused by a physical process is very vague. What kind of process? Why don't the physical processes the computers running an LLM perform count? Why does the brain qualify? If we go on to build many different types of artificial brains (using silicon or biological cells or anything else), how would you even go about recognizing which artificial thinking machines have the required physical process for consciousness?

>>>"Not that it really changes anything if the particular way quarks interact did affect the brain in a way that couldn't be explained through the simplified view of a proton. It adds a few more particles to consider and the weirdness of quantum chromodynamics, but nothing there explains consciousness either.

>>>So how do you go from particles pushing and pulling on each other to consciousness? It seems to me no matter how you arrange a bunch of particles, there is never any reason to assume that arrangement is conscious. It's just a bunch of points moving according to a few simple rules."

The pushing and pulling is another computational process. Consciousness must be some irreducible component of matter or of certain arrangements of matter, so that this matter "just is conscious". The pulling / pushing done in the correct way imposes a complex experience on that matter. I would love to tell you what it is, but my reasoning is a process of elimination, where we eliminate the popular idea that consciousness = computation now when it is very important to humanity to do so.

>>>"equally absurd".

I don't think it is, and I am very passionate about this. If computation is the only explainer of consciousness, then every large enough random collection of bits can be interpreted as representing some conscious process given a complicated enough interpretation rule set. E.g. take an exabyte of random data and extract from it a 1 mb ChatGPT conversation by deciding to include this bit and to exclude that bit. Who's to say that my rule for extracting the correct bits isn't a valid computational process? If I just write a random bit string over an exabyte memory bank over and over, and if some way of interpreting part of it at this frame, and some way of interpreting it in the next frame etc.. etc.. results in an intelligent conversation, was a conscious being simulated? Let's make it more absurd. Take a computation which expresses consciousness. Print out the conversation in bit form onto a piece of paper. Cut up and rearrange the paper into a sequence first of all the 0's and then all the 1's. Then show the result to your friend and say this can be interpreted at a conscious process, therefore is conscious. This is ridiculous. Why is a blank sheet of paper not a conscious process if some guy says by looking at the paper he imagines 50 0's in a row and 100 1's in a row, and those can be rearranged to express a thought in some manner?

>>>"how would you even go about recognizing which artificial thinking machines have the required physical process for consciousness?"

We would not be able to tell until we actually learn more physics, which is the deepest reason I think it is unethical to build them at all. My position on the physical nature of consciousness makes me believe that everything with a neuron-based brain probably has consciousness. And so if we are hell-bent on making something with consciousness, we could do it by growing a brain in a lab. That's not to say I believe only a brain can have consciousness, it's just that it is the only kind I will have confidence in for the time being. The reason I doubt our computer hardware has consciousness is that computation is abundant in the universe (basically anything can be interpreted as computation), and so I doubt that just any arbitrary hardware we've created is likely to interact the computation with the special sauce in the right way.

If that's all there is, it seems likely you could create consciousness out of marbles connected by springs. If protons and electrons don't have any special consciousness-related property other than pushing and pulling, we'd just have to recreate the arrangement of matter and the forces to get consciousness.

There is no way to make the leap from particles and forces to consciousness that seems intuitively right. No arrangement of dumb matter, whether marbles or protons, should be conscious.

On a somewhat related note: forces are rules that dictate how things move. At some level, position and movement is the only thing we can measure. When we measure something like voltage or charge, our measurement techniques actually rely on observing the movement of particles.

Consciousness doesn't seem like it's just movement. But if there were some other property of our universe that didn't cause movement, we would have no way to measure it. It would have no observable effect on matter. Which would make epiphenomenal consciousness seemingly impossible to prove or disprove even if it were some basic property of physics.

This exact argument applies to "free will", and that definition has been debated for millennia. I'm not saying don't try, but I am saying that it's probably a fuzzy concept for a good reason, and treating it as merely a behavioural descriptor for any black box that features intelligence and unpredictable complexity is practical and useful too.

Even in this thread, the number of people claiming some mystical power separating humans from all the rest of nature is quite noticeable.

Part of the issue is that our general concept of equality is limited by a first order classical logic which is a bad basis for logic

There is a way to discuss if keeping intelligent artificial life under servitude without using those terms, especially if you're on a new account.

One, it makes it appear that all ethical and social conventions for the one can be applied to the other and it's a done deal.

Two, it makes it appear that conclusions are also reversible, and that turns seemingly enlightened libertarian ideas into munition for racism.

Life solves problems itself poses or collides with. Tools solve problems only when applied.

Weather models sometimes “predict” a real pattern by chance, yet we don’t call the atmosphere intelligent.

If LLMs were truly thinking, we could enroll one at MIT and expect it to graduate, not just autocomplete its way through the syllabus or we could teach one how to drive.

While I'm not willing to rule *out* the idea that they're "thinking" (nor "conscious" etc.), the obvious counter-argument here is all the records we have of humans doing thinking, where the records themselves are not doing the thinking that went into creating those records.

And I'm saying this as someone whose cached response to "it's just matrix multiplication it can't think/be conscious/be intelligent" is that, so far as we can measure all of reality, everything in the universe including ourselves can be expressed as matrix multiplication.

Falsification, not verification. What would be measurably different if the null hypothesis was wrong?

They can certainly produce good-sounding answers, but as to the goodness of the advice they contain, YMMV.

The point being made here is about the data LLMs have been trained with. Sure that contains questions&answers but obviously not all of it is in that form. Just like an encyclopedie contains answers without the questions. So imo specifying this as 'no-one asked this before' is irrelevant.

More interesting: did OP get a sensible answer to a question about data which definitely was not in the training set? (and indeed, how was this 'definitely' established'). Not that if the answer is 'yes' that'll prove 'thinking', as opposed to calling it e.g. advanced autocompletion, but it's a much better starting point.

Specifically I wanted to know how I could interface two electronic components, one of which is niche, recent, handmade and doesn't have any public documentation so there's no way it could have known about it before.

I still see 2 possibilities: you asked it something similar enough that it came up with a fairly standard answer which just happened to be correct, or you gave it enough info.

- for example you created a new line of MCUs called FrobnicatorV2, and asked is 'how do I connect a power supply X to FrobnicatorV2' and it gave an answer like 'connect red wire to VCC and black to GND'. That's not exactly special.

- or, you did desribe that component in some way. And you did do that using standard electronics lingo so essentially in terms of other existing components which it definitely did know (unless you invented something completely new not using any currently know physics). As such it's irrelevant that your particular new component wasn't known because you gave away the answer by describing it? E.g. you aksed it 'how do I connect a power supply X to an MCU with power pins Y and Z'. Again nothing special.

You're saying it's nothing "special" but we're not discussing whether it's special, but whether it can be considered thinking.

Not verbatim in the sense that the words are different doesn't make it thinking. Also when we say 'humans think' that means a lot more than only 'new question generates correct answer' or 'smart autocompletion'. See a lot of other comments here for details.

But again: I laid out 2 possibilities explaining why the question might in fact not be new, nor the data, so I'm curious which of the 2 (or another) explains the situation you're talking about.

You're saying it's nothing "special" but we're not discussing whether it's special, but whether it can be considered thinking.

Apologies, with 'special' I did in fact mean 'thinking'

So many times I've seen it produce sensible, valid chains of results.

Yes, I see evidence in that outcome that a person somewhere thought and understood. I even sometimes say that a computer is "thinking hard" about something when it freezes up.

...but ascribing new philosophical meaning to this simple usage of the word "thinking" is a step too far. It's not even a new way of using the word!

But yeah, I am fully willing to believe that a silicon based life form could think and be alive. i just don't think we're there. Yes this thing speaks using a passable imitation of the voices of PhDs and poets, but in a way a simulated annelid is more alive.

I’ve asked LLMs to write code for me in fields I have little background knowledge, and then had to debug the whole thing after essentially having to learn the language and field.

On the other hand, for things I am well versed in, I can debug the output and avoid entire swathes of failed states, by having a clear prompt.

Its why I now insist that any discussion on GenAI projects also have the speaker mention the level of seniority they have ( proxy for S/W eng experience), Their familiarity with the language, the project itself (level of complexity) - more so than the output.

I also guarantee - that most people have VERY weak express knowledge of how their brains actually work, but deep inherent reflexes and intuitions.

The Roomba vacuumed the room. Maybe it vacuumed the whole apartment. This is good and useful. Let us not diminish the value of the tool. But it's a tool.

The tool may have other features, such as being self-documenting/self-announcing. Maybe it will frighten the cats less. This is also good and useful. But it's a tool.

Humans are credulous. A tool is not a human. Meaningful thinking and ideation is not just "a series of steps" that I will declaim as I go merrily thinking. There is not just a vast training set ("Reality"), but also our complex adaptability that enables us to test our hypotheses.

We should consider what it is in human ideation that leads people to claim that a Roomba, a chess programme, Weizenbaum's Eliza script, the IBM's Jeopardy system Watson, or an LLM trained on human-vetted data is thinking.

Train such a system on the erroneous statements of a madman and suddenly the Roomba, Eliza, IBM Watson (and these other systems) lose our confidence.

As it is today, the confidence we have in these systems is very conditional. It doesn't matter terribly if code is wrong... until it does.

Computers are not humans. Computers can do things that humans cannot do. Computers can do these things fast and consistently. But fundamentally, algorithms are tools.

Human: I'm trying to get my wolf, sheep and cabbage across the river in this boat, but the wolf keeps eating the sheep or the sheep eats the cabbage

Bot: You should put the sheep in the boat and take it across — if we delve into the biology of Canis lupus we discover that wolves don't eat cabbage!

H: Ok, so that worked great so far, the sheep is on one side and the wolf/cabbage is on the other.

B: Now, Option 1 is to bring the wolf across, or Option 2 you can bring the cabbage. I recommend (2) taking the cabbage as cabbages are smaller and easier to transport in a boat.

H: But then the sheep eats the cabbage, right? Remember that?

B: Exactly — if you put the sheep and the cabbage together on the same side of the river, the sheep is sure to devour the cabbage. That's why the key is to separate sheep from cabbages. :rocketship:

[1] https://jdsemrau.substack.com/p/nemotron-vs-qwen-game-theory...

https://youtu.be/_-agl0pOQfs?si=Xiyf0InqtjND9BnF

We train ourselves on content. We give more weight to some content than others. While listening to someone speak, we can often predict their next words.

What is thinking without language? Without language are we just bags of meat reacting to instincts and emotions? Are instincts and emotions what's missing for AGI?

I'm still not convinced they're thinking though because they faceplant on all sorts of other things that should be easy for something that is able to think.

They're not thinking, we're just really good at seeing patterns and reading into things. Remember, we never evolved with non-living things that could "talk", we're not psychologically prepared for this level of mimicry yet. We're still at the stage of Photography when people didn't know about double exposures or forced perspective, etc.

I know I am a mind inside a body, but I'm not sure about anyone else. The easiest explanation is that most of the people are like that as well, considering we're the same species and I'm not special. You'll have to take my word on that, as my only proof for this is that I refuse to be seen as anything else.

In any case LLMs most likely are not minds due to the simple fact that most of their internal state is static. What looks like thoughtful replies is just the statistically most likely combination of words looking like language based on a function with a huge number of parameters. There's no way for this construct to grow as well as to wither - something we know minds definitely do. All they know is a sequence of symbols they've received and how that maps to an output. It cannot develop itself in any way and is taught using a wholly separate process.

This is not a compelling argument. Firstly, you can add external state to LLMs via RAG and vector databases, or various other types of external memory, and their internal state is no longer static and deterministic (and they become Turing complete!).

Second if you could rewind time, then your argument suggests that all other humans would not have minds because you could access the same state of mind at that point in time (it's static). Why would you travelling through time suddenly erases all other minds in reality?

The obvious answer is that it doesn't, those minds exist as time moves forward and then they reset when you travel backwards, and the same would apply to LLMs if they have minds, eg. they are active minds while they are processing a prompt.

But it's not the LLM that makes modifications in those databases - it just retrieves data which is already there.

> Why would you travelling through time suddenly erases all other minds in reality?

I'm not following you here.

> they are active minds while they are processing a prompt.

Problem is that this process doesn't affect the LLM in the slightest. It just regurgitates what it's been taught. An active mind is makes itself. It's curious, it gets bored, it's learning constantly. LLMs do none of that.

You couldn't get a real mind to answer the same question hundreds of times without it being changed by that experience.

So what?

> I'm not following you here.

If you're time travelling, you're resetting the state of the world to some previous well-defined, static state. An LLM also starts from some well-defined static state. You claim this static configuration means there's no mind, so this entails that the ability to time travel means that every person who is not time travelling has no mind.

> Problem is that this process doesn't affect the LLM in the slightest. It just regurgitates what it's been taught. An active mind is makes itself.

People who are incapable forming new memories thus don't have minds?

https://en.wikipedia.org/wiki/Anterograde_amnesia

Now, separately, you are precisely the type of closet dualist I speak of. You say that you are a mind inside a body, but you have no way of knowing that others have minds -- take this to it's full conclusion: You have no way of knowing that you have a "mind" either. You feel like you do, as a biological assembly (which is what you are). Either way you believe in some sort of body-mind dualism, without realizing. Minds are not inside of bodies. What you call a mind is a potential emergent phenomenon of a brain. (potential - because brains get injured etc.).

How would the mind know which one it is?

Maybe your mind is being simulated right now.

I'm not assuming it is without hard proof - that's my only argument.

> Maybe your mind is being simulated right now.

I'm experiencing consciousness right now, so that would have to be a damn good simulation.

Yes, debated and refuted. There are many well known and accepted rebuttals of the Chinese Room. The Chinese Room as a whole does understand Chinese.

I'm not assuming that, that's literally the definition of mimicry: to imitate closely.

You might say I'm assuming that it is mimicking and not actually thinking, but there's no evidence it's actually thinking, and we know exactly what is IS doing because we created the code that we used to build the model. They're not thinking, it's doing math, mathematical transformations of data.

Whatever thinking fundamentally is, it also has an equivalence as a mathematical transformation of data. You're assuming the conclusion by saying that the two mathematical transformations of data are not isomorphic.

A simulation of information processing is still information processing, just like running Windows in a QEMU VM is still running Windows.

Do not confuse the mathematical description of physical processes as the world being made of math.

> You're assuming the conclusion by saying that the two mathematical transformations of data are not isomorphic.

Correct. They're not isomorphic. One is simple math that runs on electrified sand, and one is an unknown process that developed independently across a billion years. Nothing we're doing with AI today is even close to real thought. There are a billion trivial proofs that make the rounds as memes, like one R in strawberry, or being unable to count, etc.

Again, this doesn't apply to information. A simulation of a computation really is equivalent to that computation.

> One is simple math that runs on electrified sand, and one is an unknown process that developed independently across a billion years.

Right, so you admit that it's an unknown process, which means you literally cannot conclude that it is different to what LLMs are doing.

> There are a billion trivial proofs that make the rounds as memes, like one R in strawberry, or being unable to count, etc.

No, none of these are definitive proofs that they are not thinking. LLM "perceptions" are tokens, the strawberry question is basically asking it to figure out something that's below it's perceptual range. This has literally nothing to do with whether the way it processes information is or is not thinking.

If you truly feel human thinking and LLMs share more than a cocktail napkin's worth of similarity, I don't know what to say. Just treating it like a black box I can prove in minutes it's not thinking. Come on. I really don't get why people are so emotionally involved in this stuff.

It's not thinking. It's ok that it's not thinking. Maybe someday we'll get there, but it's not today.

> Knee jerk dismissing the evidence in front of your eyes

Anthropomorphizing isn't any better.

That also dismisses the negative evidence, where they output completely _stupid_ things and make mind boggling mistakes that no human with a functioning brain would do. It's clear that there's some "thinking" analog, but there are pieces missing.

I like to say that LLMs are like if we took the part of our brain responsible for language and told it to solve complex problems, without all the other brain parts, no neocortex, etc. Maybe it can do that, but it's just as likely that it is going to produce a bunch of nonsense. And it won't be able to tell those apart without the other brain areas to cross check.

https://huggingface.co/PantheonUnbound/Satyr-V0.1-4B

But oh boy have I also seen models come up with stupendously dumb and funny shit as well.

You go ahead with your imagination. To us unimaginative folks, it betrays a lack of understanding of how LLMs actually work and shows that a lot of people still cannot grasp that it’s actually an extremely elaborate illusion of thinking.