When it comes to the various kinds of thought-processes that humans engage in (linguistic thinking, logic, math, etc) I agree that you can describe things in terms of functions that have definite inputs and outputs. So human thinking is probably computable, and I think that LLMs can be said to be ”think” in ways that are analogous to what we do.
But human consciousness produces an experience (the experience of being conscious) as opposed to some definite output. I do not think it is computable in the same way.
I don’t necessarily think that you need to subscribe to dualism or religious beliefs to explain consciousness - it seems entirely possible (maybe even likely) that what we experience as consciousness is some kind of illusory side-effect of biological processes as opposed to something autonomous and “real”.
But I do think it’s still important to maintain a distinction between “thinking” (computable, we do it, AIs do it as well) and “consciousness” (we experience it, probably many animals experience it also, but it’s orthogonal to the linguistic or logical reasoning processes that AIs are currently capable of).
At some point this vague experience of awareness may be all that differentiates us from the machines, so we shouldn’t dismiss it.
> You've got to be careful when you say what the human does, if you add to the actual result of his effort some other things that you like, the appreciation of the aesthetic... then it gets harder and harder for the computer to do it because the human beings have a tendency to try to make sure that they can do something that no machine can do. Somehow it doesn't bother them anymore, it must have bothered them in earlier times, that machines are stronger physically than they are...
- Feynman
Maybe we can swap out "think" with "experience consciousness"
Impossible to answer.
Btw I mostly think it’s reasonable to think that there might be consciousness, phenomenology etc are possible in silicon, but it’s tricky and unverifiable ofc.
If the original one did, then yes, of course. You're performing the exact same processing.
Imagine if instead of an LLM the billions of people instead simulated a human brain. Would that human brain experience consciousness? Of course it would, otherwise they're not simulating the whole brain. The individual humans performing the simulation are now comparable to the individual neurons in a real brain. Similarly, in your scenario, the humans are just the computer hardware running the LLM. Apart from that it's the same LLM. Anything that the original LLM experiences, the simulated one does too, otherwise they're not simulating it fully.
> Imagine if instead of an LLM the billions of people instead simulated a human brain. Would that human brain experience consciousness? Of course it would, otherwise they're not simulating the whole brain.
However, I don’t really buy “of course it would,” or in another words the materialist premise - maybe yes, maybe no, but I don’t think there’s anything definitive on the matter of materialism in philosophy of mind. as much as I wish I was fully a materialist, I can never fully internalize how sentience can uh emerge from matter… in other words, to some extent I feel that my own sentience is fundamentally incompatible with everything I know about science, which uh sucks, because I definitely don’t believe in dualism!
We in a way can articulate the underlying chemputation of the universe mediated through our senses, reflection and language, turn a piece off (as it is often non continuous) and the quality of the experience changes.
The notion that it is not a physical process is an extraordinary claim in its own right, which itself requires evidence.
But your simulation will never fly you over an ocean, it will never be an aircraft or do what aircraft do. A simulation of heat transfer will not cook your dinner. A simulation of Your assumption that a simulation of a mind is a mind, requires evidence.
It will fly over a simulated ocean just fine. It does exactly what aircraft do, within the simulation. By adding “you” to the sentence you've made it an apples to oranges comparison because “you” is definitionally not part of the simulation. I don't see how you could add the same “you” to “it will simulate consciousness just fine”.
It doesn't move real Oxygen and Nitrogen atoms, it doesn't put exhaust gas into the air over the ocean, it doesn't create a rippling sound and pressure wave for a thousand miles behind it, it doesn't drain a certain amount of jet fuel from the supply chain or put a certain amount of money in airline and mechanics' pockets, it doesn't create a certain amount of work for air traffic controllers... reductio ad abusurdum is that a flipbook animation of a stickman aircraft moving over a wiggly line ocean is a very low granularity simulation and "does exactly what aircraft do" - and obviously it doesn't. No amount of adding detail to the simulation moves it one inch closer to doing 'exactly what aircraft do'.
> "I don't see how you could add the same “you” to “it will simulate consciousness just fine”"
by the same reductio-ad-absurdum I don't see how you can reject a stickman with a speech bubble drawn over his head as being "a low granularity simulated consciousness". More paper, more pencil graphite, and the stickman will become conscious when there's enough of it. Another position is that adding things to the simulation won't simulate consciousness just fine - won't move it an inch closer to being conscious; it will always be a puppet of the simulator, animated by the puppeteer's code, always wooden Pinocchio and never a real person. What is the difference between these two:
a) a machine with heat and light and pressure sensors, running some code, responding to the state of the world around it.
b) a machine with heat and light and pressure sensors, running some code [converting the inputs to put them into a simulation, executing the simulation, converting the outputs from the simulation], and using those outputs to respond to the state of the world around it.
? What is the 'simluate consciousness' doing here at all, why is it needed? To hide the flaw in the argument; it's needed to set up the "cow == perfectly spherical massless simulated cow" premise which makes the argument work in English words. Instead of saying something meaningful about consciousness, one states that "consciousness is indistinguishable from perfectly spherical massless simulated consiousness" and then states "simply simulate it to as much detail as needed" and that allows all the details to be handwaved away behind "just simulate it even more (bro)".
Pointing out that simulations are not the real thing is the counter-argument. Whether or not the counter-argument can be made by putting "you" into a specific English sentence is not really relevant, that's only to show that the simulated aircraft doesn't do what the real aircraft does. A simulated aircraft flying over a simulated ocean is no more 'real' than drawing two stick figures having a conversation in speech bubbles.
As far as physics go, it's all just numbers in the end. Indeed, the more we keep digging into the nature of reality, the more information theory keeps popping up - see e.g. the holographic principle.
No it isn't; numbers are a map, maps are not the territory. You are asking me to define how a map is different from a city, but you are not accepting that the city is made of concrete and is square kilometers large and the map is made of paper and is square centimeters large as a meaningful difference, when I think it's such an obvious difference it's difficult to put any more clearly.
What constitutes a real atom: a Hydrogen atom capable of combining with Oxygen to make water, capable of being affected by the magnetic field of an MRI scanner, etc.
What constitutes a simulated atom: a pattern of bits/ink/numbers which you say "this is a representation of a Hydrogen atom", capable of nothing, except you putting some more bits/ink/numbers near it and speaking the words "this is it interacting to make simulated water".
Do you deny that you could be in a simulation right now, in the matrix? What you actually think are are molecules of oxygen are actually simulated molecules. That there is no way for you to every tell the difference.
That's just semantics. I'm not here to argue what the word “real” means. Of course you can define it in such a way that the simulated aircraft isn't “really” flying over an ocean, and it would be just as valid as any other definition, but it doesn't say anything meaningful or insightful about the simulation.
Nobody contests your point that the simulated aircraft isn't going over a real ocean and isn't generating work for real-life air traffic controllers. But conversely you don't seem to contest the claim that oceans and air traffic controllers could be simulated, too. Therefore, consciousness can be simulated as well, and it would be a simulated consciousness that just doesn't fall into your definition of “real”.
"We've all been dancing around the basic issue: does Data have a soul?" -- Captain Louvois. https://memory-alpha.fandom.com/wiki/The_Measure_Of_A_Man_(e...
Both matter of course.
Function can mean inputs-outputs. But it can also mean system behaviors.
For instance, recurrence is a functional behavior, not a functional mapping.
Similarly, self-awareness is some kind of internal loop of information, not an input-output mapping. Specifically, an information loop regarding our own internal state.
Today's LLMs are mostly not very recurrent. So might be said to be becoming more intelligent (better responses to complex demands), but not necessarily more conscious. An input-output process has no ability to monitor itself, no matter how capable of generating outputs. Not even when its outputs involve symbols and reasoning about concepts like consciousness.
So I think it is fair to say intelligence and consciousness are different things. But I expect that both can enhance the other.
Meditation reveals a lot about consciousness. We choose to eliminate most thought, focusing instead on some simple experience like breathing, or a concept of "nothing".
Yet even with this radical reduction in general awareness, and our higher level thinking, we remain aware of our awareness of experience. We are not unconscious.
To me that basic self-awareness is what consciousness is. We have it, even when we are not being analytical about it. In meditation our mind is still looping information about its current state, from the state to our sensory experience of our state, even when the state has been reduced so much.
There is not nothing. We are not actually doing nothing. Our mental resting state is still a dynamic state we continue to actively process, that our neurons continue to give us feedback on, even when that processing has been simplified to simply letting that feedback of our state go by with no need to act on it in any way.
So consciousness is inherently at least self-awareness in terms of internal access to our own internal activity. And that we retain a memory of doing this minimal active or passive self-monitoring, even after we resume more complex activity.
My own view is that is all it is, with the addition of enough memory of the minimal loop, and a rich enough model of ourselves, to be able to consider that strange self-awareness looping state afterwards. Ask questions about its nature, etc.
Consciousness is nothing but the ability to have internal and external senses, being able to enumerate them, recursively sense them, and remember the previous steps. If any of those ingredients are missing, you cannot create or maintain consciousness.
> Meditation reveals a lot about consciousness. We choose to eliminate most thought, focusing instead on some simple experience like breathing, or a concept of "nothing".
The sensation of breathing still constitutes input. Nor is it a given that a thought is necessarily encodeable in words, so "thinking about concept of nothing" is still a thought, and there's some measurable electrochemical activity encoding that in the brain which encodes it. In a similar vein, LLMs deal with arbitrary tokens, which may or may not encode words - e.g. in multimodal LMs, input includes tokens encoding images directly without any words, and output can similarly be non-word tokens.
It is, but (1) the amount of looping in models today is extremely trivial. if our awareness loop is on the order of milliseconds, we experience it on the order of thousands of milliseconds at a minimum. And consider and consolidate our reasoning about experiences over minutes, hours, even days. Which would be thousands to many millions of iterations of experiential context.
Then (2), the looping of models today is not something the model is aware of at a higher level. It processes the inputs iteratively, but it isn't able to step back and examine its own responses recurrently at a second level in a different indirect way.
Even though I do believe models can reason about themselves and behave as if they did have that higher functionality.
But their current ability to reason like that has been trained into them by human behavior, not learned independently by actually monitoring their own internal dynamics. They cannot yet do that. We do not learn we are conscious, or become conscious, by parroting others conscious enabled reasoning. A subtle but extremely important difference.
Finally, (3) they don't build up a memory of their internal loops, much less a common experience from a pervasive presence of such loops.
Those are just three quite major gaps.
But they are not fundamental gaps. I have no doubt that future models will become conscious as limitations are addressed.
I imagined the Catholic Church, for example, would be publishing missives reminding everyone that only humans can have souls, and biologists would be fighting an quixotic battle to claim that consciousness can arise from physical structures and forces.
I'm still surprised at how credulous and accepting societies have been of AI developments over the last few years.
AI developments over the last few years have not needed that view to change.
I've heard this idea before but I have never been able to make head or tail of it. Consciousness can't be an illusion, because to have an illusion you must already be conscious. Can a rock have illusions?
Btw, Turing machines provide some inspiration for an interesting definition:
Turing (and Gödel) essentially say that you can't predict what a computer program does: you have to run it to even figure out whether it'll halt. (I think in general, even if you fix some large fixed step size n, you can't even predict whether an arbitrary program will halt after n steps or not, without essentially running it anyway.)
Humans could have free will in the same sense, that you can't predict what they are doing, without actually simulating them. And by an argument implied by Turing in his paper on the Turing test, that simulation would have the same experience as the human would have had.
(To go even further: if quantum fluctuations have an impact on human behaviour, you can't even do that simulation 100% accurately, because of the no cloning theorem.
To be more precise: I'm not saying, like Penrose, that human brains use quantum computing. My much weaker claim is that human brains are likely a chaotic system, so even a very small deviation in starting conditions can quickly lead to differences in outcome.
If you are only interested in approximate predictions, identical twins show that just getting the same DNA and approximation of the environment gets you pretty far in making good predictions. So cell level scans could be even better. But: not perfect.)
I think it's a good point, but I would argue it's even more direct than that. Humans themselves can't reliably predict what they are going to do before they do it. That's because any knowledge we have is part of our deliberative decision-making process, so whenever we think we will do X, there is always a possibility that we will use that knowledge to change our mind. In general, you can't feed a machine's output into its input except for a very limited class of fixed point functions, which we aren't.
So the bottom line is that seen from the inside, our self-model is a necessarily nondeterministic machine. We are epistemically uncertain about our own actions, for good reason, and yet we know that we cause them. This forms the basis of our intuition of free will, but we can't tell this epistemic uncertainty apart from metaphysical uncertainty, hence all the debate about whether free will is "real" or an "illusion". I'd say it's a bit of both: a real thing that we misinterpret.
Ie I wouldn't expect humans without free will to be able to predict themselves very well, either. Exactly as you suggest: having a fixed point (or not) doesn't mean you have free will.
I'm tempted to say an entity has free will if it a) has a self-model, b) uses this self-model as a kind of internal homunculus to evaluate decision options and c) its decisions are for the most part determined by physically internal factors (as opposed as external constraints or publicly available information). It's tempting to add a threshold of complexity, but I don't think there's any objectively correct way to define one.
> [...] c) its decisions are for the most part determined by physically internal factors (as opposed as external constraints or publicly available information).
I don't think humans reach that threshold. Though it depends a lot on how you define things.
But as far as I can tell, most of my second-to-second decisions are very much coloured by the fact that we have gravity and an atmosphere at comfortable temperatures (external factors), and if you changed that all of a sudden, I would decide and behave very differently.
> It's tempting to add a threshold of complexity, but I don't think there's any objectively correct way to define one.
Your homunculus is one hell of a complexity threshold.
We mistakenly assume, they are true because perhaps we want them to be true. But we have no proof that either of these are true.
Quick context: His view of what constitutes a subject, which is to say a thinking person in this case, is one which over time (and time is very important here) observes manifold partial aspects about objects through perception, then through apprehension (the building of understanding through successive sensibilities over time) the subject schematizes information about the object. Through logical judgments, from which Kant derives his categories, we can understand the object and use synthetic a priori reasoning about the object.
So for him, the statement "I am" means simply that you are a subject who performs this perception and reasoning process, as one's "existence" is mediated and predicated on doing such a process over time. So then "I think, therefore I am" becomes a tautology. Assuming that the "I" in "I am" exists as an object, which is to say a thing of substance, one which other thinking subjects could reason about, becomes what he calls "transcendental illusion", which is the application of transcendental reasoning not rooted in sensibility. He calls this metaphysics, and he focuses on the soul (the topic at hand here), the cosmos, and God as the three topics of metaphysics in his Transcendental Dialectic.
I think that in general, discussion about epistemology with regard to AI would be better if people started at least from Kant (either building on his ideas or critical of them), as his CPR really shaped a lot of the post-Enlightenment views on epistemology that a lot of us carry with us without knowing. In my opinion, AI is vulnerable to a criticism that empiricists like Hume applied to people (viewing people as "bundles of experience" and critiquing the idea that we can create new ideas independent of our experience). I do think that AI suffers from this problem, as estimating a generative probability distribution over data means that no new information can be created that is not simply a logically ungrounded combination of previous information. I have not read any discussion of how Kant's view of our ability to make new information (application of categories grounded by our perception) might influence a way to make an actual thinking machine. It would be fascinating to see an approach that combines new AI approaches as the way the machine perceives information and then combines it with old AI approaches that build on logic systems to "reason" in a way that's grounded in truth. The problem with old AI is that it's impossible to model everything with logic (the failure of logical posivitism should have warned them), however it IS possible to combine logic with perception like Kant proposed.
I hope this makes sense. I've noticed a lack of philosophical rigor around the discussion of AI epistemology, and it feels like a lot of American philosophy research, being rooted in modern analytical tradition that IMO can't adapt easily to an ontological shift from human to machine as the subject, hasn't really risen to the challenge yet.
Remember, this is about Cartesian duality (mind-brain duality), so the key question here is not whether a brain exists, but whether the mind exists independently of it.
Other aspects of ANN that show that Gödel doesn’t apply is that they are not formal systems. Formal system is a collection of defined operations. The building blocks of ANN could perhaps be built into a formal system. Petri nets have been demonstrated to be computationally equivalent to Turing machines. But this is really an indictment on the implementation. It’s the same as using your PC, implementing a formal system like its instruction set to run a heuristic computation. Formal system can implement informal systems.
I don’t think you have to look at humans very hard to see that humans don’t implement any kind of formal system and are not equivalent to Turing machines.
Incorrect.
The comment above confuses some concepts.
Perhaps this will help: consider a PRNG implemented in software. It is an algorithm. The question of the utility of a PRNG (or any algorithm) is a separate thing.
On the level where the learning is done and knowledge is represented in these networks there is no evidence anyone really understands how it works.
I suspect maybe at that level you can think of it as an algorithm with unreliable outputs. I don’t know what that idea gains over thinking it’s not algorithmic and just a heuristic approximation.
https://xlinux.nist.gov/dads/HTML/heuristic.html
https://xlinux.nist.gov/dads/HTML/millerRabin.html
https://en.wikipedia.org/wiki/Miller%E2%80%93Rabin_primality...
It is hard to assess the comment above. Depending on what you mean, it is incorrect, inaccurate, and/or poorly framed.
The word “really” is a weasel word. It suggests there is some sort of threshold of understanding, but the threshold is not explained and is probably arbitrary. The problem with these kinds of statements is that they are very hard to pin down. They use a rhetorical technique that allows a person to move the goal posts repeatedly.
This line of discussion is well covered by critics of the word “emergence”.
Heuristic or not, AI is still ultimately an algorithm (as another comment pointed out, heuristics are a subset of algorithms). AI cannot, to expand on your PRNG example, generate true random numbers; an example that, in my view, betrays the fundamental inability of an AI to "transcend" its underlying structure of pure algorithm.
2. AI just means “non human” intelligence. An AI system (of course) can incorporate “real world” sources of entropy such as sensors. This is already done.
Physicists like to use mathematics for modeling the reality. If our current understanding of physics is fundamentally correct, everything that can possibly exist is functionally equivalent to a formal system. To escape that, you would need some really weird new physics. Which would also have to be really inconvenient new physics, because it could not be modeled with our current mathematics or simulated with our current computers.
So if Gödel tells us that either formal systems will be consistent and make statements they cannot prove XOR be inconsistent and therefore unreliable, at least to some degree, then surely informal systems will, at best, be the same, and, at worst, be much worse?
The desirable property of formal systems is that the results they produce are proven in a way that can be independently verified. Many informal systems can produce correct results to problems without a known, efficient algorithmic solution. Lots of scheduling and packing problems are NP-complete but that doesn’t stop us from delivering heuristic based solutions that work good enough.
Edit: I should probably add that I’m pretty rusty on this. Godels theorem tells ua that if a formal system is consistent, it will be incomplete. That is, there will be true statements that cannot be proven in the system. If the system is complete, that is, all true/false statements can be proven, then the system will be incomplete. That is you can prove contradictory things in the system.
AI we have now isn’t really either of these. It’s not working to derive truth and falsehood from axioms and a rule system. It’s just approximating the most likely answers that match its training data.
All of this has almost no relation to the questions we’re interested in like how intelligent can AI be or can it attain consciousness. I don’t even know that we have definitions for these concepts suitable for beginning a scientific inquiry.
As for humans, there is no way you can look at the behavior of a human and know for certain it is not a Turing machine. With a large enough machine, you could simulate any behavior you want, even behavior that would look, on first observation, to not be coming from a Turing machine; this is a form of the halting problem. Any observation you make that makes you believe it is NOT coming from a Turing machine could be programmed to be the output of the Turing machine.
This is not exactly true, depending on what you mean by behavior. There are mathematical functions we know for a fact are not computable by a Turing machine, no matter how large. So a system that "behaves" like those functions couldn't be simulated by a TM. However, it's unclear whether such a system actually could exist in physical reality - which gets right back to the discussion of whether thinking is beyond Turing completeness or not.
It is a self-delimiting program. It is an algorithm in the most basic sense of the definition of “partial recursive function” (total in this case) and thus all known results of computability theory and algorithmic information theory apply.
> Formal system is a collection of defined operations
Not at all.
> I don’t think you have to look at humans very hard to see that humans don’t implement any kind of formal system and are not equivalent to Turing machines.
We have zero evidence of this one way or another.
—
I’m looking for loopholes around Gödel’s theorems just as much as everyone else is, but this isn’t it.
oh, cogito existo sum! checkmate theists!
dude, the simulation hypothesis does not mean things don't exist, it means they don't necessarily exist in the way you have, rather unimaginatively, imagined, and you have no way to tell.
and Occam's Razor does not solve the problem.
also, falsifiability lacks falsifiability.
don't make up complex axioms and then believe in them on faith.
Occam's razor: if the universe can be simulated, then whatever form the simulation takes is the simpler explanation for the turtles all the way down.
if the universe can't be simulated, explain how you know that.
To pull from the relevant part of Hofstadter’s incredible I am a Strange Loop (a book also happens to more rigorously invoke Gödel for cognitive science):
And this is our central quandary. Either we believe in a nonmaterial soul that lives outside the laws of physics, which amounts to a nonscientific belief in magic, or we reject that idea, in which case the eternally beckoning question "What could ever make a mere physical pattern be me?”
After all, a phrase like "physical system" or "physical substrate" brings to mind for most people… an intricate structure consisting of vast numbers of interlocked wheels, gears, rods, tubes, balls, pendula, and so forth, even if they are tiny, invisible, perfectly silent, and possibly even probabilistic. Such an array of interacting inanimate stuff seems to most people as unconscious and devoid of inner light as a flush toilet, an automobile transmission, a fancy Swiss watch (mechanical or electronic), a cog railway, an ocean liner, or an oil refinery. Such a system is not just probably unconscious, it is *necessarily* so, as they see it. This is the kind of single-level intuition so skillfully exploited by John Searle in his attempts to convince people that computers could never be conscious, no matter what abstract patterns might reside in them, and could never mean anything at all by whatever long chains of lexical items they might string together.
Highly recommend it for anyone who liked Gödel, Escher, Bach, but wants more explicit scientific theses! He basically wrote it to clarify the more artsy/rhetorical points made in the former book.It's accurate. But it feels really weird.
It's not uncommon for great scientists to be totally out of their depth even in nearby fields, and not realize it. But this isn't the hard part of either computability or philosophy of mind.
Penrose is an authority in some fields of theoretical physics, but that doesn't give any value to what he has to say on consciousness or AI.
On that topic, he has clearly adopted an unscientific approach: he wants to believe the soul exists and is immaterial and seeks evidence for it.
Simultaneously, actual dualists flock to his theories because they think association with Penrose lends credibility to their religious stuff.
What's more, whatever you like to call the transoforming of information into thinked information by definition can not be a (mathematical) function, because it would require all people to process the same information in the same way and this is plainly false
No this isn't the checkmate you think it is. It could still be a mathematical function. But every person transforming information into "thinked information" could have a different implementation of this function. Which would be expected as no person is made of the same code (DNA).
Note that using words like "function" and "mathematical" are more the biases of computer science/Penrose while philosophy of the mind & psychology has more typically used slightly different ideas of "attitudes" and "events" to guide the discussion. I don't think this really radically shifts many central disputes, except (which Penrose might view as a critical "except") for all the funny business quantum mechanics / quantum computation can potentially bring in and the (undisputed) physicality of our brains and (also undisputed) lack of understanding of many details of biological computation.
It's not entirely clear, though, that the universe is deterministic- our best experiments suggest there is some remaining and relevant nondeterminism.
Turing machines, Goedel incompleteness, Busy Beaver Functions, and (probably) NP problems don't have any relevance to simulating complex phenomena or hard problems in biology.
What you said sounds good, but I don't think it's philosophically robust.
read this as: literally creating gravity by simulating it hard enough.
I'm not really arguing with you, i just think if i simulate entropy (entropic processes, "CSRNG", whatever) on my computer ...
That's not obviously true, especially given how the more we keep digging into physics, the more everything seems to be "just information".
We don't know if consciousness is computable, because we don't know what consciousness is.
There are suggestions it isn't even local, never mind Turing-computable.
It’s possible that we haven’t found a way to express your thinking function digitally, which I think is true, but I have a feeling that the complexity of thought requires the analog-ness of our brains.
I don’t think that’s a certainty.
This is a big assumption. I'm not saying it's wrong, but I am saying it's not reasonable to just handwave and assume that it's right.
FWIW human brain does indeed consume a lot of energy, accounting for over 20% of our body metabolism. We don't know how to attribute specific watts consumed to specific thoughts because we don't understand the functioning of the brain enough, but there's no obvious reason why it shouldn't be possible.
As well, perhaps, worth noting that because a subset of the observable universe is performing some function, then it is an assumption that there is some finite or digital mathematical function equivalent to that function; a reasonable assumption but still an assumption. Most models of the quantum universe involve continuously variable values, not digital values. Is there a Turing machine that can output all the free parameters of the standard model?
Sure, just hard code them.
> As well, perhaps, worth noting that because a subset of the observable universe is performing some function, then it is an assumption that there is some finite or digital mathematical function equivalent to that function; a reasonable assumption but still an assumption. Most models of the quantum universe involve continuously variable values, not digital values.
Things seem to be quantised at a low enough level.
Also: interestingly enough quantum mechanics is both completely deterministic and linear. That means even if it was continuous, you could simulate it to an arbitrary precision without errors building up chaotically.
(Figuring out how chaos, as famously observed in the weather, arises in the real world is left as an exercise to the reader. Also a note: the Copenhagen interpretation introduces non-determinism to _interpret_ quantum mechanics but that's not part of the underlying theory, and there are interpretations that have no need for this crutch.)
Some things. But others are very much not: in particular, space and time are not quantized, and in fact are not even quantizable. A theory in which there is some discrete minimal unit of space (or of spacetime) is trivially incompatible with special relativity, so it is incompatible with quantum mechanics (QFT, specifically).
This is easy to see from the nature of the Lorenz transformation: if two objects are at a distance D = n*min for some observer, they will be at a distance D' = n*min*gamma for some other observer, where gamma is always < 1 for an o server moving at a higher speed in a direction aligned with the two objects. So the distance for that second observer will be a non-integer multiple of the minimum distance, so your theory is no longer quantized.
Note that this is separate from the problem with GR-QFT inconsistencies. All of our current theories are based on and only work if spacetime is continuous. While it's not impossible that a new theory with quantized spacetime could exist and work, it's not at all required.
The one thing about spacetime that we do believe might be quantizable, and would have to be quantized for GR and QFT to be compatible, is the curvature of spacetime. But even if spacetime can only be curved in discrete quanta, that would not mean that position would be quantized.
That would be super lucky if possible - almost all reals are not computable. How would we initialize or specify this Turing machine? Going to use non-constructive methods?
Given how quickly reality unfolds, it's a bit of a stretch to assume that "simulate it to arbitrary precision" means "computable in a digital representation in real time." I mean, if we have Turing machines that did each computation step n in 2^{-n} time.
The standard model doesn't use arbitrary reals. All the parameters are rational numbers with finite precision.
Obviously, your Turing machine can only hard code a finite amount of information about the parameters, eg whatever finite prefix of their decimal expansion is known.
Btw, the speed of light is one of those parameters that's 'known' with absolute precision thanks to some clever definitions. We can do similar tricks with many of the other parameters, too.
It likely is a fact, but we don't really know what we mean by "think".
LLMs have illuminated this point from a relatively new direction: we do not know if their mechanism(s) for language generation are similar to our own, or not.
We don't really understand the relationship between "reasoning" and "thinking". We don't really understand the difference between Kahneman's "fast" and "slow" thinking.
Something happens, probably in our brains, that we experience and that seems causally prior to some of our behavior. We call it thinking, but we don't know much about what it actually is.
AIs are not going to be like humans because they will have perfect recall of a massive database of facts, and be able to do math well beyond any human brain.
The interesting question to me is, when will we be able to give AI very large tasks, and when will it to be able to break the tasks down into smaller and smaller tasks and complete them.
When will it be able to set its own goals, and know when it has achieved them?
When will it be able to recognize that it doesn't know something and do the work to fill in the blanks.
I get the impression that LLMs don't really know what they are saying at the moment, so don't have any way to test what they are saying is true or not.
I think you’re right, LLMs have demonstrated that relatively sophisticated mathematics involving billions of params and an internet full of training data is capable of some truly, truly, remarkable things. But as Penrose is saying, there are provable limits to computation. If we’re going to assume that intelligence as we experience it is computable, then Gödel’s theorem (and, frankly, the field of mathematics) seems to present a problem.
Humans have a special thingy that makes the consciousness Computers do not have the special thingy Therefore Computers cannot be consciousness.
But Dualism gets you laughed at these days so Dualists have to code their arguments and pretend they aren't into that there Dualism.
Penrose's arguments against AI has always felt to me like special pleading that humans (or to stretch a bit further, carbon based lifeforms) are unique.
If this were to be true, it would follow that computers as we build them today would fundamentally not be able to match human problem-solving. But it would not follow, in any way, that it would be impossible to build "hyper computers" that do. It just means you wouldn't have any chance of getting there with current technology.
Now, I don't think Penrose's arguments for why he thinks this is the case are very strong. But they're definitely not mystical dualistic arguments, they're completely materialistic mathematical arguments. I think he leans towards an idea that quantum mechanics has a way of making more-than-Turing computation happen (note that this is not about what we call quantum computers, which are fully Turing-equivalent systems, just more efficient for certain problems), and that this is how our brains actually function.
That was my understanding on Penrose's position as well which is just a "Consciousness of the Gaps" argument. As we learn more about quantum operations the space for Consciousness as a special property of humans disapears.
I just watched an interview where he made that exact statement nearly word for word.
His only argument is that it is not computable, not that it’s not physical. He does think the physical part involves the collapse of the wave function due to gravity, and that somehow the human brain is interacting with that.
So to produce conciseness in his view, you’d need to construct something capable of interacting with the quantum world the same way he believes organic brains do (or something similar to it). A simulation of the human brain wouldn’t do it.
what if we are?
and our brain is the "billion parameter model", continuously "trining", that takes input and spits out output
I understand we as a species might be reluctant to admit that we are just matter and our thinking/consciousness is just electricity flowing
btw, I'm not selling anything :)
On the contrary, we have 86B neurons in the brain, the weighting of the connections is the important thing, but we are definitely 'running' a model with many billions of parameters to produce our output.
The theory by which the brain mainly works by predicting the next state is called predictive coding theory, and I would say that I find it pretty plausible. At the very least, we are a long way from knowing for certain that we don't work in this way.
However, this doesn't mean in any way that an LLM might not produce the same or even superior output than a human would in certain very useful circumstances. It just means it functions fundamentally differently on the inside.
Obviously the brain isn't running an exact implementation of the attention paper, and your point about how the brain is more malleable than our current llms is a great point, but that just proves they aren't the same. I fully expect that future architectures will be more malleable, if you think that such hypothetical future architectures will be fundamentally different from the current ones then we agree..
The neurons (cells) in even a fruit flies brain are orders of magnitude more complex than the "neurons" (theoretical concept) in a neural net.
> the weighting of the connections is the important thing
In a neural net, sure.
In a biological brain, many more factors are important: The existence of a pathway. Antagonistic neurotransmitters. NT re-incorporation. NT-binding sensitivity. Excitation potential. Activity of Na/K channels. Moderating enzymes.
Even what we last ate or drank, how rested, old, hydrated, we are, when our lats physical activity took place, and all the interactions prior to an input influence how we analyse and integrate it.
> but we are definitely 'running' a model with many billions of parameters to produce our output.
No, we are very definitely not. Many of our mental activities have nothing to do with state prediction at all.
We integrate information.
We exist as a conscious agent in the world. We interact, and by doing so change our own internal state alongside the information we integrate. We are able to, from this, simulate our own actions and those of other agents, and model the world around us, and then model how an interaction with that world would change the model.
We are also able to model abstract concepts both in and outside the world.
We understand what concepts, memories, states, and information mean both as abstract concepts and concrete entities in the universe.
We communicate with other agents, simultaneously changing their states and updating our modeling of their internal state (theory of the mind, I know that you know that I know, ...)
We filter, block, change, and create information.
And of course we constantly learn and change the way we do ALL OF THIS, consciously and subconsciously.
> At the very least, we are a long way from knowing for certain that we don't work in this way.
If the process in the brain is modellable at all, then it is certainly a model with at a minimum many billions of parameters. Your list of additional parameters if anything supports that rather than arguing against it. If you want to argue with that contention, I think you need to argue that the process isn't modellable, which if you want to talk about burden of proof, would place a huge burden on you. But maybe I misunderstood you. I thought you were saying that it's ludicrous to say we're using as many as billions of parameters, but perhaps you're trying to say that billions is obviously far too small, in which case I agree.
My second point, which is that there's a live theory that prediction may be a core element of our consciousness was intended as an interesting aside, I don't know how it will stand the test of time and I certainly don't know if its correct or not, I intended only to use it to prove that the things you seem to think are obvious are not in fact obvious to everyone.
For example, that big list of things that you are using as an argument against prediction doesn't work at all because you don't know whether they are implemented via a predictive process in the brain or not.
It feels that rather than arguing against modellability or large numbers of parameters or prediction you're arguing against the notion that the human brain is exactly an llm, which is an idea so obviously true I don't think anyone actually disagrees with it.
> perhaps you're trying to say that billions is obviously far too small, in which case I agree.
No, it doesn't, and I don't.
The processes that happen in a living brain don't just map to "more params". It doesn't matter how many learnable parameters you have...unless you actually change the paradigm, an LLM or similar construct is incapable of mapping a brain, period. The simple fact that the brains internal makeup is itself changeable, already prevents that.
> prediction may be a core element of our consciousness
No it isn't, and it's trivially easy to show that.
Many meditative techniques exist where people "empty their mind". They don't think or predict anything. Does that stop consiousness? Obviously not.
Can we do prediction? Sure. Is it a "core element", aka. indispensable for consciousness? No.
But language processing is just one subset of human cognition. There are other layers of human experience like sense-perception, emotion, instinct, etc. – maybe these things could be modeled by additional parameters, maybe not. Additionally, there is consciousness itself, which we still have a poor understanding of (but it's clearly different from intelligence).
So anyway, I think that it's reasonable to say that LLMs implement one sub-set of human cognition (the part that has to do with how we think in language), but there are many additional "layers" to human experience that they don't currently account for.
Maybe you could say that LLMs are a "model distillation" of human intelligence, at 1-2 orders of magnitude less complexity. Like a smaller model distilled from a larger one, they are good at a lot of things but less able to cover edge cases and accuracy/quality of thinking will suffer the more distilled you go.
We tend to equate "thinking" with intelligence/language/reason thanks to 2500 years of Western philosophy, and I believe that's where a lot of confusion originates in discussions of AI/AGI/etc.
[1]: https://medicine.yale.edu/lab/colon-ramos/overview/#:~:text=...
Related is the platonic representation hypothesis where models apparently converge to similar representations of relationships between data points.
https://phillipi.github.io/prh/ https://arxiv.org/abs/2405.07987
To put this another way, I think that you can say that much of our own intelligence as humans is embedded in the sum total of the language that we have produced. So the intelligence of LLMs is really our own intelligence reflected back at us (with all the potential for mistakes and biases that we ourselves contain).
Edit: I fed Claude this paper, and "he" pointed out to me that there are several examples of humans developing accurate conceptions of things they could never experience based on language alone. Most readers here are likely familiar with Helen Keller, who became an accomplished thinker and writer in spite of being blind and deaf from infancy (Anne Sullivan taught her language despite great difficulty, and this Keller's main window to the world). You could also look at the story of Eşref Armağan, a Turkish painter who was blind from birth – he creates recognizable depictions of a world that he learned about through language and non-visual senses).
You’re arguing that we know artificial reasoning exists because we are capable of reasoning. This presupposes that reasoning is computable and that we ourselves reason by computation. But that’s exactly what Penrose is saying isn’t the case - you’re saying we’re walking Turing machines, we’re intelligent, so we must be able to effectively create copies of that intelligence. Penrose is saying that intelligence is poorly defined, that it requires consciousness which is poorly understood, and that we are not meat-based computers.
Your last question misses the point completely. “If we are something else, then out CoT won’t be computable…” it’s like you’re almost there but you can’t let go of “we are meat-machines, everything boils down to computation, we can cook up clones”. Except, “basic computability theory” says that’s not even wrong.
> Personally, I prefer to think about it in terms of basic computability theory:
Gödel's incompleteness theorem applies to computing. I'm sure you're familiar with the Halting Problem. Gödel's applies to any axiomatic system. The trouble is, it's very hard to make a system without axioms. They are sneaky and it's different than any logic you're probably familiar with.And don't forget Church-Turing, Gödel Numbers, and all the other stuff. Programming is math and Gödel did essential work on the theory of computation. It would be weird NOT to include his work in this conversation.
> are we at most just Turing machines or something else?
But this is a great question. Many believe no. Personally I'm unsure, but lean no. Penrose is a clear no but he has some wacky ideas. Problem is, it's hard to tell a bad wacky idea from a good wacky idea. Rephrasing Clarke's Second Law: Genius is nearly indistinguishable from insanity. The only way to tell is with time.But look into things like NARS and Super Turing machines (Hypercomputation). There's a whole world of important things that are not often discussed when it comes to the discussion of AGI. But for those that don't want to dig deep into the math, pick up some Sci-Fi and suspend your disbelief. Star Trek, The Orville and the like have holographic simulations and I doubt anyone would think they're conscious, despite being very realistic. But The Doctor in Voyager or Isaac in The Orville are good examples of the contrary. The Doctor is an entity you see become conscious. It's fiction, but that doesn't mean there aren't deep philosophical questions. Even if they're marked by easy to digest entertainment. Good stories are like good horror, they get under your skin, infect you, and creep in
Edit:
I'll leave you with another question. Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?
Another question. How do you go about detecting whether consciousness has arisen?
> Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?
A Turing machine can in principle beat the Turing test. But so can a giant lookup table, if there's any finite time limit (however generous) placed on the test.
The 'magic' would in the implementation of the table (or the Turing machine) into something that can answer in a reasonable amount of time and be physically realised in a reasonable amount of space.
Btw, that's an argument from Scott Aaronson's https://www.scottaaronson.com/papers/philos.pdf
> There's no evidence that hypercomputation
I'm not the right person to ask this tbh. As far as I know it doesn't even well in theory. (Though I don't think they would make it useless to study) > can in principle beat the Turing test
What's the relevance of the Turing test? It's been beaten for over half a century.I would be very interested if you have any sources on anyone beating the Turing test in anything close to Turing's original adversarial formulation.
ELIZA
PARRY
Eugene Goostman
Mitsuku (Kuki AI)
Or see the Loebner Prize where judges aren't just average people but experts (so harder).The Turing Test was never about intelligence or thinking, even said so by Turing himself. He made the test because those words are too vague. He specifically wanted to shift the conversation to task completion, since that is actually testable. Great! That's how science works! (Looking at you String Theory...) But science also progresses. We know more about what these things intelligence and thinking are. These still are not testable in a concrete sense, but we have better proxies than the Turing Test now.
The problem is that knowledge percolates through society slowly and with a lag. We've advanced a lot since then. I'm sure you are likely willing to believe that most LLMs these days can pass it. The Turing Test was a great start. You gotta start somewhere. But to think we came up with good tests at the same time we invented electronic computers, should give you surprise. Because it would require us to have been much smarter then than we are now.
Eliza never won that adversarial version even against layman.
In what sense did Eliza ever 'win' any Turing test?
> I'm sure you are likely willing to believe that most LLMs these days can pass it.
No, I haven't seen any evidence of that.
To repeat: I am interested in evidence that any non-human can beat the Turing test in the original form given in Turing's paper, where you have the judge (human), and two contestants A and B. One of the contestants is a computer, one is a human. Everyone can see what everyone else is writing, and the human contestant can help the human judge. (But the computer can try to fake that 'helping', too.)
Turing specifically wrote: "The object of the game for the third player (B) is to help the interrogator."
I can believe that Eliza has occasionally fooled some random humans, but I can't believe Eliza managed to fool anyone when a third party was around to point out her limitations. (Especially since Eliza ain't smart enough to retaliate and fabricate some 'obvious computer limitations' to accuse the third party of.)
Most LLMs today still have some weaknesses that are easy to point out, if you let your contestants (both kinds) familiarise themselves with both humans and the LLMs in question at their leisure before the test starts.
Just for fun, I just tried out Kuki AI, and it's not going to fool anyone who actually wants to uncover the AI through adversarial cross-examination.
The chat excerpt giving of 'Eugene Goostman' in https://en.wikipedia.org/wiki/Eugene_Goostman also suggests that it would fall apart immediately in an adversarial setting with the full three participants.
However, I do agree that we have made progress and that today's LLMs could hold up a lot longer in this harsher setting than anything we had before. Especially if you fine-tuned them properly to remove telltale signs like their inability to swear or their constant politeness.
> But to think we came up with good tests at the same time we invented electronic computers, should give you surprise.
I never claimed the Turing test is the best test ever, nor even that it's particularly good. I was saying that it hasn't been beaten in its original form.
For example, the Turing test isn't really a fine grained benchmark that lets you measure and compare model performance two multiple decimal places. Nor was it any good as a guideline for how to improve our approaches.
> In addition, there were two one-time-only prizes that have never been awarded. $25,000 is offered for the first program that judges cannot distinguish from a real human and which can convince judges that the human is the computer program. $100,000 is the reward for the first program that judges cannot distinguish from a real human in a Turing test that includes deciphering and understanding text, visual, and auditory input. The competition was planned to end after the achievement of this prize.
In addition to the detectability problem, I wrote in the adjacent comment, this question can be further refined.
A Turing machine is an abstract concept. Do we need to take into account material/organizational properties of its physical realization? Do we need to take into account computational complexity properties of its physical realization?
Quantum mechanics without Penrose's Orch OR is Turing computable, but its runtime on classical hardware is exponential in, roughly, the number of interacting particles. So, theoretically, we can simulate all there is to simulate about a given person.
But to get the initial state of the simulation we need to either measure the person's quantum state (thus losing some information) or teleport his/her quantum state into a quantum computer (the no-cloning theorem doesn't allow to copy it). The quantum computer in this case is a physical realization of an abstract Turing machine, but we can't know its initial state.
The quantum computer will simulate everything there are to simulate, but the interaction of a physical human with the initial state of the Universe via photons of the cosmic microwave background. Which may deprive the simulated one of "free will" (see "The Ghost in the Quantum Turing Machine" by Scott Aaronson). Or maybe we can simulate those photons too, I'm not sure about it.
Does all of it have anything to do with consciousness? Yeah, those are interesting questions.
[0] See the Taylor Series expansion. This is an only physics joke
"Thinked information" is a colour not an inherent property of information. The fact that information has been thought is like the fact it is copyrighted. It is not something inherent to the information, but a property of its history.
Where does this question come from? Especially where does the 'smaller' requirement come from?
We think, that is a fact.
Therefore, there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function.
Now, the question is: can we create a smaller function capable of performing the same feat?
If we assume that that function is computable in the Turing sense then, kinda yes, there are an infinite number of turing machines that given enough time will be able to produce the expected results. Basically we need to find something between our own brain and the Kolmogorov complexity limit. That lower bound is not computable, but given that my cats understands when we are discussing to take them to the vet then... maybe we don't really need a full sized human brain for language understanding.
We can run Turing machines ourselves, so we are at least Turing equivalent machines.
Now, the question is: are we at most just Turing machines or something else? If we are something else, then our own CoT won't be computable, no matter how much scale we throw at it. But if we are then it is just matter of time until we can replicate ourselves.