Although I more or less agree with you, in general, how complex would the AI have to be before you can consider it aware?

I categorically reject the notion that LMMs (Large Markov Models) can ever be aware or intelligent. Comparing weighted next-word-engines to feeling, thinking, aware beings is insulting.

Are you intentionally echoing that scene from starship troopers where the pundit makes the same claim about 'smart bugs'?

I don't base my ideas on what constitutes humanity, intelligence, or sentience on hundred million dollar fiction.

Alright then you should also consider not basing it off of whether you find the concept insulting. It doesn't seem to be the strongest rebuttal available.

That it's insulting has nothing to do with my reasoning. You have it backwards.

What's an LMM? I've never seen the term.

> Comparing weighted next-word-engines to feeling, thinking, aware beings is insulting

Why is it reasonable to be so reductionist about e.g. GPT-4 but not be so reductionist about a biological brain? E.g., why can't I say that your brain is nothing but a bunch of biological neurons trained using its input and intialized based on your genetics? It's equally true, and equally missing the point.

I think that machine learning probably can produce something akin to a brain, but LLMs are not really it even if they use the digital equivalent of a neuron. As much as I understand what I read about LLMs they really seem to be descendants of Markov chains. I think they are valuable and can go a long way, but LLMs themselves will not be "it". I think that we will get to a ceiling with them within 10 years if we will not think about something else. I think the ceiling can be made pretty high though.

However most probably in 10 years we will all laugh how all of our predictions missed by a long shot.

LMM = Large Markov Model. I use that term because models like GPT-4 and friends are for all intents and purposes Markov chains with more data, more compute, some lossy compression, and a bit of nearest neighbor search. Next-word-engines.

> why can't I say that your brain is nothing but a bunch of biological neurons trained using its input and intialized based on your genetics?

Because we don't think one word at a time, and we don't restart from scratch for every subsequent word.

GPT is not Markovian; it has state.

Then it's a markov-like with state. Or as I've taken to calling them lately Markov+state. (I couldn't resist, sorry.)

A truck towing a trailer isn't just a car because it pivots in the middle and has more wheels. It's fundamentals of operation are still closer to a car or truck without trailer than a bicycle.

Humans can form thoughts and get to mostly correct answers even as a gut feeling, and the language to explain why/how need not even be present. We don't form thoughts one word at a time.

No it's not Markov-like. GPT models are not Markov processes by definition. They take into account all previous words in the sequence when generating the next word. They have a type of memory in the form of attention to previous states.

>Because we don't think one word at a time

In what sense does an LLM think one word at a time that doesn't also apply to a person typing at a keyboard? I'm typing one word at a time right now, I assume you aren't about to declare me a markov chain. When I read my brain presumably ingests one word at a time (not sure if it's one exactly, but it can't be much more than one). It is of course true that I have some notion of what I'm going to say before I right the first word, but seemingly so does an LLM.

If it was truly thinking one word at a time, it wouldn't be able to consistently use 'an' vs 'a' correctly, for example.

>we don't restart from scratch for every subsequent word.

LLMs don't restart from scratch for every word, via the attention heads they can look back through the entire context. Otherwise the memory required for inference wouldn't scale with the context length.

> In what sense does an LLM think one word at a time that doesn't also apply to a person typing at a keyboard?

Because you already have the thought formed before you started typing.

> When I read my brain presumably ingests one word at a time (not sure if it's one exactly, but it can't be much more than one)

And these models ingest many words at a time, up to the context length.

Seems to me it's fundamentally inverted from how we operate,both input and output.

>Because you already have the thought formed before you started typing.

Can you prove that GPT-4 doesn't? Clearly there is a sense in which thinks more than one word ahead, since as I mentioned above it would not otherwise be able to use 'a' vs 'an' correctly.

As far as I am aware, exactly to what extent these models have determined what tokens will be generated before they produce anything is an open question in mechanistic interpratability research. I would be very interested if you knew of some work that answers this question empirically.

Imagination. When the AI is capable not just of describing something but of extrapolating the rest of its existence in order to complete a task. For instance, any puppy understands the basics of information flow. A dog can lie. A dog can be fed by owner A and then tell owner B that A forget. The puppy can imagine the mental states of both owners, extrapolate that they have not coordinated the meal delivery today, identify that information gap, then leverage that understanding into getting two meals instead of one. That's the level of complexity I expect before an AI can be said to be aware of its environment.

And moreover, this is more than a philosophical difference. As our understandings of AI and neuroscience improve we can now talk about "world models" as something almost tangible and constructible. Complex animal nervous systems build and use embodied world models to interpret the senses and allow for tractable future prediction and action generation. Yann Lecunn's paper on AGI from August 2022 gets into defining computational equivalents, and multimodal AI models like PaLM-E are taking first steps toward explicitly tying LLM-style text prediction with collections of perceptual observations about the environment.

So it's not that artificial systems cannot, as a category, have models of their environment and the actions they can take within it, this "cargo cult AI" concept comes up when people jump the gun and see those capabilities in much simpler systems, even including ChatGPT-4. And it's disheartening to see narrative on the subject driven more and more by people who have not taken the time to learn about the subject matter, for all the interest they seem to show in talking about it.

I think with reinforcement learning and adversarial play you don't need it to understand for it to have unintended consequences and "break out of it's cage"

Surprising behaviors We’ve shown that agents can learn sophisticated tool use in a high fidelity physics simulator; however, there were many lessons learned along the way to this result. Building environments is not easy and it is quite often the case that agents find a way to exploit the environment you build or the physics engine in an unintended way.

Emergent tool use from multi-agent interaction https://openai.com/research/emergent-tool-use

Here's an example where using a LLM enhanced a reinforcement algo performance.

https://arxiv.org/abs/2302.06692

The mouse is driven by survival. The AI is driven by best answer.

An interesting fallacy I see emerging is 'this system doesn't have animal-style intelligence, therefore it is lesser'

The entire point of neural nets is removing the biases of animal intelligence and letting the computer brute force solutions during training. We are now learning the early stages of the amazing things that can lead to.

From an outside perspective, it's not crazy to argue that symbolic reasoning is a crutch that animals developed since they are imperfect data collectors and limited by their wetware compute resources. Those constraints may not end up being meaningful for these models (to be clear, I don't even mean Transformers per se, we are likely to continue developing really clever model architectures that may look completely difference from what we know today).

I am 100% on board with the substance of you argument, but I'd encourage you to really think critically about the implications.

Why do you assume that the animal is also not brute forcing the problem? Brute forcing is the basis of evolution. The algorithm running inside the head of the mouse is the survivor of a million iterative generations as the species brute-forced the entire "get to the food" survival problem. I encourage those touting computers as something new to comprehend the concept of deep time, that no matter how many times you run simulations, the natural world has almost certainly run more.

> Brute forcing is the basis of evolution.

The goal function of animal intelligence is very different from the goal function of model training.

> The algorithm running inside the head of the mouse is the survivor of a million iterative generations as the species brute-forced the entire "get to the food" survival problem.

That’s my entire point. There are problems beyond “get the food.” Said differently, the fact that computers aren’t good at “get the food” is not a strong criticism of computers. It just means their goal functions are different.

I didn’t say computers are “better,” just that they have a fundamentally different approach to problem solving that may end up being better. It will probably end up being complimentary! This isn’t either or; AI is a tool built by humans to expand their own capabilities.

> I encourage those touting computers as something new to comprehend the concept of deep time, that no matter how many of times you run simulations, the natural world has almost certainly run more.

And yet no animal figures out how to evolve wheels to move faster.

The entire point I’m making is that, sure it’s hard to make computers do certain things (like walk on two legs), but there are plenty of different avenues to achieve things.

I encourage evolution maximalists to think a lot more about how engineered and complex the world around them is.

> Brute forcing is the basis of evolution. The algorithm running inside the head of the mouse is the survivor of a million iterative generations as the species brute-forced the entire "get to the food" survival problem.

Here is an interesting corollary: the same is the case with humans. Evolution being what it is, whatever makes the mouse brain tick, and whatever makes our brains tick, must have been relatively easy for evolution to stumble upon and scale up, and it must have been delivering continuous improvements to survival. Otherwise, natural selection wouldn't be able to home in on the solution.

In short: the fundamentals of intelligence have to be something simple enough to be discoverable by accident.

This is one of the main reasons I believe that LLMs aren't unrelated to how our own brains work. We tried to make a simple yet generic system, somewhat emulating what we see in nature, and then we kept scaling it up, until it surprised us with suddenly gaining capabilities we consider to be advanced cognitive functions. Just how many such structures could be there in nature, so that evolution and AI researchers stumbled on two entirely unrelated ones? It's too much of a coincidence.

Here is me putting GPT-4 in a vaguely described maze, giving it an underspecified goal, making it a player in a game, myself acting as DM:

https://cloud.typingmind.com/share/c0a68cb2-5f59-4e83-b383-b...

I don't think GPT-4 is memorizing solutions here. I can see extrapolation and some degree of imagination in there, but of course you could say it's memorizing higher-level patterns. At some point though, you have to consider the mouse is also running hard-wired high-level patterns, and ask yourself if the difference here is really a matter of kind, or just degree.

A text maze. It is effectively brute-forcing a choose-your-own-adventure book. An aware mind would escape the maze by just skipping ahead to read the good parts of the book. That's what I did.

Not necessarily. It depends on what the mind wants.

You skipped ahead because you were impatient - or efficient - and didn't value the game anywhere near as much as the reward. GPT-4 just played along as instructed.

Now, if you were a player in a real RPG game, you'd probably play along with the story DM gave - because you'd care about the process more than about getting to the end.