So true, motivation is at the heart of decision making, and if you don’t have any wants/needs, you are a blank slate. Unlike animals, ‘human’ drives are mostly styled or corrupted by culture. So AI has to confront both biological drive and cultural conditioning.

Isn’t this just basic reinforcement learning? We’re not too many steps away from having an AI equipped with a good language model and a reinforcement learning mechanism to be let loose on the internet.

I also have this question. Is the RL MDP actually encoding cause and effect? Or just learning (bidirectional) correlations between states and actions?

I wonder if Pearl thinks that RL replicates his do-calculus under the hood, or if that's an innovation we're missing.

The difference between RL and this "want" is how the goals are constructed. Currently in computational systems, it is the designer that creates the ontology of what an agent interacts with. What that means for ML, RL and evolutionary algorithms is that the agent has a predefined goal and predefined devices to accomplish said goal (the agent's sensors and actuators must explicitly be programmed in order to utilize them).

Biological agents have are seemingly able to derive goals from both internal and external signals, and are able to use affordances (aka what is usable through their environment/body) to accomplish those goals. These affordances are not predefined and the space of possible usages for any given tool is practically infinite (a screwdriver can tighten a screw, or pry open a door [0]). Biological evolution also doesn't actually have goal persay, its more of "what works sticks". Whereas evolutionary/learning algorithms have a defined objective function.

The big differentiator seems to lie in the fact that biological agents are able to engage in a semiotic or meaning-making process where given an internal state and its affordances the agent is able to make movements that betters it's state in the future without explicitly being told.

[0] https://www.frontiersin.org/articles/10.3389/fevo.2021.80628... (title: How Organisms Come to Know the World: Fundamental Limits on Artificial General Intelligence)

It is possible that machines can be programmed with a set of main goals and can then set themselves sub-goals in order to achieve those higher goals..

The question is whether us humans would be left out to compete for resources, AI machines may at some point become able to out-compete us for resources. Do we really need to satisfy our desire to play God?

Another less than perfect possibility to come out of this could be AI based war machines - goal driven self sufficient entities that have two goals, to survive and to kill.

I think you have this backwards. The missing magic is consequences. From there, AIs that avoid bad consequences (reduced compute budget?) and seek good consequences (??) may develop a 'want' heuristic that favors those things. Or rather, the ones that do will be more successful and the ones that don't will die out.

You might be partially correct but you seem to omit built-in wants e.g. hunger or thirst or lust.

Furthermore, if a machine has no wants/needs then it won't take any action at all. It will just stand there and won't even experiment to find the good or bad outcomes.

Perhaps, while slowly learning, the machine will cry.

The machine is me.

If not doing anything has bad consequences, then it will do something to avoid bad consequences. It seems to me that "want" and "avoid bad consequences" are quite isomorphic

It would need "to want" "to avoid bad consequences". It is rather circular because of the word "bad" which implies "do not want".

If it is indifferent to all consequences, you are back to square 1, so no, consequences are not the missing magic. Consequences are a function that transform one "want" into a different "want".

You aren't back to square one. Even indifference will result in selection so long as the indifference is demonstrated in different ways by different AI. The ones which have behavior which accidentally maps slightly better to good outcomes will so better. So long as that gets propagated more strongly to the next generation of AIs than average, you get selection for good outcomes without any explicit 'want' function

You're right that want is important. But what I think you miss (I may be wrong) is that "want" is not an empirical category.

Does a cloud want to rain? Or does it just rain because it obeys physical laws as part of a huge complex system?

Do I want to have a cup of coffee? Or a job? Or a girlfriend? Or am I likewise just doing things because my atoms obey physical laws as part of a huge complex system?

This is not a scientific question. It is a teleological question. That doesn't make it less important, on the contrary. It's the answers to teleological questions which make anything important, including the scientific questions which happen to be important.

So the question is not, "what can we do to make a machine want things", it's "When should we ascribe what a machine does to its own wants".

And my answer is "never". Not as long as it's a product of our wants, which they always will be.

Yes, you want to work, you want to have a cup of coffee. You may not like it, but you decided to do it and then executed a very complex series of steps to accomplish the goal. You're not a cloud, working/drinking is not an inevitable and direct physical process you're undergoing by yourself.

I obviously agree that I have both will and wants.

But I can't say that the physical processes in me are in any fundamental way different, than physical processes anywhere else in nature. Most are deterministic, some may be inscrutable, and some may even be random or unknowable - but all those things can be said about the weather, too. You cannot derive my wants from my capabilities (unless you're prepared you ascribe wants to clouds as well, I guess).

The actual physical process of your body using the water once you ingest it is not much different, sure. But you can choose not to drink (and die some days later). A cloud can't choose not to rain.

I'm pretty sure I can't will myself to thirst myself to death, actually.

But you're again missing the point. It feels to me like I have choices. I believe you're right, I have choices. You're kicking down an open door when you're arguing this.

What I'm telling you is that I didn't come to this belief empirically. And indeed I believe it is impossible to justify empirically.

You can. It takes some drugs, but you can. It's not uncommon for people to die because of dehydration at parties (taking stuff like extasis, etc).

You first need individual embodiment, then you will want all sorts of things.