Humans weren't designed to have a will, and yet we seem to have them.
> it would have a reinforcement learning system on top of lower sensory and action modules.
Isn't that what OpenAI is doing with Universe? It's simulated sensory/action modules now but I don't see why they couldn't be hooked up to real ones.
I have no idea how you could possibly infer this.
If your complaint is my claim that we have a will, I'm using the common-sense version encoded into our legal and cultural system. I agree that we don't have a good concept of what intentions are, or how they causally connect to actions, but I do know that for at least some of my actions I experience something called "intent" before I undertake the actions.
My overall point was that the capacity for intent can arise through an evolutionary process without being designed in, but it does rest on the two assumptions I just listed.
Maybe for single-use or "constrained" technologies (to be honest I don't even believe that - how does a B-52 Stratofortress reflect the values of Orville and Wilbur Wright?). But isn't the whole point of generalized AI that it's not like other technologies? Even if "regular" technology reflects the values of its inventors, what reason is there to believe that an AI will? AI is a technology that can use itself.