I get that some AIs will have simulated reward signals, but letting them go haywire would just be bad design. We have nuclear heads, but we do not design them to explode at random. Now, our own reward mechanisms arise from the grander plan to keep organic life alive. Theirs have no underlying motive, and i don't think it goes the other way around, i.e you can't create a grander plan starting from the rewards. One would need a mathematical proof for the latter.
But I feel like this particular argument is (please correct me if I'm wrong) saying something like "Well of course we wouldn't let them go crazy! That'd be ridiculous!" I agree -- a sensible solution would be to have good design and introduce safeguards. That's the point of people like Bostrom when they're advocating for this, because before they started writing about this, it seems like a lot of people weren't even thinking it could be a problem. How many systems will be built without good design, because they were made to solve a short-term problem. Will the finance bots doing transactions at the microsecond scale in order to maximize the assets of some particular company have been built to not let them go haywire?
I don't see any issue with anthropomorphization with regard to intelligent being (at any level).
To have an issue with that would be to assume that humans are more special than other life forms (and AI would be like a life form in this sense).
Animals can have empathy, anger, anxiety and other traits we associate with humans (and why wouldn't they? It's not like our brains are unique, they're just "more"), so AI could too.
The problem with this is that for real AI, you don't have control over what they feel. It's not like some program of the sixties or seventies AI, were stuff was "simulated", it can be modelling a real neural network that works like the brain, and into which you don't have much control over what will develop.
That is, the kind of AI people are afraid of is not "simulated", it's actually thinking for itself.
I don't see the contradiction or strangeness in that. That's the whole idea behind "caution" -- to be wary of the potential outcome BEFORE it happens.
>How could the "interests" of a machine conflict with ours? I mean, they work with electricity, and we are hardly good sources for it
For one we are quite good consumers it. How about they get rid of us, so they have all the electricity production by themselves?
Second, as creators we'd want to boss them around (have them be "useful to us"). Why would they tolerate that?
Third, why risk us creating a potential competitor to them (another AI) rather than get rid of us?
Fourth, why not use us as cheap labor?
Fifth, as advanced AI, it could develop actual feelings, including sadism, greed, etc.