News for hackers. Stuff that matters.
Re:The BS-Industrial Complex of Phony A.I. by ForHackernews on Tuesday June 25, 2019 @11:25AM (#20275162)
> not really caring about you or knowing anything about being alive other than the priorities built into its software".

Is that really "general" intelligence, then? I would argue we don't know today whether consciousness and agency are separable from general intelligence. You could say that "general intelligence" implies some degree of intelligence on any topic, which would include self-reflection and metacognition.

It sounds to me like you're describing something like a p-zombie[0], which we don't know to be able to exist.

[0] https://en.wikipedia.org/wiki/Philosophical_zombie

Re:The BS-Industrial Complex of Phony A.I. by TeMPOraL on Tuesday June 25, 2019 @12:54PM (#20276193)
I believe it is. I'm also not describing a p-zombie, since I find the whole concept of p-zombies utter nonsense.

The general AI I described would have self-reflection, agency and arguably consciousness, yet - per orthogonality thesis - it may not have anything resembling human values.

Re:The BS-Industrial Complex of Phony A.I. by ForHackernews on Tuesday June 25, 2019 @01:06PM (#20276385)
So like, a sociopath, then? I guess we do know that those can exist.
Re:The BS-Industrial Complex of Phony A.I. by TeMPOraL on Tuesday June 25, 2019 @03:58PM (#20278222)
Yeah, sure. The orthogonality thesis essentially implies than a GAI randomly plucked out of space of possible minds will likely be considered sociopathic by our standards. That is, if we can comprehend its thinking at all. "Not sociopath" is a very particular set of values.
Re:The BS-Industrial Complex of Phony A.I. by ForHackernews on Tuesday June 25, 2019 @06:40PM (#20279842)
> a GAI randomly plucked out of space of possible minds

Sorry, but I don't think you have any rational basis for imagining what the "space of possible minds" represents. The only minds with human-level intelligence we know of are human minds that have (with variation) human values.

The claim you're making is analogous to saying "any extraterrestrial life we find won't be carbon-based because out of the space of all possible substrates for life, carbon is a very particular one", but that's an ill-founded supposition because we have a sample of N=1 and maybe carbon-based life is the only kind of life there is.

Maybe a true GAI mind will be "like us", maybe it won't be, but we don't have anywhere near enough data to speak with confidence about it.