Re:The Reluctant Prophet of Effective Altruism
by dotsam on Wednesday August 10, 2022 @03:30AM
(#32408786)
I definitely don't want to sound like I'm saying we should stop worrying about hackers and bad actors, they are of course a concern. I certainly don't want to advocate for dismissing real dangers and power structures in the real world in favour of speculation over some theoretical risk. But I do want people to think about it and take it seriously, in the same way I would have wanted people in 1933 to take seriously the possibility of nuclear weapons even though they were then only theoretical.
I think the sticking point in this is that you don't think that AGI is ever possible - is that right?
If so, I know a comment on a HN thread is very unlikely to change your mind on it! But when the stakes of being wrong are high, I find it useful to go from a position of "it won't happen, so I won't worry about it" to something like "on balance I'm pretty sure it won't happen, but I may be wrong, and in that case I would be worried about it".
I'd then feel much better about spending some time researching the topic in more depth.
The alignment concern falls flat with me because it assigns agency to a computer program. At what point is an AI's intentions its own, and not a cost function put in place by a programmer, directed by an investor? I am more concerned about the actions of programmers and investors than some theoretical virtual self.