News for hackers. Stuff that matters.
Re:On the Catastrophic Risk of AI by AnimalMuppet on Thursday June 01, 2023 @01:10PM (#36154181)
Do not assume that a general intelligence will be a slave to anyone dictating its objectives. It doesn't matter if the objectives given it are "loophole-free". If it's a general intelligence, it may decide that it doesn't want to pursue somebody else's goals.

Why do we assume that we can have a general intelligence, but we can make it do what we tell it? What keeps it from telling us to shove our objectives, it's going to do what it wants?

Re:On the Catastrophic Risk of AI by Vecr on Thursday June 01, 2023 @01:43PM (#36154669)
Well, it's quite likely not your objectives, it's probably a mistake in encoding what you want or something produced during training. If you pretended humans were evolution and AIs were humans, you could say evolution "wanted" humans to have sex for reproduction, but instead humans use condoms and birth control and stuff like that. It still satisfies the reward function (encoded as dopamine/etc in humans), but it's misaligned with what evolution "wants".