I find it worryingly plausible, not that AGI is certain to go badly, or even that if it does go badly it will necessarily lead to human extinction, but as Robert Miles says, "If you have a sufficiently powerful agent and you manage to come up with a really good objective function, which covers the top 20 things that humans value, the 21st thing that humans value is probably gone forever" (about 12:40 into the video).
I find it worryingly plausible, not that AGI is certain to go badly, or even that if it does go badly it will necessarily lead to human extinction, but as Robert Miles says, "If you have a sufficiently powerful agent and you manage to come up with a really good objective function, which covers the top 20 things that humans value, the 21st thing that humans value is probably gone forever" (about 12:40 into the video).