Re:Past Performance is Not Indicative of Future Results (2020)
by fossuser on Monday August 02, 2021 @12:19AM
(#28032839)
AGI isn't really about artificial humans.
It's about very good general problem solving software that's way beyond the capabilities of humans while not being aligned with human interests. Not because the software is evil, but because aligning values is an unsolved problem (humans aren't even totally aligned - and values also change).
If you have an intelligence that's very good at achieving its goal and you don't have a good way to align its goal with human goals, you can very quickly get into trouble if that intelligence thinks much faster than you do.
It's about very good general problem solving software that's way beyond the capabilities of humans while not being aligned with human interests. Not because the software is evil, but because aligning values is an unsolved problem (humans aren't even totally aligned - and values also change).
If you have an intelligence that's very good at achieving its goal and you don't have a good way to align its goal with human goals, you can very quickly get into trouble if that intelligence thinks much faster than you do.
https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pe...
https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden...
https://www.lesswrong.com/posts/BEtzRE2M5m9YEAQpX/there-s-no...
https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien...