News for hackers. Stuff that matters.
Re:Sam Altman: Why You Should Fear Machine Intelligence (2015) by stanislavzza on Saturday November 25, 2023 @07:44AM (#38413148)
It's not just crashing/halting. See Rice's Theorem. The machine can't predict its own future behavior in most ways that are important.

I'm not an expert in this stuff, and my point was that a serious treatment of super-intelligence should address such limitations to computations. In particular the "super" part seems to imply solving exponential-complexity problems in linear time. I remember looking in NB's book and not finding it.

Re:Sam Altman: Why You Should Fear Machine Intelligence (2015) by nathan_compton on Sunday November 26, 2023 @02:36PM (#38424273)
I don't see why that is the case. I'm much more intelligent than a racoon and I have yet to grapple with Rice's Theorem. There is no reason I can think of to believe that Rice's Theorem is a serious constraint on intelligence beyond my own. In general, such an agent isn't particularly interested in proving facts about its own program (at least I can't see why it would care any more than we are interested in proving mathematical properties of our own brains). It is interested in maximizing some objective function, which can transparently be done without thinking much at all about Rice's Theorem (all systems which train neural networks and indeed, even simpler optimization problems, pursue such maximization with nary a thought towards Turing or Rice's Theorems).