The answer is still yes. Your "thinking out loud" is privileging an already refuted hypothesis.
Well that's a fine assertion, but how do you justify it?
Your "thinking out loud" is privileging an already refuted hypothesis.
WTH does that even mean? There is no hypothesis, there's a vague notion of an area to consider, which - if considered thoroughly - might or might not yield a hypothesis.
The main thing is that the constraints of the physical world are nowhere near the limitations of human capabilities. Yes, there's a theoretical limit to how much computation you can do in a certain amount of space with a certain amount of energy. No biology or technology currently in existence even gets close to those limits.
Without any such fundamental limit there's just no reason for AGI not to present a threat.
But there is lots of other evidence too.
These are basic questions, the answers are known, published, and the article even mentions exactly where you can find them.
I've read and re-read TFA and I don't find that it addresses the issue I'm thinking about. It's not so much asking "are we the smartest possible creature", or even asking if we're close to that. It's also not about asking whether or not it's possible for a super-AGI to be smarter than humans.
The issue I was trying to raise is more of "given how smart humans are (whatever that mean) and given whatever the limit is for how smart a hypothetical super-AGI can be, does the analogy between a super-AGI and a nuclear bomb hold? That is, does a super-AGI really represent an existential threat?"
And again, I'm not taking a side on this either way. I honestly haven't spent enough time thinking about it. I will say this though... what I've read on the topic so far (and I haven't yet gotten to Bostrom's book, to be fair) doesn't convince me that this is a settled question. Maybe after I finish Superintelligence I'll feel differently though. I have it on my shelf waiting to be read anyway, so maybe I'll bump it up the priority list a bit and read it over the holiday.
Right, like I said, this whole thing was just "thinking out loud" more than any thoroughly researched and validated idea. And I didn't do a very good job of even explaining what I was getting at. It's not so much that the question is "are we the smartest creatures possible". It's more like:
"as smart as we are, and given the constraints of the physical world, is there room for a hypothetical 'smarter than human AGI' to represent an existential threat, or something that justifies the analogy to nuclear weapons?"
See also skybrian's comment which actually states it better than I did originally:
https://news.ycombinator.com/item?id=10755109