My second claim is not there to invalidate the first, but rather to point out that there are two "AI problems", and that one is a lot closer to science fiction than the other.

I've expounded on the denial: we will need to first figure out what motivates humans before we can replicate their levels of intelligence. So in order to solve HGI, we will need to understand HGI, which means we will be able to control HGI. We will understand personalities and values in a systematic way, and we will be able to choose the personalities and values of any one instance of HGI.

> we will need to first figure out what motivates humans before we can replicate their levels of intelligence

I would not assume this with certainty.

There are many systems in nature that consist of simple pieces, when composed, yield enormous complexity. Replicating intelligences might be like black boxes which we don't understand but can generate, or grow, if you will, from simpler primitives. We might get lucky and discover the means to generate intelligences without understanding them.

As an example, "Machine learning works spectacularly well, but mathematicians aren’t quite sure why." https://www.quantamagazine.org/20151203-big-datas-mathematic...

>Replicating intelligences might be like black boxes which we don't understand but can generate, or grow, if you will, from simpler primitives. We might get lucky and discover the means to generate intelligences without understanding them.

On that premise, the risk of accidentally discovering the right bit-string that morphs into a all powerful evil AI seems to be minuscule. Since at least human intelligence consists of multiple dimensions/factors and only some combinations thereof can be described as "intelligent", given a certain environment. Accidentally discovering something "thinkable": maybe, brute forcing the right combination for human-surpassing intelligence? Most likely not.

You make it sound like strong AI will be programmed with ALGOL 68 control structures and loops, hand-coded (i.e. the right bit-string). If that is your conception in I agree with you. However this sounds much too like the "expert systems" of the eighties.

Biology doesn't have any need for "the right bit-string" (modulo DNA, different purpose) to make intelligence work, and one shouldn't have such an expectation of an emergent strong AI, either. Instead of "the right bit-string", think more "the right model"

I didn't particularly think of ALGOL with the bit-string, but more of the parameter space of the mentioned black-box model. Which I'd assume, coding for a black-box, is quite huge, with lots of local maxima that could appear 'intelligent'. I was referring to discovering the global maxima in that parameter space more or less by chance, as highly unlikely.

I agree with the problem being more one of finding the 'right model', as I'd hope that this will reduce the parameter space greatly. That's why I agree with gggp's post that we'd need to know more about human intelligence, thus also human motivation.

"Anyone who expects a source of power from the transformation of these atoms is talking moonshine." - Rutherford & "It's like shooting birds in the dark in a country where there are only a few birds," - Albert Einstein - The Pittsburgh Press - Dec 28, 1934 - On the idea that man might some day utilize the atom's energy. Einstein was essentially making the same "parameter space" argument you're making.

There will have to be more true breakthroughs to get to AI, yes; I do agree though that it will take a "long" time, however before this happens.

Look at where we are now with ML, and ask what we would need to do to "accidentally" create HGI.

That title's wording is clickbait, not an argument. Mathematicians know exactly why neural networks work – because they contain a model of the data, and they try to develop ways to build (train) those models so that they do it better. The model may not have human concepts associated with it, but that doesn't mean it's somehow "out of control".

In order to achieve HGI, one would have to train models of quite a few domains (the same way schools create models in people's minds), as well as tons of ethical and practical heuristics to weigh an infinite number of reasons for or against doing something.

> Replicating intelligences might be like black boxes which we don't understand but can generate, or grow, if you will, from simpler primitives.

I strongly doubt that.

>As an example, "Machine learning works spectacularly well, but mathematicians aren’t quite sure why." https://www.quantamagazine.org/20151203-big-datas-mathematic....

If you read the HN comments on that very article, I recall posting some links to papers giving theoretical grounding for why deep learning works so well.

This article is meant to be illustrative, its particulars are not the crux of my main point (to which you seem to imply only doubt sufficies to refute)

Well no. What I think refutes your point is that we possess increasingly good foundational theories for what minds are and how they work. I doubt that any intelligence we successfully create is going to be a black-box.