You do not, in fact, get general intelligence by accident by throwing a lot of connections in one place, just like you do not get a human just by throwing a lot of organic molecules and water in one place.

So it is possible that we reach AI just by randomly permuting connections and weights. Of course it's more likely we intelligently set (or "evolve") these connections and weights, but this allows us to set an upper bound on computation/time needed.

I don't think this was the case. Yes there is evolution but it is not random. Actually most of the molecules sitting around did not evolve to intelligence. In case our evolution we had natural selection. In case of AI we have artificial selection (selection made by humans) and even if we consider ourselves smart enough to do this we cannot prove that we are able to make it happen (choosing the correct AIs/algorithms to survive) until it happens. Maybe I cannot express this clear enough but the advantage natural evolution has over artificial evolution is the huge number of "experiments" - meaning it had time enough to do a lot of small changes until something worked.

I think you have that backwards; natural evolution is absurdly slow because it takes a very long time to cycle through generations of animals whereas genetic programming on a computer to evolve algorithms can happen billions of times a day because computers are much faster at cycling through possibilities.

> meaning it had time enough to do a lot of small changes until something worked.

Computers can do it faster.

And I agree with you that AI can be much faster.

I still think the artificial selection can be influence by us - humans - so we might add flows in the sistem from the beginning. Of course AI can learn to identify them maybe. But maybe not. Like in the case when looking from inside a system one cannot see how the system really is so it cannot fix it.

Of course what I say are just some hypothesis, nothing proven and I think they cannot yet be falsifiable.

Well, not unless you have a couple billion years to sit around.

And, there's a least some evidence that it's even worse than that -- intelligence does not appear to be common in the universe. So it might well be a couple of billion years * a million planets worth of organic molecules and water (or a hundred million planets. Or a billion planets).

Don't tell that to the deep learning cult! They'll have to think about how it all works!

I have not met any actual deep learning researchers who think they are creating AGI. I think that it's non-experts who write about deep learning research who think that deep learning is creating AGI.

And I'm not even sure I would call it "parameter fitting". I'd call it, "continuous circuit searching via gradient descent", because that's all you're doing: searching for a nice analog circuit for your task within a large, continuous space of such circuits.

Unless intelligence itself conforms to natural principles, unless it "cuts with the grain" of nature's preexisting laws.

Which, frankly, it does. Information theory and thermodynamics are a thing.

Further, to get better you need to optimize for something. You might get an AI that's really good at writing fiction, but how do you measure that and write one that's even better at writing fiction?

The thing about machinery is that it scales. If you have one AI that's really good at writing fiction, you can replicate it and generate all the fiction you like. Not true of human authors.

I do expect there will be other bottlenecks after think time is solved, but even without intelligence much beyond human, the ability to scale in other directions is pretty scary.

You're assuming that it's possible to do efficient knowledge transfer without losing general intelligence.

Maybe in order to have general intelligence about something, you need to learn it from scratch, over time, rather than just "copying" the information.

Another problem is that you're assuming that you can build a powerful enough computer, and still retain the possibility to have them be programmable. That is, that you have easily programmable memory cells within the computer which can change its behavior. But to have easily programmable memory cells, they need space and infrastructure to access them, and they will interfere with critical timing paths. That's why if you build a special purpose ASIC for a task, it will always be faster than a CPU or GPU.

Maybe all the things we consider useless for intelligence, like playing, sleeping, exercising etc., is actually necessary. We've certainly started to see that those who focus too much on studying by cramming don't necessarily become smarter.

You can put me in the "skeptics" camp when it comes to superhuman intelligence. It may be possible, and it's fun to discuss, but it seems to me that the ones who fuzz about it are making an incredible amount of assumptions.

So it seems like the question is whether computers as we know them can do general intelligence at all? If they can, it will be easy to clone.

If they can't, then it would imply that general intelligence is something more than information processing as we know it; perhaps some kind of dualism?

It's not a software or hardware problem it's a data problem as it's not obvious what part of neural net A encodes fir and what part encodes cat's and how you map that to B's encoding of fir and cats while connecting them.

Now, AI is not necessarily going to be neural net's, but it's also not necessarily understandable what all the little bit do, just that they work.