That's the thing. It's not necessarily about an evolution. If I "accidentally" make a break though right now. The AI could evolve into a super intelligence before dinner time.

Simply stated, an AI that writes AI.(Forget the halting problem for a moment) How many iterations can it create in 3 hours?

Think about how long it takes a baby to learn (a few years). There is no reason an AI would self-improve multiplicatively in hours or days or even weeks.

With a powerful enough computer the learning curve of a simulated baby could hypothetically be condensed to minutes, or seconds. Which would appear to us in normal time to be an intelligence explosion.

The simulated baby would extremely tax to the limit the entire resources of whatever supercomputer trained it. People have to run deep learning programs on multiple GPUs right now, and performance is constrained by memory bandwidth, which does not follow Moore's law.

It's a hypothetical, not expected to happen in conventional supercomputers with the current state of machine learning.

Imagine a massively parallel optical computer with the same transistor density as the human brain, the size of an olympic swimming pool running at the speed of light, and networked with 1000s of other similar computers around the world.

Foomp, superintelligence, you won't even be able to pinpoint the source.

Sure, but you would get the baby situation I posited years before what you're envisioning. You're skipping a bunch of steps. My hypotheticals (if they ever do happen) would occur way before your hypotheticals. Progress does not go in "Foomp" steps.

Everything goes in foomp steps if you condense time enough. Condense the last 200,000 years of human evolution to a couple minutes, and it goes 'Foomp', and the super foompy part doesn't happen until the last 2/10ths of a second with industrial civilization wherein the development of global networks compounds our collective intelligence exponentially, yielding unforeseeable emergent properties.

So certainly you would get the baby situation first, but going from manageable baby to astral foetus could potentially happen rapidly and unexpectedly as the rate of progress accelerates to unfathomable speeds, which is what's happened already in going from tribal man to modern civilization, and if you extrapolate that very consistent and reliable trendline, it leads to progress happening in a foomp step perceived as a foomp in real time. Really, all life is just one big accelerating foomp.

This makes no logical sense, You're not addressing my point about the fact that the intermediate progress will be chronologically ordered in time.

> If I "accidentally" make a break though right now. The AI could evolve into a super intelligence before dinner time.

We don't need to rely only on humans to design every aspect of neural networks. We are already computationally searching for AI designs that work better. In a recent paper, hundreds of variations of design for the neuron of the LSTM network have been tried to see which one is the best and which of its components are important.

Also, we can play with networks like clay - starting from already trained networks, we can add new layers, make layers wider, transfer knowledge from one complex network into a simpler one and initialize new networks from old networks so as they don't start from scratch every time. We can download models trained by other groups and just plug them in (see word2vec for example). This makes iterative experimenting and building on previous success much faster.

I don't think evolving a super intelligence will happen by simple accident, it will be an incremental process of search. The next big things I predict will be capable robots and decent dialogue agents.

Probably dependent on your computer ;)