From the perspective of embedded cognitive systems, intelligence is not an abstract property but a faculty derived from an agent embedded in a world with an ability to act upon that world in specific ways. This is reiterated in cognitive linguistics by Lakoff’s metaphor work et al. I take this to mean that the question of AGI itself is missing the point. To wit all recent provoking demos of AI (Alpha Go, DOTA) are embedded within specific environments and actions of a model-specified agent. It is then extrapolated that AGI will appear automatically induced by a series of more and more competent AIs.

Incremental evolution as a means of indiction as we know from biology can take a long time, and the landscape is complex. Hence the view that it might be coming but not soon and not with a roadmap.

I've seen the singularity described as "The Rapture for nerds", which is pretty apt. It's going to be amazing, the future is so bright, we won't have to worry about any problems plaguing mankind now, and it's just around the corner! Any day now! Aaaaaaany day now!

Exhausting. Maybe we'll get a future AI-run utopia a la The Culture. Maybe not.

I like this term. Going to use it any chance I get now.

Much of the skepticism isn't about whether this can be done in theory. It's about "how close" we are given the current state of the art (beating Go, chess, deep NN, etc).

One can be (I am) simultaneously blown away by these accomplishments and believe that we have hardly scratched the surface of AGI.

And while this isn't necessarily a deal breaker for AGI, it's not encouraging that we still understand close to nothing about consciousness.

> I don’t think anyone here disputes the logical possibility

> Much of the skepticism isn't about whether this can be done in theory

The math tells us it cannot be done with computers. Read Penrose.

I'm familiar with Penrose.

He's interesting, and I don't have a firm opinion on this (ie maybe it turns out it is impossible), but presenting the case as "the question of AGI has been settled by math" is really misleading. This is an area of open debate among philosophers, mathematicians, physicists, and evolutionary biologists.

You seem to be missing a key ingredient: billions of years and endless branching iterations with the totality of all non-sequestered resources available. Even if you scale that to account for the more rapid iteration theoretically possible with machines, it doesn’t paint a rosy picture for any of our lifetimes.

Plus, in organisms the software and hardware are both subject to mutation, and exist in the context of a global system of competition for resources. That only tangentially resembles work done on AGI, and only in the software component. We can only design and build hardware so quickly, and that adds more time. I’m not hearing pushback against the possibilitY of AGI, just singularity “any day now” claims that seem mostly calibrated to sell books and seminars.

And similarly, I’d love to understand why you’d believe in something for which there’s zero evidence and ask with incredulity why others don’t also believe it. Isn’t it reasonable to be skeptical until it actually happens? Isn’t it possible that we don’t understand what intelligence is, and that there could be an undiscovered proof that computers can never attain it? It seems too early to call.

Computers haven’t to date ever done anything on their own, what reason is there to think they ever will? We made them. Neural networks are nothing more than high dimensional least squares. Maybe humans are also least squares, but my money is on there being more to it than that.

> ...they don’t actually need to reproduce the lungs and the intestine...

Obviously, but the one thing that all intelligent life does is reproduce and contain a survival instinct. No computers have a survival instinct, and thus no reason to get smarter that we don’t imbue them with artificially.

Interestingly, people that spend a lot of time studying their own brain by meditating are usually very confident that what most people call "free will" does not exist - is merely an illusion.

Not to mention that physics tells us that we should be simul-able, after all physics knows only about deterministic and random processes.

It's already implemented in at least one physical system.

Aren’t there some kinds of math that are proven to be unable to solve some problems while other kinds of math can? Trisecting an angle can’t be done with a ruler and compass, and a 5th order polynomial can’t be solved analytically. Both of these things can be done numerically.

Is it possible then that some physical systems can’t act in the same ways as other physical systems? A granite rock can never burn at the same temperature as a tree, but they’re both physical objects. Is it possible that binary logic on silicon doesn’t have the physical properties for intelligence that animal brains do?

There is solid evidence that teams of mathematicians, scientists and engineers will be able to catch up with what nature has wrought in the next few centuries, if not much faster (decades on the optimistic end). Intelligence could be one of the easiest part of biology to replicate, given how shonky human intelligence is when tested against objective standards.

For my part I have faith that computers are great at memorization, and will continue to improve on that front. However, I'm less convinced on their ability to "understand", which is admittedly poorly defined, but intuitive. It seems to me there's still a missing piece between machine learning (essentially all things that are called "AI" these days) and the kind of generalization that we expect out of even a human 1 year old or your average vertebrate.

But that can be fixed in simulation. That's why most RL research (RL being the closest branch to AGI) is centered on games. Games are simple environments we can provide to the AI agents today. In the future I don't see why a virtual environment could not be realistic, and AI agents able to 'understand'.

...is strong AI a more sophisticated invention than cells? Is AI more sophisticated than DNA? Is AI a more sophisticated invention than a habitable orb of magma orbiting a nuclear explosion in space? It's all crazy. It's all amazing. ...We are all lucky enough to be here for a little while to see whatever this is. We're a point on a stream.

Even more to the point the cycle time for human evolution is at a minimum 14 years (biological limit to reproduce) and around 25 years (today's societal constraints). Whereas cycle time for new computer generations is what 12-18 months?

So at best every 14 years we get a new 'human model' with some random variations. Meanwhile in that time frame our computing hardware is 2000x improved (hopefully improvements in software accelerate that improvement further).

Again our own consciousness wasn't created it emerged which means that we are a product of this emergence.

Remember that we are also constrained by our own intelligence. Computers might not be the problem.

You just described AI in the 90's: Artifical Life, subsumption architecture, evolutionary algorithms. Theoretically we should be able to evolve intelligence but the search space is impossibly high and we don't understand why life is evolvable. Even if we did evolve an AI through a simulation of evolution - there's a small chance we would understand it.

It took nature what like 4 billion years to come up with the human brain in a single species... We including Kurweil think that this can be replicated in say 100, I mean not even that according to the singularity, 40 years. I seriously doubt it. Can the human brain eventually be surpassed, yes, but it's very likely that we will go extinct before then.

How long did it take evolution to build a structure taller than 100 m?

Technology just moves on different timescales, with different methods and different objectives. Evolution is slow and dumb, and not goal oriented.

In some domains, technology might never catch up to biology. In others we've beaten it soundly.

Consciousness transfer is probably 10s of thousands if not millions of years away.

A comment above about being as intelligent as a mouse seems to touch on this problem domain rather elegantly : "navigate the forest floor looking for food and avoid predators".

We have no evidence that life evolved from inorganic matter (i.e. that life 'began' at some point). That's a widely made assumption but it remains only that. The alternative is that life has always existed (and continually evolved).