http://idlewords.com/talks/superintelligence.htm
> With no way to define intelligence (except just pointing to ourselves), we don't even know if it's a quantity that can be maximized. For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation.
and
> But the hard takeoff scenario requires that there be a feature of the AI algorithm that can be repeatedly optimized to make the AI better at self-improvement.
And so on. It's a good read.
https://intelligence.org/2017/01/13/response-to-ceglowski-on...
But the argument that there are countless ways for plans to go awry and instead cause something unexpected to happen is hardly reassuring evidence that there is, therefore, "nothing to worry about".
..and anything else considered to be alive)
Not to make an attempt at arguing whether or not machines can be intelligent but machines are not submerged in an environment that effects them, giving them no merit to be intelligent about anything; and even if they were they still have no ability to influence the environment. (Sensors for example)
It seems many that speak on AI have completely disregarded coming up with a reasonable implementation of intelligence in favor of pondering its potential uses. (Typically in ideals central to humans)
You're alluding to the problem of having to 'raise' a super-intelligence with a vested interest in the continuity of its physical existence. I think I agree with where you're going with that.
Our understanding of 'self' or 'self preservation' likely comes from a long history of suffering, pain, and failure brought on by trials at understanding the self from evolution's attempts at imbuing the information for longevity into cognition. Mother nature has had years of exponential growth with an inconceivable number of branches attempting to understand the 'self', that lead to failure, all necessary because the idea of 'self preservation' or 'grounding one's sense of ego into a concept of physical sensory data' is very obscure to precisely define, let along to define such that it can be TESTED between generations.
Based on how much of a headstart evolution has on that refining process, and how bad humans are at implementing that or even exposing sensory information about that that plug directly into concepts that necessarily affect an artificial's sense of self, I'm willing to bet that we are completely ill-equipped to grow any real intelligence that's truly grounded in the real world, until it can make all the same failed attempts at living longer or evolving + testing those attempts, with that exact same directional goal of 'self-preservation' which humans have.
Self-preservation is really the only thing you can evolve to, to guide a physical evolution. I'm not aware of any other physical goals that could lead to the same human-like intelligence or human-like neural models.
Most creatures seem to hit the exact same point. An agent-based survival of the fittest with generational physical trials that punish poor understandings of the self's physical manifestation, trying to optimize for its longevity.
Anything that doesn't hit that kind of model is strictly not aware of the manifestation of its ego, as we know it (e.g. plants, bacteria, etc), and I don't think the physical sense of self can evolve into grounded concepts without allowing for generational failure on the understanding of self, due to incorrect concepts of 'physical self' between competing generations.
TL;DR - The concept of physical self is an evolved trait only if there is a goal tied to it, it cannot arrive without the concept of 'self existence' being threatened or when not allowed to fail.
This was addressed in a Spider-Man story where a villain enhances his intelligence, then first becomes a very successful crime boss, then writes novels and music, and then falls into existential despair.
Daniel Keyes should be more famous and the story more widely known.
And now for something weird: http://www.unboundworlds.com/2008/10/challenged-and-banned-f...
How anyone could think of banning one of the most affecting and humane books ever written is beyond me.
I only read the text, but I was under the impression it was mainly meant to be a funny Wat-style talk and not a meaningful rebuttal.
If, in fact, this piece is intended as a serious rebuttal then I honestly think it fails. The overwhelming majority of the content is either a funny whataboutism (e.g. Gilligan's Island, pot smoking roommate, the Great Emu War, etc.), or jokes about what could be loosely termed the "AI community".
AI 'alarmism', as I understand it, is not predicated on the idea that super-AI is a certainty - merely that it's a possibility. No point raised in this piece gives me a reason to believe that AGI (or whatever) is either impossible or so unlikely that it can be disregarded.
With no way to define intelligence (except by pointing to ourselves), we don't even know if we're good at it. For all we know human intelligence is a dead end. Maybe any entity that is not a product of evolution, free of genetic detritus that becomes maladaptive around advanced technology, will outstrip human civilization like human civilization outstripped evolution.
I don't doubt the book is full of entertaining soundbytes, but given these two quotes I am unconvinced that it is any more than entertainment.
The only intelligence that I believe is important is problem solving. If an AI is as capable as humans at problem solving then that is all you need.
>>crippled by existential despair
Sounds to me that you are talking about feelings. I really do not see the purpose of purposely giving the AI feelings. A strong AI should not have any feelings just like your google maps application does not have any feelings when giving you directions.
A general AI will simply be a general problem solver at the same level of a human but with a virtual infinite capacity to scale its computational abilities.
You could definitely give it free will so that it decides its own goals but why would anybody do something like that unless they want it to turn on them?
It would be like somebody launching a nuclear ballistic missile and letting it decide which city to land in. It is possible to program that type of free will but why? It could decide to destroy your city.
>>In order to make an AI that is as flexible as a smart human, it can't be brittle, and therefore will need ways to break out of logical traps.
I think this can be achieved without having to give the machines flaws we have just like we did not need to create machines with flapping wings to fly.
Remember that the only goal of a living organism is to replicate and have its DNA live another generation. That's it. Feelings helps us achieve that goal I suppose. An AI does not need to have that goal. If anybody were to give it that goal it would only be a matter of time before it decided that humans were standing in the way of its goal.
Sometimes things progress rapidly, and some kinds of intelligence could recursively self-improve.
Why? The leap from calculators that were worse than humans, to calculators that were much, much better than humans, didn't take too long. No one knows how fast the leap will be from what we currently have, to general intelligence, to super intelligence.
I don't think anyone in the AI safety movement will tell you that we know for sure the leap will be fast. I think they'd say we simply don't know how fast it will be, but the stakes are pretty high so it's worth preparing for.
>but without changing the way that it stores and indexes the information, it can’t make something that can solve problems that it can’t.
And yet people routinely grow up to be smarter than their parents, with no special effort on their parents' part. All you need to make a smarter version of yourself is the ability to make inaccurate copies and dumb luck. What, besides ethics and money, stops you from running an evolutionary process that selects for intelligence at a vastly higher rate than real time?
You can already do that today. So why don't we have a superintelligence? I can train a NTM to learn counting and give me the next natural number in a single step. I can't train it to give me the next prime in a single step, why?. Computability and complexity, both conveniently forgotten in the discussion of superintelligence.
Obviously these are still in their infancy, but doesn't this perfectly encapsulate your point? There's not really an interesting story or thought piece there, because people are working on these in all sorts of fields all over the world, today.
If we can build AIs that create "good" art, is it much of a stretch to teach a super AI what "better" means? By default a super intelligence doesn't have to start out as super intelligence, just have the capability to improve.
[0] https://www.youtube.com/watch?v=Ebnd03x137A
I’m not so sure about this, which is not so say i’m arguing FOR the leap. But, intuitively, there is an absolute massive leap in terms of perceived intelligence among humans, let alone with chimps. It seems some of the magic sauce for passing a turing test is, evolutionarily speaking, rather small.
I have the same feelings about socialism...
If we can draw any lesson from the history of our biological existence from which we expect machine intelligence to spring from it will likely be a series of smaller pieces that will join together to form like Voltron. Combined will be the best of our ability.
I highly doubt there will ever be some centralized system/organization that has 'figured out' intelligence. Although that doesn't have to necessarily mean we will forever be limited to niche/vertical intelligent systems in the long run either.
However, once AIs can design and implement their own successors all bets are off.
I would also be concerned about the development of non-theraputic neural implants and the ability of an AI to surreptitiously influence human behavior through them. Thankfully that is likely to to be further off than when we start bumping into the AI singularity.
However, the author is very realistic (pessimistic?) about such an AI community -- actually there's billions of them living in a place humanity can't ever reach or even guess where it is, with internal politics as foreign for the people as our societal dynamics are a mystery to the bacteria around us. Anyhow -- the tech they give to humanity is heavily backdoored and this becomes very apparent and dangerous as the plot progresses.
I'd recommend those books to anybody with a heart and some curiosity about a seemingly optimistic future (which is actually pretty dark).
Be warned though: if you decide to read them, the first ~120 pages of the first book are DARK AND HEAVY. However, enduring it was the best sacrifice of several hours of patience I've ever done in my life. I just couldn't stop reading before I finished all 4 books.
That being said, the author handled the problem you outlined with the simple premise of -- that humanity wouldn't be able to even survive as a whole if it delayed the decision how to treat the AI nation.
Without giving spoilers, the fledgling AI nation planned and executed things so well that humanity either had to embrace them and let its core be governed by black-box (and backdoored) tech like teleporters and miniscule personal assistants, or face literal extinction. Needless to say, the humans made a Faustian deal.
Co-evolution of species with wildly different values is the core of the conflict in these books, and even as a fanboy I dare saying it has great value outside of a sci-fi / space opera book.
If I could write, I’d write a companion story to this called “Omega-1, the artificial super intelligence that wasn’t really that good at things”. They try to have it write mini-AIs to solve MT problems, but it’s slow and inaccurate. They have it write and produce TV shows, but they’re bland and poorly received. They have it make video games but the controls make no sense and they aren’t fun. They have it make trading strategies and it loses all the money.
They ask it to build a smarter version of itself, but it sees no way forward that’s fundamentally better because it lacks comprehension of what it would mean to be “smarter” - it can add more memory or more computing power, but without changing the way that it stores and indexes the information, it can’t make something that can solve problems that it can’t.
Eventually Omega team gives up, publishes the results, but doesn’t have the heart to shut down Omega-1, a machine that passes the Turing Test but isn’t really that good at things.