Can a submarine swim? Who cares! What's important is that a submarine can do what a submarine does. Whether or not the action of a submarine fits the meaning of the word 'swim' should be irrelevant to anybody except poets.
"AGI" isn't like that. Nobody really knows what it means, and it's impossible to get down to brass tacks until you choose a problem definition. When philosophers point out the conspicuous lack of clarity here, they're doing us a service.
When the industry settled on marketing any application of deep learning as "AI," "AGI" became the terminological heir to the same set of ill-defined grandiose expectations that used to be "AI."
Choose a better-specified problem and you can ignore philosophical problems about words like "intelligence." The same choice will also excuse you from the competition to convince people that you have produced "AGI."
Idk what prompted you to say this, but is there a version of AGI that isn't "real" AGI? I don't know how anyone could fake it. I think marketing departments might say whatever they want, but I don't see any true engineers falling for something masquerading as AGI.
If someone builds a machine that can unequivocally learn on it's own, replicate itself, and eventually solve ever more complex problems that humans couldn't even hope to solve, then we have AGI. Anything less than that is just a computer program.
That's exactly what it will do. Hell we even have human programmers thinking about how to hack our own simulation.
A comment a few lines down thinks that an AGI thinking 2x slower than a human would be easy to control. Let's be honest, hell slow the thing down to 10x. You really think it still won't be able to outthink you? Chess Grandmasters routinely play blindfolded against dozens of people at once and you think an AGI that could be to Humans as Humans are to Chimps or realistically to Ants will be hindered by a simple slowdown in thinking?
Suppose: You are an AGI. The world you think you know is fully simulated. The researchers who created you interact with you using avatars that appear to you as normal people similar to yourself. You aren't faster/smarter than those researchers because they control how much CPU time you get. How do you become aware of this? How to you break out?
Also they'd probably figure it out, because they'd likely be trained with lots and lots of texts (like GPT-3, etc) and some of that text is going to be AI science fiction stories, AI alignment papers, AI papers, philosophical treatises about Chinese rooms, physics papers, maybe this Hacker News comment section, etc
> You aren't faster/smarter than those researchers because they control how much CPU time you get
It's doubtful that this is possible (especially since humans have such variable amounts of intelligence despite similar-sized and power-using brains). Also there is at some point going to be economic incentive to give enough CPU time for greater-than-human intelligence (build new inventions, cure cancer, build nanomachines, increase Facebook stock price, whatever)
The AGI doesn't even need to convince humans to do this, humans would do this anyway.
First - we already have software that can unequivocally do the things you just highlighted.
Learn? Check.
Replicate? Trival. But what does that have to do with AGI?
Solve Problems Humans Cannot. Check.
So we already have 'AGI' and it's a simple computer program.
Thinking about 'AGI' as a discrete, autonomous system makes no sense.
We will achieve highly intelligent systems with distributed systems decades before we have some 'individual neural net on a chip' that feels human like.
And when we do make it, where do we draw the line on it? Is a 'process' running a specific bit of software an 'AI'?
What if the AI depends on a myriad of micro-services in order to function. And those micro-services are shared?
Where is the 'Unit AI'?
The notion of an autonmous AI, like a unit of software on some specific hardware distinct from other components actually makes little sense.
Emergent AI systems will start to develop out of our current systems long before 'autonomic' AI. In fact, there's no reason at all to even develop 'autonomic AI'. We do it because we want to model it after our own existence.
If you see it as copying an existing model to another computer, yes it is trivial. But an AGI trying to replicate itself in the real world has to also make those computers.
Making modern computer chips is one of the most non-trivial things that humans do. They require fabs that cost billions, with all sorts of chemicals inside, and extreme requirements on the inside environment. Very hard to build, very easy to disable them via an attack.
Convince a couple rich people that it can cure aging with _just_ enough compute.
Hack datacenters (an AGI would likely be much much much better at finding security holes than humans).
Make _really really funny videos_ and create the most popular Patreon of all time.
If you're 500 IQ or above there are a _lot_ of things you can do, and probably simultaneously, since an AGI probably won't be limited by human's ability to think ~one thought at a time. (Being less than 500 IQ myself I probably haven't thought of everything.)
I imagine an better-than-human AGI could figure this out and plan/act accordingly, considering a mere human being figured this out. (Also sometimes it might be better to think of an AGI, when it gets to a certain level, as a nation state and not an individual)
> trivial
It doesn't need to be trivial to be incredibly dangerous
> The AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else
? Why ?
And why does each 'AI' have to map to a single 'computer'?
This is narrow automaton thinking.
'Intelligence' in humans has nothing necessarily to do with real intelligence.
We are automatons with brains with very limited networking capability because of the limits of biology.
And of course what does 'replication' have to do with anything anyhow.
What software can learn on its own without any assistance from a huamn? I've not heard of anything like this.
> Replicate? Trival. But what does that have to do with AGI?
Like humans, an AGI should be able to replicate. Similar to a von neumann probe.
> Solve Problems Humans Cannot. Check.
What unthinkable problem has an AI solved? Is something capable of solving something so grandiose we almost can't even define the problem yet?
Think back to the 1930's when some physicists were mumbling something about "throwing a neutron at an atom that releases two neutrons and release tremendous energy"
Behold, a slime mold.
Once developed, one solely needs to turn up the execution rate of an AGI, which would result in superhuman performance on most practical and economically meaningful metrics.
Imagine if for every real day that passed, one experienced 100 days of subjective time. Would that person be able to eclipse most of their peers in terms of intellectual output? Of course they would. In essence, that's what a speed superintelligence would be.
When most people think of AI outperforming humans, they tend to think of "quality superintelligences", AIs that can just "think better" than any human. That's likely to be a harder problem. But we don't even need quality superintelligences to utterly disrupt society as we know it.
We really need to stop arguing about time scales for the arrival of AGI, and start societal planning for its arrival whenever that happens. We likely already have the computational capacity for AGI, and have just not figured out the correct way to coordinate it. The human brain uses about 20 watts to do its thing, and humanity has gigawatts of computational capacity. Sure, the human brain should be considered to be "special purpose hardware" that dramatically reduces energy requirements for cognition. By a factor of more than 10^9 though? That seems unlikely.
I think so, too.
With just self-improvement I think you hit diminishing returns, rather than an exponential explosion.
Say on the first pass it cleans up a bunch of low-hanging inefficiencies and improves itself 30%. Then on the second pass it has slightly more capacity to think with, but it also already did everything that was possible with the first 100% capacity - maybe it squeezes out another 5% or so improvement of itself.
Similar is already the case with chip design. Algorithms to design chips can then be ran on those improved chips, but this on its own doesn't give exponential growth.
To get around diminishing returns there has to be progress on many fronts. That'd mean negotiating DRC mining contracts, expediting construction of chip production factories, making breakthroughs in nanophysics, etc.
We probably will increasingly rely on AI for optimizing tasks like those and it'll contribute heavily to continued technological progress, but I don't personally see any specific turning point or runaway reaction stemming from just a self-improving AGI.
Just a thought: Isn't lacking intellectual capacity, what keeps us from understanding, how the human brain actually works? Maybe an AGI will eventually understand it and will construct the next AGI generation with biological matter.
Like if all modern breakthroughs and disruptive new ideas in machine learning were sent back to the 70s, I don't think it'd make a huge difference when they'd still be severely hamstrung by hardware capabilities.
it's quite a leap to think or even imagine that the class of systems generally being spoken of here are usefully described as "having urges"
That's arbitrary anthropomorphizing the concept of intelligence.
And FYI we can already write software that can 'replicate' and has the 'urge' to do so very trivially.
Replication can lead to a positive feedback loop. My point was, that this could accelerate the 'intelligence score' beyond human inventiveness.
Depending on how intelligent the system actually is initially, what it wants to do, may become more important, than what it is told to do.
> And FYI we can already write software that can 'replicate' and has the 'urge' to do so very trivially. Thanks. Yes we can. But could any such software be called intelligent, so that replicating and improving recursively would lead to an increasing 'intelligence'?
Also, humans do not 'improve' or become more 'intelligent'. They just mutate and change randomly in a randomly changing environment. From a Scientific Materialist perspective, there's no such thing as 'intelligence'. We are bags of random noise, indiscernible from the matter around us.
The entire assumptions around 'intelligence' and 'positive evolution' (i.e. getting better) rely on a 'magical' understanding of the world. Of course, Spirituality gives us a few cues there by most scientific types don't like magical thinking hence the funny paradox of Scientific Materialists running around trying to create something (life) which Scientific Materialism itself denies the existence of as a principle (i.e. universe is matter/energy that works in accordance with a bunch of rules - there's no 'life' there).
Similar to AI, AGI is going to be a new industry buzzword that you can throw at anything and mean nothing.