I agree with you, just wondering your thought on how we will know the latter has come true? As I understand it, you're describing ASI, not AGI. The way I personally define AGI, we are already there.
Sure, but you don't need very many people to do that relative to the billions of workers that currently exist.
If you wipe out 90% of all jobs instead of 100% the practical implications for society as a whole are roughly exactly the same.
The Great Depression was "only" 25% unemployment.
I don't consider this a good precedent at all.
When this happened, humans went off and started doing what we now consider "knowledge work". They escaped massive unemployment by being smarter than the tractors. How do they escape AI job loss when AI becomes smarter than the average human? or smarter than the smartest human?
In prior displacements humans moved up the stack. There is no longer room for them at higher levels of the stack if AI beats them at cognitive labor.
Humans keep a few around for entertainment but that’s about it.
The point is that people, like horses, can do other things. Humans don't even have to be better than robots at anything. It's the principle of comparative advantage.
...like what? If we start with the assumption that AI can broadly replace humans in knowledge jobs, what industries are absorbing tens of millions of humans for labor?
I see this as a categorical difference. The difference between AI and agricultural mechanization is that AI is highly heterogeneous. During the mechanization of agriculture that you talk about (which also happened over generations), there were alternative industries to move to. What happens when AI is competing in the majority of industries? Are there going to be 10 million new plumbers?
There's a better chance that a humanity-aligned AI would make sure that humans have something to do and keep us alive versus a capital-aligned one. At the same time, if AI would be the deathblow to capitalism, that'd be very, _very_ ironic.
On the other hand, if we are going to survive as a species for a long time it probably depends on us learning how to not be the most intelligent thing around.
Even if it leads to our demise, I don’t think willfully holding ourselves back is the way forward
What the fuck?
It’s just a risk / reward question. The upside is worth the risk imo. And frankly in the infinite universe it is inevitable that we encounter super intelligence at some point if we live long enough. We may as well try to hone it.
What would be the limiting factor?
For example, much has been recently discussed about how OpenAI's math results, as impressive as they are, remained within "the convex hull" of existing math literature. Thus, some have argued that the way LLMs work will keep them squarely within this convex hull, and also there seem to be clear examples from human history where humans came up with breakthroughs that were outside the convex hull of ideas that existed at that time.
There has been a lot of recent debate, both for and against, this idea that current AI approaches are limited to this convex hull. So I'm not going to try to convince you one way or the other, but I do think it is at least a plausible explanation of how/why current approaches end up hitting a wall.
I'll also show my bias which is to say I certainly hope they hit a wall, because I don't understand how anyone can think that humans won't be truly and completely fucked if AI ends up far surpassing human intelligence.
some proofs of famous conjectures (for example unique games conjecture) have been described as creating new, insane, almost alien ideas. Scott aaronson has a blog post on this.
Edit: OK, I found his rationale in the comment section, and I don't find it particularly convincing:
> (1) Einstein and Gödel and Gauss and Shakespeare “merely filled out some points in the convex hulls” of everything that came before them, if you take an expansive enough view of what those convex hulls actually were. What a pathetic piece of cope.
> (2) Dana tells me she now mostly understands the proof of the UGC, and is amazed by the new ideas in it, and wants to give talks about it soon. So, maybe people can listen to that and then lecture her on how it’s “just a point in the convex hull” of the ideas that she’s worked with for 20 years!
First, to just use Einstein as an example, I remember reading many years ago (like at least a decade plus) that many scientists felt like if Einstein had never lived, the theory of special relativity (i.e e=mc^2) likely would have been found pretty quickly in any case - the work had already been done putting the nail in the "luminiferous ether" coffin, and people were starting to connect the dots. I think most people would agree that special relativity was within the "convex hull" that existed at that time. However, I've also heard that general relativity was so far outside the bounds of scientific thought of the time that if Einstein hadn't discovered it that in may have been 50-100 years or more before we figured it out - it was far outside the bounds of the convex hull of that time.
All that said, I'm not a mathematician qualified to determine whether what has been discovered by OpenAI is inside or outside the convex hull, and I do think that the vast majority of human intelligence operates within the convex hull in any case, and what AI can do is hugely, immensely powerful. Still, I'm more inclined to agree with this other commenter on Scott's blog:
> I am a little more in the “convex hull” camp when it comes to AI creativity. In my own interactions on research problems, I find the AI to be singularly uncreative. But these are not math problems.
> From my own introspection, my creative ideas come from a kind of visualization: situating myself in a quasi-physical abstract place that is very difficult to explain. But from there I can simply “see” things by “looking around”.
> This seems to me to be a byproduct of being an embedded agent, not just a prompt-responder. Embedded agents reflect and cogitate using their model of the world. Yann LeCun in particular pushes this as a necessary requirement for true intelligence.
> That said, I don’t agree with LeCun’s claim of necessity. Just as reasoning turned out to be something mysteriously emergent from language in the LLMs, so too could this kind of creative behavior. But I have no idea how.
In any case, I think most researchers have been filling the convex hull (and have been enjoying doing so?) for most of their research. So even if Einstein provided something truly necessary, there's a reason there's only one Einstein (or grothendieck or insert famous name here). I know that's moving the goalposts but yeah.
That said, other humans can and will put AI to use in ways that don't benefit humanity. It's actually already happening - newsfeeds waste hours of people's lives and attention; give teens low self esteem etc etc. and guess what. Society is adjusting - governments via legislation, device makers via screen time limits, communities by spreading the word and raising awareness
Smart AI wielded by people without your best interests in mind may be tough to counter; but you don't have to do it alone: humanity is getting wise to the threat. We don't have great tools but it'd be against the grain of history to assume we won't come up with them in time. Can't give specifics - as they say, predictions are hard, especially about the future..
We've made it through bigger more violent upheavals and we'll make it through this.
No, it's not the same as mechanization. The fundamental difference, and why I find it illogical to argue either for "AI will open up tons of new work avenues" or "AI will kill all jobs" is that it all hinges on whether AI "caps out" at an intelligence level below humans (in at least some domains/capabilities), or if it is able to surpass human intelligence at all levels.
If the former is true, then yes, there will be a lot for humans to do. If the latter is true, then no, there will be no reasonable cognitive role for humans to play (beyond perhaps for some legal, "nostalgic" reasons or what not). So whether one decides to be optimistic or pessimistic really depends on if you think AI will stay at a level below humans. I really don't know, so I'm not "taking sides" in this debate.