Realistically what actually ends up happening imo, we get human level AGI and hit a ceiling there. Agents replace large portions of the current service economy greatly increasing automation / efficiency for companies.
People continue to live their lives, as the idea of having a human level AGI personal assistant becomes normalized and then taken for granted.
That is a hard limit on intelligence, but neural networks can't even reach that. What is the actual limit? No one knows. Maybe it's something relatively close to that, modulo physical constraints. Maybe it's right above the maximum human intelligence (and evolution managed to converge to a near optimal architecture). No one knows.
As far as we know, it's impossible to model any part of the universe perfectly, although it usually doesn't matter.
One million Von Neumanns working on that 'problem' is not something I'm looking forward to.
> People continue to live their lives
Presumably large numbers of those people no longer have jobs, and therefore no income.
> we get human level AGI and hit a ceiling there
Recently I've been wondering if our best chance for a brake on runaway non-hard-takeoff superintelligence would be that the economy would be trashed.
Half-joking. In seriousness, something like a UBI will most likely happen.
It's impossible to run out of work for people who are capable of it. As an example, if you have two people and a piece of paper, just tear up the paper into strips, call them money and start exchanging them. Congrats, you both have income now.
(This is assuming the AGI has solved the problem of food and stuff, otherwise they're going to have to trade for that and may run out of currency.)
- A simple calculator can beat any human at arithmetic
- A word processor with a hard disk is orders of magnitude better than any human at memorizing text
- No human can approach the ELO of a chess program running on a cheap laptop
- No human polymath has journeyman-level knowledge of a tiny fraction of the fields that LLMs can recite textbook answers from
Why should we expect AGI to be the first cognitive capabilities that does not quickly shoot past human-level ability?
Changed my mind. Think you’re right. At the very least, these models will reach polymath comprehension in every field that exists. And PhD level expertise in every field all at once is by definition superhuman, since people currently are time constrained by limited lifespans.
Increasingly this just seems like fantasy to me. I suspect we will see big changes similar to the way computers changed the economy, but we will not see "capital as we know it become basically worthless" or "the modern economy and society around it collapse overnight". Property rights will still have value. Manufacturing facilities will still have value. Social media sites will still have value.
If this is a fantasy that will not happen, we really don't need to reason about the implications of it happening. Consider that in 1968 some people imagined that the world of 2001 would be like the film 2001: A Space Odyssey, when in reality the shuttle program was soon to wind down, with little to replace it for another 20 years.
I was with you on the first two, but the second one I don't get? We don't even have AGI right now, and social media sites are already increasingly viewed by many people I know as having dubious value. Adding LLM's to the mix lowers that value, if anything (spam/bots/nonsense go up). Adding AGI would seem to further reduce that value.
> social media sites are already increasingly viewed by many people I know as having dubious value
I think we have all been saying this for 15 years but they keep getting more valuable.
We would probably get the ability to generate infinite software, but a lot of stuff, like engineering would still require trial and error. Creating great art would still require inspiration gathered in the real world.
I expect it will bring about a new age of techno-feudalism - since selling intellectual labor will become impossible, only low value-add physical or mixed labor will become viable, which won't be paid very well. People with capital will still own said capital, but you probably won't be able to catch up to them by selling your labour, which will recreate the economic situation of the middle ages.
Another analogy I like is gold. If someone invented a way of making gold, it would bring down the price of the metal to next to nothing. In capitalist terms, it would constitute a huge destruction of value.
Same thing with AI - while human intelligence is productive, I'm pretty sure there's a value in its scarcity - that fancy degree from a top university or any sort of acquired knowledge is somewhat valuable by the nature of its scarcity. Infinite supply would create value, and destroy it, not sure how the total would shake out.
Additionally, it would definitely suck that all the people financing their homes from their intellectual jobs would have to default on their loans, and the people whose services they employ, like construction workers, would go out of business as well.
Indeed. Or as I’ve said before: a return to the historical mean.
Because if AI has one "killer app", it's to control robots.
Dark pun intended.
I see it as capital becoming infinitely more valuable and labor becoming worthless, since capital can be transmuted directly into labor at that point.
In this case, you need capital to stockpile the GPUs.
If ASI and the ability to build robots becomes generally available and virtually free (and if the exponential growth stops), the things that retain their value will be land and raw materials (including raw materials that contain energy).
If ASI is reached but is controlled by only a few, then ASI may become the most important form of capital of all. Resources, land and pre-existing installations will still be important, though.
What will truly suffer if the ASI's potential is realized, is the value of labor. If anything, capital may become more important than before.
Now this MAY be followed by attempts by governments or voters to nationalize the AI. But it can also mean that whoever is in power decides that it becomes irrelevant what the population wants.
Particularly if the ASI can be used to operate robotic police capable of pacifying the populace.
I also don't think there's any way the governments of the world let real AGI stay in the hands of private industry. If it happens, governments around the world will go to war to gain control of it. SSI would be nationalized the moment AGI happened and there's nothing A16Z could do about it.