Here's the thing: I do actually kind of believe that. Not because I think that AI will be super intelligent and will do so out it's own volition. I think that despite AI being barely functional people will put it in charge of power generation to save a buck and it will just fail.
You think they’re dumb enough to risk throwing barely functional AI in there on the chance it’ll save a few bucks when it’s overwhelmingly more likely to cost 100x more in failures and repairs?
GPT4 has been a great teacher, I love it, but I think a lot of people and industries are already "faking it" and they'll be happy to turn any decision over to AI as long as it makes them look good in the short term. We won't see this from the outside either, like the lawyer who let GPT secretly do his work, others will do the same, but won't get caught like the lawyer did.
https://developer.nvidia.com/blog/exelon-uses-synthetic-data...
https://stanton.house.gov/2022/9/house-passes-stanton-s-dron...
Now that you say it that way....
...Enron....
I do think that that level of shortsightedness is absolutely likely in any industry, any company.
So... what's your answer to the question then? Are you just assuming we are likely to be annihilated and chill with that?
Imagine we had started taking climate change seriously in 1990 instead of... Whatever we're doing now.
This is very much not what they do.
I’m definitely on this camp. Because it is happening already, I’m seeing it before my very eyes.
My partner’s employer, a tiny ~50 people company, is already making the copywriters and graphic designers use AI.
I work as a developer for a large media company. The chairman, like everybody else not living under a rock for the past few months, has become aware of ChatGPT and wants us to integrate AI stuff in the CMS. We’re working on it, there’s hundreds of people using this CMS daily to create content.
Even the best case scenario here, where people are not laid off and just become more productive, results in the profession of many people changing a lot overnight. People who might have loved writing articles or designing stuff from scratch will soon be mere supervisors of AI’s work.
I think a lot of some guy’s post in Reddit that made it to HN: he was a 3D artist who loved his work, but was forced by his company to use midjourney, stable diffusion, dall-e or whatever and now was just doing some touches in Photoshop. He hated his “new” job.
I haven’t really used ChatGPT, Copilot and the like to generate code yet, because I don’t think I’d like it. I don’t want to correct/touch up some AI’s code as that removes all the joy.
It’s weird how roles are changing: we used to have some “AI” autocorrecting the stuff we typed and now we’re the ones correcting the AI.
Interesting times for sure, but the “large swaths of people are going to lose their jobs or have any kind of joy removed from them” I’m sure it’ll happen. And no one has a plan to deal with this nor time to come up with something. And having 90% of the workforce being “prompt engineers” is sci-fi worthy dystopia.
Lots of attempts will be made. We'll see how they pan out. The fact that attempts are being made does not mean they will all succeed. A lot of it could backfire. I would bet good money there are CEOs out there right now contemplating firing their whole support staff and replacing it with ChatGPT. And I would love to know who they are so I could short their company's stock.
This smacks of misinformation. If you're referring to that dreaded basilisk, very few people actually believe that scenario has any likelihood at all. Also, it has nothing to do with time travel anyway.
The other two things are more or less guaranteed because they're simple extensions of processes that are happening right now and can be seen all around us.
Since he's become, for many, the "face of AI risk", that gives it much more (undeserved) appearance of credibility than you imply.
(And apologies if I misled people into believing the superintelligence basilisk has time travel powers)
I don't think that's true; he's stated before he doesn't and didn't believe it was true. His reaction was more about the stupidity of posting information hazards on public fora rather than that specific information hazard being credible. However, he could be lying post-facto.
And I do agree that it kind of sucks that Yud is the "face of AI risk" because it doesn't even matter how correct he might be; a lot of people just don't like him (I'm indifferent). There shouldn't be a "Face of AI risk" because there won't be a single person that everyone like. The idea is bigger than him.
Or weird nanobot bullshit.
Is there a really boring possible outcome that leads to the destruction of humanity?
The second biggest risk is just humans being afraid. A scared human is a dangerous, irrational human.
It doesn’t have to be boring, but boring sounds more likely to me, because it’s soemthing we don’t think much about… because it’s boring.
I don't want to completely derail this thought but I am a bit older than some of the crowd on HN and I was in high school in the 90s so my perspective is a bit different I guess on this. Do Millennials think that Gen-X generation wasn't fighting against client change? I remember even further back to elementary school and just having a constant flow of ecological/environmental stuff thrown at us. It used to be deforestation, species extincition and the ozone layer ... but it was taken very seriously even then.
I mean, I just looked at the Paris Agreement [1] Wikipedia and the "Lead Up" section mentions:
The UN Framework Convention on Climate Change (UNFCCC), adopted at the 1992 Earth Summit is one of the first international treaties on the topic.
At any rate, you may find yourself in 30 years faced with the same kind of attitude ... like people in 2050 saying "imagine people in 2023 actually took climate change seriously!" and you might then feel the same as I do.
Even boomers were taking the risks of climate change very seriously.
There was quite a lot, including public demonstrations, etc. Not so different from now, really, except that now the problem has become urgent enough that it's harder to be a denier.
Although I was a child then, I even remember a lot of serious agitation happening in the 70s.
The Bernoulli Equation, which gave us the basic understanding of how lift can be generated with an airfoil, was developed in the mid-1700s. [1]
We don't have any fundamental understanding today of how an AGI could be built. We have a bunch of interesting ideas, with varying degrees of evidence for them, but we don't have the same kind of solidly-established scientific foundation for understanding intelligence and the human brain today that we did for understanding lift in 1900.
"Some people didn't understand the science, and so called it fiction" isn't a very strong argument.
No one has provided us with an ironclad scientific foundation for AGI. Some people keep declaring very firmly that it must logically be possible, but even if there are no gaps in your logic whatsoever, that's not the same thing as providing equations for how it would work.
Anyone who was working with transformers could have seen ChatGPT on the horizon, it wasn’t surprising at all that scaling an autoregressive model can result in something seemingly intelligent.
Where is 1 in 10 coming from? Is this a ‘gut feeling’ because one does not understand LLMs or is this factually based?
What is my estimate for AI being an existential risk in the next couple of decades? Depends on if we find something that actually resembles AGI which is impossible to predict. Based solely on current technology + scaling I would personally put the chance at essentially 0%.
It's a totally made up example. Its purpose was to engage you in a conversation about what's reasonable if you believe an existential threat is possible in the near term. It doesn't seem you are willing to engage that possibility.
> Anyone who was working with transformers could have seen ChatGPT on the horizon
So, who did?
> [LeCun is] highly critical
I did a quick search and found this[1] trash-tier article explaining his views summed as:
> He thinks the will to dominate and intelligence are separate things. Orangutan do not have a desire to dominate but are intelligent. They are territorial.
I have absolutely no idea where this distinction comes from. Being territorial is a form of domination, it's just not the same as expansionist behavior. If an AI is territorial, and the Earth is its territory, it doesn't need to be expansionist to annihilate threats on Earth.
Anyway, LeCun's former colleague Hinton was on NPR yesterday telling the masses that he thinks we could have dangerous AI in less than a decade. It's fair to say there's wide disagreement. I don't think it's fair to say that handwaving means we shouldn't be worried about the problem.
Finally, your question:
> What is so impressive about ChatGPT that it poses an existential threat?
This is a bit of a logical misstep, or I was unclear previously. I don't think anyone sees ChatGPT or anything directly related to it as an existential threat. Rather, the rate of change in capabilities of LLMs is worrying, because if we see a similar rate of change in systems with more agency, it could be a threat. Or, in the words of the article about LeCun's belief:
> Lecun says that in a few year LLM (Large Language Models) will go away and replaced with systems that will be guidable to desired goals.
So... he is predicting exactly the same situation that everyone is afraid of, and he's asserting that it will work out fine. Somehow.
The point is not that any one particular situation will occur. It's that there are an unknown number of factors in what makes an AI "smart enough" and in what makes it "misaligned". There are hypothetical scenarios where a hyperintelligent AI creates goals for itself that necessitate our extinction, and we don't know how those scenarios come into existence or how to stop them. Assuming they won't happen is just burying our heads in the sand.
> essentially 0%.
I sure hope you're right, but I don't think that's a common position to hold.
[1] https://www.nextbigfuture.com/2023/05/metas-yann-lecun-is-co...
> The point is not that any one particular situation will occur. It's that there are an unknown number of factors in what makes an AI "smart enough" and in what makes it "misaligned". There are hypothetical scenarios where a hyperintelligent AI creates goals for itself that necessitate our extinction, and we don't know how those scenarios come into existence or how to stop them. Assuming they won't happen is just burying our heads in the sand.
It’s not assuming it won’t happen or burying our heads in the sands, it’s calling for measured responses.
My point is that we can worry and have the discussion without being alarmist and enacting premature regulation.
Autonomous AGI, whenever that happens, absolutely poses a threat. But it’s extremely exaggerated to think what we have today, or even what is known on the horizon, represents that.
People assume advancements will continue to happen at their current pace. The last 6 years of advancements has been the product of scaling and refining self-attention + transformers from 2017/8.
> so who did
I’m not sure what you’re getting at. No one had the will/resources to burn billions on a language model that has ??? business plan, doesn’t mean no one saw it coming.
I can appreciate that. I haven't been listening to him for that long; having no idea what situation you're discussing, I'll take it at face value.
> My point is that we can worry and have the discussion without being alarmist and enacting premature regulation.
That's pretty reasonable. However, I will reiterate that this:
> it’s extremely exaggerated to think what we have today, or even what is known on the horizon, represents that.
is not what I'm asserting. It's the unknowns which are dangerous. We're inventing something new in almost every way imaginable. It's not like we're talking about a new theoretical steelmaking process which could upset the economy, we're talking about the potential to build a new life form with godlike abilities. Alarm seems warranted to me. Sure, it might not happen soon... but also, it might happen soon. Shouldn't we be trying to prevent it, or control it, before it happens?
> People assume advancements will continue to happen at their current pace.
I think people assume advancements will accelerate, actually.
You can conjure fantastic (positive or negative) unknowns about anything. Unless, for the negative case, you have a concrete reason for believing some class of net-adverse events are probable along some particular route and some concrete basis for evaluating whether alternative actions will both mitigate that risk and not create greater new ones, you have no rational basis for acting based on those unknowns.
It's not something conjured, though. Unless you believe superhuman AGI which can set its own goals is impossible, or that it's possible but it can't possibly have goals that pose a threat to humanity, then it's reasonable to believe we can invent superhuman AGI which threatens our own existence. Technologically, I think it's possible to do so.
So if we agree that this is possible, can we also agree that humanity is rushing to reach that point as rapidly as possible? I think we are. Even though we're mostly focused on LLMs right now, and even if LLMs are not a path to AGI (which I agree they probably aren't on their own), it's not like there's no research happening on other pathways. Additionally, we can't be sure that current LLM research won't contribute to AGI - maybe the real path is a synthesis of multiple forms of AI.
So there are some very specific unknowns standing between us and possible doom. We're not making up fantasies wholesale.
- Dangerous AI is possible - We're racing to create it as soon as we can - We don't know how to tell if we've crossed a threshold into having dangerous AI
That last point is an known unknown which is extraordinarily dangerous. It's like if you're blindfolded and moving your hand toward a running chainsaw. You know that if you move your hand far enough it will be shredded, you know you're moving your hand forward, you just don't how close your hand is to the chainsaw. You should be concerned in that situation.
It is something believed without any evidentiary basis which enables rationally weighing the actual risk or evaluating mitigation strategies so as to weigh their probable benefits against their probable costs and the associated changed in the overall risk profile; I'm not suggesting literal magical incantations when I say “conjured”.
> Unless you believe superhuman AGI which can set its own goals is impossible,
Depends what you mean by “set its own goals”, I could certainly see an argument that the natural intepretation is impossible in a universe fully governed by physical laws, whether the intelligence is artificial or not. But I don’t actually know that this is important, or even that the mucj weaker assumption that it’s goals might not exactly represent human intent (which is definitely possible of AGI itself is) is important, since human intention can itself be an existential risk.
> or that it's possible but it can't possibly have goals that pose a threat to humanity
Its definitely possible for an intelligence, even a human one, to have goals that are a threat, including an existential one, to humanity. That’s an existing x-risk without superhuman artificial intelligence. It may even be mitigated by superhuman artificial intelligence (as might other x-risks like “earth gets smashed by an asteroid”), if it is possible for such to exist without goals that are antithetical to human survival, in which case delaying superhuman AI could increase existential risk.
But, while this is all nice and conceivable, we have no way of balancing the net impact, positive or negative, of any course on AI to existential risk due to friendly or hostile superintelligences.
> So if we agree that this is possible, can we also agree that humanity is rushing to reach that point as rapidly as possible?
No, I can't agree with that, as it would require that every resource that could advance superhuman AI, and specifically dangerous* superhuman AI, was being directed as effectively as possible to that end. I don’t think we are anywhere close to that. Because AI has some mindshare beyond immediate economic utility, we’re probably heading toward AGI, if it is possible and if our current understanding of AI isn’t so wrong as to make our work irrrlevant even in eliminating bad routes, slightly faster than immediately-apparent economic incentives would promote, but nowhere close to as rapidly as possible.
But that's exactly true of either position you take. Imagine I'm arguing that aliens probably exist and you're arguing they probably don't. There is no such thing as a belief based on evidence in this case; we only have logical reasoning and assumptions and beliefs.
> Depends what you mean by “set its own goals”, I could certainly see an argument that the natural intepretation is impossible in a universe fully governed by physical laws, whether the intelligence is artificial or not.
This feels like moving the goal posts... all it needs to be able to do is make decisions that are dangerous to humanity. I felt that definition was obvious. All it needs to do do is mimic an oil executive, because, as you say, human intent is also an existential risk. Imagine an oil executive that could get the upper hand in all possible negotiations and desired to make the entire global economy subservient to the needs of oil production.
> delaying superhuman AI could increase existential risk.
Oh, yeah I totally agree that we can't truly know one way or the other. But it kind of feels like deciding to invent and deploy nuclear bombs as a deterrent to war. Did it end war? Absolutely not. Many would argue it didn't even that specific war. We are of course fortunate that nuclear war has not yet destroyed civilization, but a) there's still time, b) we aren't necessarily better off than if we had no nukes, and c) not all risks are equal - AI could be more dangerous.
> as it would require that every resource that could advance superhuman AI
Actually, you're right. I was definitely overstating our progress and didn't think through that argument clearly. But maybe it would be safer if we were devoting all global resources toward a coordinated AGI alignment and development effort...
https://m.youtube.com/watch?v=2HMPRXstSvQ
We are currently facing the biggest radiologist shortage in the last 30 years.
> It’s the unknowns which are dangerous.
Electricity, the Industrial Revolution, the internet, gene editing/bionengineering also came with unknown existential risks.
> We’re inventing something new in almost every way imaginable.
> I think people assume advancements will accelerate, actually.
This is highly debatable.
> we're talking about the potential to build a new life form with godlike abilities.
Evidence? This has been stated in sci-fi books from before I was born and I don’t see any proof that we’re building something remotely close to this.
Did they? I've never heard about electricity creating fears about human extinction.
The industrial revolution does pose an existential risk: climate change.
Did bioengineering garner worries about existential risk? It certainly had and has a lot of people worried about various risks. And then lots of countries banned human cloning: https://en.wikipedia.org/wiki/Human_cloning#Current_law
> Evidence? This has been stated in sci-fi books from before I was born
Yes, many sci-fi authors of the past were very forward-thinking and predicted future technologies... that's sort of the point.
> and I don’t see any proof that we’re building something remotely close to this
Again, this is a fundamental misunderstanding of the risk.
1. We are building AI. AI is possible.
2. We are advancing AI. AI advancement is possible.
3. We have not built superhuman AGI. But superhuman AGI is probably possible.
The fact that AGI doesn't exist today is frequently argued as a reason we won't have it in the near future, but it's a non-sequitur. We will definitely have more advanced AI in 5 years than we do today, and we can't say what that AI will be capable of. Therefore, it's possible it will be AGI.
And yet, until much later when we understood the mechanisms and could evaluate mitigations, any restriction on industrialization based on purely speculative “existential risk” that we could neither adequately explain nor provide a factually-grounded framework for evaluating relative risks of alternatives would mostly likely not have made things better, and could have made them (and even the real existential risk, as well as experienced conditions at the time) much worse.
It’s true that once we understood the concrete mechanisms and could evaluate alternatives, not mitigating the risk has been irresponsible, but that was…significantly later than the Industrial Revolution itself.
Really? I don't agree that strictly limiting plastic production and carbon emissions 100 years ago would have had zero or negative effect on the timeline of apocalyptic climate change or any other existential threat. It might have made us more vulnerable to natural pandemics... but it also would have slowed the emergence of natural pandemics and virtually prohibited artificial pandemics.
Neither do I, but then, 100 years ago doesn't clearly meet the description, anyhow. The discovery of the greenhouse effect and CO2’s role in it was after not very long after the dates usually used for the end of the (first) industrial revolution (which is usually what is meant without modifiers, not the second—fourth industrial revolutions) and well over 100 years ago.
Replace "100 years" with whatever timeframe is perfect for you and I'll make the same argument. Please respond to that instead of nitpicking.
At any time before we had the information to understand thr mechanism of problems and evaluate mitigations, its very unlikely that we would choose tbe right mitigations. Sure, if we choose thr mitigations that we would choose based on information today before we had that information, that would be great, but its at least as likely that any attempt to mitigate risk of unknown mechanism vefore we had that infornation would have done exactly the wrong thing.
Basically, “Well, I have information today by which I can design a policy which would have been beneficial if implemented before that information was available” is not an argument that people seeking to mitigate a vaguely imagined risk without any understanding of its mechanism would have the means to design a productive intervention.
I'm saying that with what we know now, it's clear we could have made different choices that would have worked out better for us earlier in industry.
You're saying that the information at the time made it impossible to correctly make the optimal choice. (By that I mean, making the optimal choice would have appeared to be the wrong choice, based on information available.)
I don't agree with your point; I think we knew that burning carbon was going to change the atmosphere for the worse and we could have been more careful by simply burning less carbon. However, I concede that it's unknowable what science and technology would look like if we had done that.
In analogy to AI, that is very much a point of conversation in alignment/safety discussions. Should we wait until we have better tools to figure out alignment, since it's so insanely difficult to figure out now? Frankly, nobody knows for sure the answer to that question either. My contention is that you (and others) are assuming the answer to that question is yes, but that's not necessarily correct, even based on all the information and analogies we have now.
"Dumb" AGI is yet to be proven theoretically possible let alone superhuman.
In a way, all of those things have led to where we are now. And might I point out that a form of bioengineering may have caused the latest global pandemic? Not taking these things seriously because "they haven't killed us all yet" seems a little shortsighted.
> And might I point out that a form of bioengineering may have caused the latest global pandemic?
Even accepting this hypothesis as true, is the answer that we should have regulated cell culture in the 50s when HeLa cell culture became a thing?
Would that have prevented a nation-state from potentially causing a pandemic?
I mean, we should have better regulated carbon emissions generations ago. We'd have a lot more time to deal with it, because we haven't been able to deal with it in the century+ we've known about it. And AI extinction risk is likely to move faster than carbon-related climate change.
I don't pretend to have the answer of what the balance is for oversight and regulation to optimize safety and innovation - but the first step for those responsible is to recognize the potential dangers and unknowns, and produce plans that can be discussed and debated.
> Even accepting this hypothesis as true, is the answer that we should have regulated cell culture in the 50s when HeLa cell culture became a thing?
How about gain-of-function research? How about nuclear weapons? I lean libertarian but also recognize we have a responsibility to safeguard humanity.
I think that, specifically, is what is not in the cards.
Human-level intelligence can exist. You, a natural human-level intelligence, cannot dispute this.
I expect that it's possible for superhuman intelligence to exist. I don't think I need to prove that it's physically possible for a superhuman intelligence, natural or otherwise, to exist. Do you think it's not possible? If so, I'd say you need to prove why you think human-level intelligence is literally the pinnacle of intelligence that is physically possible in the universe (especially when narrow AI are already superhuman at many tasks).
Artificial intelligence exists and can be created by humans. This could be disputed if you wanted to gerrymander some really specific borders around what "intelligence" is, but if we don't agree that artificial intelligence exists today, just say than and we'll be done talking.
AGI does not exist. I assert that it can exist, because narrow AI can be improved. Do you believe it's not physically possible for human-level AGI to ever exist? If so, you need to prove to me why we can create narrow AI that is superhuman but cannot ever synthesize them into a human-like intelligence. (And also please give me the scale by which you're measuring an alien brain against ours.)
And if we both agree on all the above, then I take my assertion further: It is possible for humans to create superhuman AGI. It's really not a big leap from human-level AGI (in fact, I think that if it's possible to create AGI which matches humans exactly, it's probably easier to surpass human levels than to match them exactly, because it's hard enough to measure human intelligence).
So if it's physically possible for us to create superhuman AGI, what exactly am I proving? Do I need to invent the technology in order to prove that the concept is possible?
Of course.
> I expect that it's possible for superhuman intelligence to exist.
I agree.
> Artificial intelligence exists and can be created by humans.
This is where we disagree, although as you say, it depends on what you mean by "intelligence". I think in the context of this discussion, we're talking about some sort of superhuman intelligence with its own will and perhaps consciousness.
In the largest possible sense, it's probable that creating such a thing can be done. Where we probably disagree is that I think we're a very, very long way from being able to do that.
Nothing we have now makes me suspect that we're anywhere near that sort of thing.
> It is possible for humans to create superhuman AGI. It's really not a big leap from human-level AGI
Agreed. But I assert that we're nowhere near being able to create human-level AGI right now.
> in fact, I think that if it's possible to create AGI which matches humans exactly
I think this isn't possible at all unless you also create an entire human body.
> what exactly am I proving?
The assertion that AGI is a thing that is likely to happen soon. Personally, I think it's vanishingly unlikely to happen within our lifetimes, and very unlikely within the lifetime of our children.
Freaking out about the risks of AGI right now is a serious overreaction. Calmly pondering it would be more appropriate.
So the fundamental disagreement is over how close we are to dangerous AI.
If you're right, we might delay useful AI by decades or centuries being overly cautious. Useful AI could either alleviate untold suffering, or it could cause untold suffering and inequality by misuse.
If I'm right, we could go extinct for lack of caution.
And given that we really don't have any way of knowing with any amount of certainty how close we are to AGI, you see why people are concerned, right? Calmly pondering feels like a serious underreaction. Vigorous, frequent, and open debate would be more appropriate. I'm not saying let's turn off all our computers for a decade.
When we improve LLMs, won't they just be a fancier auto-complete?
They're not a "new life form with godlike abilities".
They sure are fooling a lot of people though.
Not necessarily, and if so, only if your first postulate is true. Which is really difficult to say because "fancy auto-complete" is too vague to argue. Human speech can also be described as fancy auto-complete.
Luckily we don't have to argue that because I specifically am saying it doesn't have to be LLMs. You've missed the point entirely.
Or are we talking about a variation of the system we have right now, where someone could use the AI as a part of a control system and then the "algorithm" doesn't operate the way we want and causes an outage? Because that happens all the time without LLMs.
I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!"
No, but that's actually irrelevant. The reason it's irrelevant is because of the answer to:
> I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!"
Nobody is saying the second thing (that I'm aware of). This is the chain of reasoning:
1. "LLMs are an amazing technology [which advanced at an unexpected rate]"
2. Because LLMs advanced at an unexpected rate, this indicates an acceleration of research
3. Such acceleration could conceivably happen or be happening now in other forms of AI
4. The path to AGI is COMPLETELY unknown.
5. The origins and structures of consciousness and agency in intelligent systems are completely unknown
6. It is impossible for humans today to know if they've crossed a threshold into creating AGI, or a new form of intelligent alien life.
7. AGI is inherently alien and there's no reason to expect it to think the way we do.
8. Even human goals and systems are often (or even mostly) misaligned - look at any for-profit corporation or totalitarian state for an extreme example of how their existence harms people in many cases (pollution, murder, exploitation, discrimination, etc)
So, I think I've covered the bases. I'll sum it up like this: We don't know what AGI looks like, we can't expect it to have our best interests in mind because it's an alien that's smarter than us, and we don't know when or how we'll achieve AGI. So unless you don't believe superhuman AGI is possible at all, we're in a very scary time. We have know way of knowing if it will take 5 years or 100, but we also know that people all over the world are racing to make it happen as soon as possible.
We can't know if we've started the process to summon alien entities from another dimension through a rift in spacetime, either.
On the other hand, we don't have any scientific reason to believe that superhuman AGI isn't possible. We know AI is possible, because we have it right now. What's preventing it from getting smarter than us? The human soul?
To clarify, most believe we likely have started on the path to superhuman AGI. We just don't know how or when we'll reach an arbitrary threshold where it becomes dangerous. There are many finish lines and we can't see any of them, but we're running as fast as we can down the path of least resistance and hoping we don't cross a finish line we don't like.
Sorry, what? According to which scientific theories is AGI possible, and what principles are those theories based on? What testable hypotheses say that it is, and which experimental evidence has been produced in favour of AGI being possible?
Human-level intelligence is possible. See: humans
Do you think it's physically impossible to recreate human-level general intelligence artificially?
Do you think it's physically impossible for any intelligence to be superhuman, even a natural intelligence?
Unless you answered "yes" to either of the above, why shouldn't it be physically possible for humans to create superhuman AGI?
Is there any physical law whatsoever that even hints that intelligence should be able to create AGI at its own level, but in no way more intelligent?
I never said there was a physical theory that predicts superhuman AGI. But in contrast, there are other concepts such as time travel which are prohibited by certain theories we have. It seems very unlikely for time travel to be possible. Superhuman AGI is not prohibited from existence.
I have no idea. I have no basis for concluding it's either possible or impossible.
> Do you think it's physically impossible for any intelligence to be superhuman, even a natural intelligence?
No, I think a godlike being could materialise an entity with superhuman intelligence, or natural selection could evolve one (perhaps even on earth -- perhaps it already has!).
> Unless you answered "yes" to either of the above, why shouldn't it be physically possible for humans to create superhuman AGI?
Even assuming I believe both the above are possible that doesn't mean I think it's possible they could occur on planet earth within any meaningful time frame. I also think that interstellar travel is compatible with the laws of physics but just because we've sent humans to the moon and robots to mars we have no idea whether interstellar travel is feasible or when it might occur. For me the response to the advent of GPT is like people saying "wow, we've sent men to walk on the moon -- if we keep going people will soon be visiting mars/alpha centauri!".
There's no rational basis (scientific, logical, anything) for knowing whether AGI is more akin to mars in this analogy (and we're really quite close) or to alpha centauri (and we're no where near).
> No, I think a godlike being...
Superhuman intelligence is a reality in many fields already: chess, Go, self-driving cars at certain conditions, calculations and many more. This is not necessary to have it for everything at all.
And it's not like this estimate is based on a lot of concrete reasoning. So yeah, I would expect it to fluctuate wildly at an unexpected development in the field initially, then settle down around a slight change. Which is basically the definition of a hype cycle.
I believe that AI can't possibly pose an existential risk in the next decade or two. I believe AI poses a great risk, but an economic one, and not an existential one.
> Actually I'd love to know what your estimate actually is for AI becoming an existential threat
My estimate is: never. At least not in the form of some superintelligent AGI.
This is a totally unreasonable take, unless you believe that AI can't possibly pose an existential risk within the next couple decades or so. Actually I'd love to know what your estimate actually is for AI becoming an existential threat - does 30 years sound short to you? Because to a lot of AI experts, 30 years was their estimate before ChatGPT, and could be considered wildly optimistic.
You clearly don't, but imagine you felt that AI had a 1-in-10 chance of shutting down all power generation on earth. This would collapse civilization. Would you be worried about it? As a reminder, you're still allowed to be worried about other things as the same time. It's a simple y/n.