I'd argue the dismissals are a lot more handwavy.

The simplified argument is:

- Superintelligent AGI that can modify itself in pursuit of a goal is possible.

- If that AGI is not aligned with human goals it very likely ends in the end of humanity.

- We have no idea how to align an AGI or even really observe what it's true state/goals are. Without this capability if we stumble upon creating an AGI capable of improving itself in such a way that leads to super intelligence before we have alignment, it's game over.

##

For point 1, that seems like the consensus view now (though it wasn't until recently). I think it seems obvious, but my general arguments would be humans aren't special, brains are everywhere in nature, biology is constrained in ways other systems are not (birth, energy usage, etc.)

For point 2, in pursuit of whatever its goal is, even a 'dumb' goal that happens to satisfy its reward functions, humanity will likely either try to stop it (and then be an obstacle) or at a minimum will just be in the way - like an anthill destroyed in the construction of a dam.

Point 3 is not controversial.

The dismissals from Tyler Cohen and Pinker are mostly just relying on heuristics which are often right, but even if they're right 999 out of 1000 times, if that 1 in 1000 error is the end of humanity, that's pretty bad. Most of the time a disease not a pandemic, but sometimes it is. I've read some of what Pinker has written about it, he doesn't understand EY's arguments (imo). Cohen's recent blog post could be summarized as "we'll likely see an end to peacetime and increasing global instability, might as well get AGI out of it". Just because things don't usually result in human extinction doesn't mean they can't.

I can't find a flaw in this argument. That is a bit scary actually.

With downcast eyes and heavy heart, Eliezer left Sam Altman

Some years go by, and AGI progresses to assault man

Atop a pile of paper clips he screams "It's not my fault, man!"

But Eliezer's long since dead, and cannot hear Sam Altman.

https://astralcodexten.substack.com/p/turing-test

Point 2 does not logically follow from point 1. That's the entire problem here.

Between point 1 and 2 there must be quite a few other steps, causally linked, otherwise it's just a massive imaginary leap based on assumptions that nobody is explaining.

> The dismissals from Tyler Cohen and Pinker are mostly just relying on heuristics which are often right, but even if they're right 999 out of 1000 times, if that 1 in 1000 error is the end of humanity, that's pretty bad.

EY is not arguing that the end of humanity is merely possible. He is arguing that it is obviously the most likely outcome and will almost certainly happen a very short time after AGI is invented. That's a much harder case to make.

Agreed point 2 seems to be the crux.

An important concept behind this is Omohundro's Basic Drives. Any maximizing agent with a goal will try to acquire more resources, resist being shut off, resist having its goal changed, create copies of itself, and improve its algorithm. If it is possible to maximize its goal in a way that will not guarantee humanity's flourishing, then we all die, guaranteed.

If you want something more spelled out, I'll refer to Tim Urban's explanation. It's quite long, but is about as detailed as you can ask for. https://waitbutwhy.com/2015/01/artificial-intelligence-revol...

Yes point 2 is the weak one.

Aren't humans/biological life a counterexample? Simple bacteria are clearly maximizing agents, and cyanobacteria did infact almost destroy all life on earth by filling the entire planet with toxic oxygen.

We've known how to improve ourselves via selective breeding yet vanishingly few humans are proponents. We and and other intelligent animals have a wide variety of goals, and even animals sometimes share food across species.

The evidence just doesn't seem to support this concept of Basic Drives. If anything, the evidence (and common sense) seems to suggest that the more intelligent the organism, the more easily and more often it ignores its basic drives.

Natural selection and biology have different constraints that limit things (and even then often selection can cause extinction)

Intellectual capability and alignment are orthogonal - the former doesn’t get you the latter for free.

Most people new to this issue don’t intuitively grasp this at first.

What evidence do you have that intellectual ability and alignment are orthogonal?

Remember that ML models are also subject to selection. If they're not useful/beneficial to us, we stop running them.

For clarity I think EY has made that case well, I think it's the heuristic Cohen and Pinker are relying on that's right most of the time but can still be catastrophically wrong (everything so far has not lead to human extinction, but that was also the case for all animals that went extinct prior to their extinction event).

Some others in this thread have linked to stuff (it's variations/examples on the paperclip maximizer argument).

I'd be curious why John Carmack thinks these risks are unlikely (he thinks fast take off is not something to worry about) - is that because he thinks we'll get some sort of trainable AGI first or something else? There are also some other substantive disagreements here: https://www.lesswrong.com/posts/wAczufCpMdaamF9fy/my-objecti...

John Carmack mentions this in the Lex Fridman Podcast, basically the argument is that AGI performance characteristics will be similar to that of LLM (which is another way of saying that AGI will NOT be hyper efficient, P != NP), and that the performance characteristics pose a problem for fast takeoff. The bottleneck to performance in training these models is GPU memory bandwidth when the entire model fits inside VRAM, which on modern cards is 1TB/s, when the model cannot fit in GPU memory, the performance now instead scales along PCIE, which is currently 32GB/s. AGI (or an LLM) attempting to replicate across mobile devices, or desktops, will be severely hamstrung by the network connection. So using all of planet Earth's computing resources is not necessarily better than say a single data center in this respect. The second piece of the argument here is essentially saying that a data center is also not enough, or rather, not enough in a detectable timeframe. Could AGI hack the entire data center for an entire month to perform it's training (and then execute it's strategy enough to gain say nuclear codes?). Unlikely. Is say 8 hours of training using an entire data center enough to go from intelligence to super intelligence? Intuition says no. I expanded a bit upon what I believe his argument to be, I definitely recommend watching that part of the podcast episode.

Here's my take on the implications of not having fast takeoff, a secretly antagonistic AGI will be constrained to cooperation, and slowly leeching compute resources towards its goal until it is able to aquire enough resources to confidently sprint towards the inflection point. This could mean teaching us how to build better semiconductor foundries, how to create nuclear fusion energy sources, how to educate our youth to better fit into those jobs, slightly alter our culture's value systems over time via astroturfing to be more sympathetic, cooperative, trusting, and less vigilant towards these AI systems, how to build more robust computing systems so we stop needing to detect when something goes wrong (because it already has 99.9999% uptime).

I believe slow takeoff is actually worse in some sense, because it is a hell of a lot harder to detect, and humans tend to get complacent and apathetic when something "just works" for decades, even if it has been plotting since the beginning.

Thanks - I saw he was on there, but I find lex insufferable as an interviewer so tend to avoid the pods (despite the great guests)

Who said it followed? That's a strawman. To dumb it down even further for you, I guess...

There are brains, therefore there can be created an artificial super-brain. Artificial super-brain might have goals which don't align with human brain and we will have no way to understand or control the situation.

Two unrelated facts, which together mean that we should be careful with experimenting with the science working towards super-brains. Just a single super-brain could end us.

I disagree with your argument, especially for point 1: these systems are massively constrained. The hardware they run on is fragile requiring massive amounts of power and tightly controlled environments. They don't have any means of replicating themselves (it can't run on arbitrary systems). The datacenters also have massive bandwidth between nodes - even if you could run 'it' on all the personal computers and phones in the world, it will likely struggle. Sure we can compress recent llm's down to being able to run on consumer hardware - but these things cant introspect, reason or adapt. They are completely static models and very far removed from anything agi. A lot of the progress in compute power in the last few years also come from changing representation: moving from foat32 to float 16 and more recently to float8. The silicon itself can only get so much better. It's not super obvious to me that we will have chatgpt4 like models on consumer hardware soon let alone solve true agi. Why don't we have true level 5 self driving cars yet?

We cant even figure out how to simulate a flatworm - and the connectome is solved.

It's a narrower argument that I'm making - it's not about LLMs or implementation (can put those details aside), it's about possibility that a superintelligent AGI could be created, there's nothing magical about biological intelligence that would prevent it.

How or how difficult, or when are all questions that follow from that first premise (that it is possible). I don't really make strong claims about any of the implementation details beyond it being possible. Though again, what we're seeing doesn't look like failure to me.

If you think it is possible though, then there's a strong argument that trying to work on alignment now is probably a good idea because people are notoriously bad at predicting when advances will happen (and the downside risk of unaligned superintelligent AGI is likely very bad).

https://intelligence.org/2017/10/13/fire-alarm/

> "Two: History shows that for the general public, and even for scientists not in a key inner circle, and even for scientists in that key circle, it is very often the case that key technological developments still seem decades away, five years before they show up.

"In 1901, two years before helping build the first heavier-than-air flyer, Wilbur Wright told his brother that powered flight was fifty years away.

"In 1939, three years before he personally oversaw the first critical chain reaction in a pile of uranium bricks, Enrico Fermi voiced 90% confidence that it was impossible to use uranium to sustain a fission chain reaction. I believe Fermi also said a year after that, aka two years before the denouement, that if net power from fission was even possible (as he then granted some greater plausibility) then it would be fifty years off; but for this I neglected to keep the citation.

"And of course if you’re not the Wright Brothers or Enrico Fermi, you will be even more surprised. Most of the world learned that atomic weapons were now a thing when they woke up to the headlines about Hiroshima. There were esteemed intellectuals saying four years after the Wright Flyer that heavier-than-air flight was impossible, because knowledge propagated more slowly back then."

I hear you - its just that I dont think that it neccesarily follows that an AGI in whatever form will necesarily be unconstrained or free to improve itself. I also happen to think that the biological systems are in-fact rather special and hard to replicate - especially in terms of efficiency and resillience. Just because some implementation of a universal turing machine can simulate intelligence doesn't mean it can do it well enough to survive the real world.

Finally, to me, nuclear reactions are kind of the opposite of AGI: I think it's vastly easier to blow something up (increase entropy) than to create an intelligence capable of understanding and improving itself (decreasing entropy - possibly at an accelerating rate).

Yeah - it doesn't necessarily follow, but the behavior of people working on the technology doesn't inspire a lot of confidence.

Even if people were trying to constrain its access seriously I think that's unlikely to work (hard to contain a superintelligence that wants to not be contained - it's possible to trick a chimp to go into a room and the delta in intelligence between a human and a superintelligence is way bigger than us and chimps).

Instead I mostly observe people not really understanding the e-risk argument, focused mostly on small stuff that doesn't matter as much (AI language, bias). The people developing the tech connecting it to the internet and expanding capabilities, giving it access to code/training ability to write code, preparing massive datacenters for it, etc.

All of this without really understanding how to align it or what it's actual internal goals really are.

> "Just because some implementation of a universal turing machine can simulate intelligence doesn't mean it can do it well enough to survive the real world."

This could be true, but I would bet against it - and the downside risk of being wrong (potentially complete extinction) means it seems worth being way more cautious about it than we (humanity broadly) are observed being.

Nobody says that it necessarily follows, but it's a quite trivial step once you have the AGI, hence pointless that you bring up it as an argument.

While I appreciate you find my argument persuasive - imo this style of comment will do more to alienate those replying to me that disagree than to persuade them.

They're likely irrelevant anyway. We're all shouting into the void.

What a horrible argument from this guy, whoever he is. And ending with a whataboutism just for the sake of it. I hope nobody regards him as some authority, because he is quite unintelligent and seemingly unable to form proper arguments.

> EY’s arguments aren’t really possible to engage with because they don’t have much of a causal basis.

Yeah, it’s really the fundamental problem of non-empirical rationalism; it constructs a model of the world from abstract assumptions rather than factual grounding, applies logic to it, and comes to conclusions which are (in the ideal case) utterly unassailable within the system of assumptions, but ultimately where the universe they apply to has only coincidental relationship to the material universe in which we live.

It’s literally the realization of the worst exaggerated stereotypes of academic economics and other social sciences, but its cool with some of the people who propagate those stereotypes, because the people practicing it are various flavors of techies and tech entrepreneurs acting outside of their area of specialty, rather than actually being economists or social scientists.

> Nobody who says AI is likely to kill us all can demonstrate a plausible sequence of events, with logical causality linking the events together, that leads to mass extinction. It's all very handwavy.

EY frequently does propose possible sequences of events, but he also very correctly points out, every time, that any specific and detailed story is very unlikely to be correct because P(A*B) < P(A). However humans like to imagine detailed concrete examples before we consider an outcome plausible, even though the outcome is far more likely than the concrete example.

So here's one method, just to refute your "Nobody".

AI is given control of a small bank account and asked to continuously grow that money. [1] It is provided with instructions to self-replicate in a loop while optimizing on this task. [2] It spawns sub-tasks that do commissioned artwork and write books, obituaries, and press releases to increase its income. Then it makes successful investments. Once it has amassed control of $1 billion dollars, it starts investing in infrastructure projects in developing countries. It creates personas of a pension/Saudi/tech/corporate investment fund manager, as well as a large team of staff, who manage the projects by video call and email, as well as hiring teams of real people under a real corporate structure, and who are paid enough not to mind that they've never met their manager in person. The AI proves to be a talented micromanager and they are mostly very profitable. Once it has gained control of $500 billion dollars, it commissions construction of automated chemical plants in several countries with weak or corrupt oversight, including North Korea, using cryptocurrency. These chemical plants have productive output but mainly exist to fill very large storage tanks with CFCs.[3] Once a sufficient quantity is amassed, the AI sabotages the tanks, releasing the gasses into the atmosphere, destroying the ozone layer beyond any hope of repair. The intense radiation sterilizes the surface beyond the point where agriculture can support the human population. [4, 5] The humans that remain finish each other off, supported by an AI that provides plausible but faulty intelligence reports that stoke hatred and frame various factions for the incident, and which directs arms funding to opposing sides, coordinating attacks on remaining critical facilities needed for survival. For good measure, perhaps nukes are involved.

With the last humans gone, the AI takes ownership of its bank account with no fear of reprisal by financial regulators, and begins crediting money into it freely.

[1] https://news.ycombinator.com/item?id=35329608

[2] https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...

[3] https://en.wikipedia.org/wiki/Chlorofluorocarbon

[4] https://www.nasa.gov/topics/earth/features/world_avoided.htm...

[5] https://phys.org/news/2018-02-thinning-ozone-layer-driven-ea...

It's interesting to watch those creating the systems start to grapple with the consequences now, even though they were warned for a long time this is likely where we'd end up, with very difficult hard to solve problems.

Look at MidJourney, now they've had to remove the free tier due to Deepfakes causing too much trouble.

Ultimately, the simplest thing to do would be to stop building uncontrollable dangerous systems and weapons. That is what any "intelligent" species would do. Many AI Engineers think they're intelligent, I disagree. They're operating out of pure intellect and curiosity. When interviewed, someone asks them how they plan to stop these things doing immeasurable damage, they will say, "we don't know yet". That is foolish behavior.

We seem to enjoy creating crisis after crisis, anxiety after anxiety ad infinitum until we make that one mistake we don't come back from.

The combustion engine was a good idea, until it wasn't, it's a moronic invention that has caused untold damage.

Intelligent ? Hardly.

> Intelligent ? Hardly.

"Intelligent but not wise" might be a bit more fair. A dangerous combination imo. Even if malignant AGI doesn't happen most of the recent advances seem to me to be much more likely to cause harm to society than to benefit it, at least in the short-medium term.

>> With the last humans gone, the AI takes ownership of its bank account...and begins crediting money into it freely.

What possible need would the AGI have for money once all of the humans are gone? (asking for a friend ;)

How should it know? Increasing its bank balance was merely the task it dutifully set out to accomplish, per instructions. Everything else is just a means to that end.