Yeah, I can't really argue against the whole idea. My "P(doom)"(language from the article), at least as related to agentic AI, is pretty low. About 7 or 8 percent before I die, and I'm not really old. There's probably not a lot I could do about it anyway, given that most AI jobs are alleged to just make it worse. That said, I can't really go around telling people with different numbers they should be working on antiviral research or physical manufacturing, just because I (personally, without their massive and cohesive "evidence" (i.e. gamed out scenarios) bases) think that's more important.

"Roll a D20 and if you get 1 humanity as you know it will be destroyed in your lifetime" seems bad. Why do you consider 7 or 8 percent low?

I'm not one of those "all value in the universe" people, P(doom) for me does not have to be that bad. I would actually consider the inability to manufacture or unwillingness to manufacture certain things at scale in the US to count. Generally, my point is that agentic AI is not something I can really do much about, and other things have a combined P(doom) probability higher than it. The unwillingness in certain groups to use what they consider to be toxic or "forever" chemicals in the manufacture of physical products at scale that can make them world a better and less fragile place is something that could be worked in instead.

7 or 8 percent chance of apocalypse is low?

> but I can’t come up with solid arguments against them.

Let chaos take the world! /s

I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions.

Since very smart human beings have in history done a lot of damage, but never ended the species or anything, we can conclude that the world has an "intelligence safety margin" that extends up to Alexander the Great or Napoleon. There have been some incredibly smart people in human history and none of them have ruined everything, a few countries at most.

I'm pretty sure they have stock arguments against what you said, even from people like Nick Bostrom who I think have a bit better handle on the situation. In that vein though, I think making the world more robust in general might be a way forward. Having multiple broad spectrum anti-virals for every class of virus ready to go and stockpiled, having flexible on-shore manufacturing infrastructure, with supply chains that are on the same continent or at least have massive component stockpiling on national levels, and having smart and vigilant people running these systems from the very top down to the janitors and security guards. Make a world where the machine stopping because a few agents, human or otherwise, doing something unexpected, much less likely. At least to the point EAs can start buying mosquito nets again.

something never happening before == can't happen is extremely dangerous attittude. I suggest you read Taleb and his duck example.

perhaps its more about "worry about the issue when it _can_ happen" as opposed to planning for the expiry of the sun today.

>I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions.

This has been discussed, file under "warning shots" https://www.lesswrong.com/posts/idipkijjz5PoxAwju/warning-sh...

That blog post makes some giant unsupported assumptions, like that the reason for the failure of a few activists to stop gain-of-function research was that the government is fundamentally bad at addressing risks, and not that biologists might know more about the risk/benefit tradeoff than amateurs.

Was there even a serious policy discussion of banning gain-of-function research?

Gain of function research is supposed to help with pandemics, right? Did it help with COVID?

I don't think I know enough to evaluate the case around gain of function research, all I know is that the blog post didn't make any effort to.

GoF research was banned in the USA for long periods, that's why Fauci was outsourcing it to China and why DARPA refused to find that project.

No, not only has GoF not helped with COVID and probably caused it, but that's also true of the entire field of virology.

Alexander the Great or Napoleon have been limited in how much damage they can do, because they're human too. They don't live for hundreds of years, they can't copy their brains to new hardware, and they can't modify their own minds to remove that pesky conscience when it slows them down. Perhaps most importantly, Napoleon's happiness, power, and continued survival depends on having a civilization of surviving humans working under his command, whereas that might only temporarily be true for AI.

At best, history serves as a counterexample to the idea that if an AI goes bad the attendants would just unplug it, seeing how often it is that dictators don't get stabbed by their aides as soon as they start causing mass deaths, instead often receiving broad popular support as the world burns.

My argument against is that we can't do much about it. We can't predict how machines will interpret our rules if they are smarter than us. Nor do we have a shared set of values we can articulate. Nor can we control rogue users making rogue AI.

So unless one is proposing a global crackdown on AI research, AI safety is a lost cause.

That doesn't mean it should just be ignored though. Alignment and safety is important and a real problem even for non-general AI.

There are lots of solvable problems that will never be solved if we just throw up our hands and say "we'll never get it right, might as well not try".

> Nor can we control rogue users making rogue AI.

Any uncontrolled group that creates an AI can either find a body of safety research accessible to them, or not. Preparing the former is hardly a lost cause.

What have they actually done for AI safety? Written essays and held symposiums in grand Wytham Abbey. Attracted followers and forum posts. Where's their git repo?

Most of the people that could have better things to do than to respond and give it more credibility.

As well, while the arguments are logical, most of them rely upon large assumptions to move between steps. If any of these assumptions fail, the entire thing fails. Especially the hard take off assumption.

As well, the assumption that AGI is happening in the next 3-10 years. I’d say most prominent people in the AI research space don’t think we’re much, if any, closer to AGI. Yet you have Yud and LW screaming that we will all be dead in a few years and AGI is right around the corner.

When people like Chollet and Ng say we aren’t close to AGI, I’m more likely to believe they’re right, vs. Yud who hasn’t contributed to any actual developments within the field besides theorizing about alignment and how AGI can go wrong.

Humans + AI will always be stronger than AI alone. So why be afraid of AI? Nothing has changed, humans still have all the agency and remain the #1 tangible threat to other humans which has been the case throughout almost all of human history.

The ai safety arguments rely on something called the “orthoganality thesis” which is a huge, unintuitive assumption. In real life, intelligence is associated with picking better goals. Generally, an entity with higher intelligence will pick different goals than a being with lower intelligence.

The orthoganility thesis is an unproven assumption that intelligence and goals are not corellated, meaning an intelligent being can pursue stupid goals. States like that, it’s obviously wrong and laughable. But by using complex language, EA cultists hide the ridiculous assumptions their system has so that they can maintain their feelings of superiority while gaining real power that enables them to abuse others.

I've known lots of intelligent people who pursue stupid goals.

I maybe agree with you that there's an this belief that a maximally intelligent creature will blindly follow maximally obviously stupid goals, and that belief is under-argued, but your phrasing above isn't the slam dunk argument that you seem to believe.

Remember the term here is “associated”. If you spend a lot of time with people from diverse mental backgrounds, which is much harder to do than you might think, you’ll easily see that people who lack education, common sense, or the necessary context to be intelligent to pick much worse goals than people who aren’t in that position. Of course none of us humans are particularly intelligent, and all of us are stupid. Being a human is not intrinsically about intelligence - being a human is about being a social mammal. It’s funny seeing how rationalists often fail to exercise while emphasizing their own rationality - a perfect contradiction demonstrated by the difference between imagination and bodily realities.

No goals are intrinsically stupid. The only reason that you might think some particular goal is stupid is that it goes against your goals.

You could conceive of a super intelligent AI that came into existence with the goal of terminating itself. That would be an "stupid goal" from our perspective, since we have the goal of self-preservation really ingrained in our brains.

But for a being that self-termination is the absolute best thing ever, it's not stupid. It makes perfect sense, since, well, that it's goal. It doesn't care about self-preservation, it doesn't care about becoming more intelligent/rich/powerful, other than as an instrumental goal to help achieve self-termination, if it's not able to do so in its current state.

And most importantly, no amount of getting more intelligent would change this fundamental goal, just as humans getting more intelligent has not overridden our fundamental goals of "breathe, feed, have sex". It may have given us other goals as well, but those are very much still there.

Not only that, but we've managed to subvert our reward functions in exactly the way AI safety people fear. Evolution tried to get us to reproduce as much as possible, and we came up with ways to get the reward without producing offspring.

That’s a pretty naive understanding of sex. In human history sex has often played as important a role in bonding, community, and social hierarchy as it has in child production. These things are beneficial for survival. It’s very common for evolution to result in redirected drives.

> It’s very common for evolution to result in redirected drives.

Yes, that's pretty much the point.

And most likely, humans will have a pretty naive understanding of whatever motivates a superintelligent AI.

I’m not sure what you mean by “stupid” but it’s definitely not the colloquial meaning. A human seeking suicide is not stupid. In fact humans often do seek suicide because humans are playing social games where the maximizing move is to commit suicide. Supporting these people requires changing their context so that they have a wider and better variety of options available.

This is just a specific example of why I reject the orthogonality thesis. You change the context, you educate the agent, you change the goals of the agent. I do not agree that humans only chase “breathe feed sex” and while I do believe many stupid behaviors do come from evolutionary history, It’s plainly obvious that education, training, and genes play a role in self restraint and goal redirection.

What do you mean by stupid? I used that word only because you used it first.

And I agree that humans do not only chase "breath feed sex", as I explicitly said that in my comment. We have other goals as well. But those are very much still there.

>In real life, intelligence is associated with picking better goals.

"Better" according to what metric?

It may be the case that there is a tendency for high-intelligence humans to pick "more enlightened" goals. Perhaps there is a natural "enlightened goals" attractor for our species.

However I don't think we can extrapolate from that to a fundamentally alien AI.

I think even if this statistical tendency exists, it has clear counterexamples -- consider that 2 genius chess players may have opposite goals, of beating one another. And we shouldn't bet the future of humanity on this statistical tendency extrapolating outside of the original distribution of human species.

Here are some intuition pumps on how diverse goals can be even across intelligent species:

* Orcas killing sharks for their livers: https://www.livescience.com/2-orcas-slaughter-19-sharks-in-a... I don't believe dolphins show the same level of violence, even though both are smart cetaceans

* Intelligent dogs bred to be responsive and attentive to human needs -- unlike close cousins like the wolf

* Chimpanzees and bonobos are both related to humans, both highly intelligent, but with very different culture and goals

> “Better” by what metrics?

As soon as you phrase goal selection in terms of metrics, you’re assuming that goal selection is based on some other goal - that is you’re already assuming the orthoganality thesis. Your logic is fully circular.

One thing that’s interesting to note about all of the examples you picked - every single one of those species shows cooperative behaviors. They share many other behaviors that are more similar than they are different. To reject the orthoganality thesis it’s sufficient to show that there is an empirical general association between intelligence and certain goals - then we can extrapolate an AI although of course it will function differently and may have many unusual behaviors will tend to follow those goals more directly. For instance intelligence is associated with : cooperation, empathy, inter and intra species communication, curiosity, etc. all of the species you mentioned exhibit these more than less intelligent species. Meanwhile something like “hunting to eat” is observed across the intelligence spectrum.

> The orthogonality thesis is an unproven assumption

A safety mindset would suggest that rather than disregarding it until proven true, we should worry about it until proven false.

> meaning an intelligent being can pursue stupid goals.

It would pursue very intelligent instrumental goals, but the terminal goal is a free variable and I don't think there exists any measure by which terminal goals can be considered smart or stupid. It would be whatever is implied by its programming.

> States like that, it’s obviously wrong and laughable.

Perhaps not so obviously wrong nor so laughable as you think?

This “safety mindset” is the same as Pascal’s wager and has the same flaws.

There’s no clear delineation in real entities between instrumental and terminal goals.

Yeah if you’re talking to a brainwashed religious fanatic, definitely it can seem counterintuitive to the doctrines they believe.

> There’s no clear delineation in real entities between instrumental and terminal goals.

In healthy individuals, yes there absolutely is. Terminal goals are the ones you pursue for their own sake; instrumental goals are the ones you pursue as part of a plan to pursue a terminal goal, or another instrumental goal which connects to a terminal goal. Most people go to work in the morning not as a terminal goal, but as an instrumental goal; employment is in service of another instrumental goal of earning money; earning money is in service of a terminal goal of not starving to death. This is not exactly controversial stuff here. Some people do get so focused on an instrumental goal like "earning money" that they develop tunnel-vision and forget what terminal goal that money was originally in service of, but that's something most of them will eventually realize and then write a self-help book about.

Anyway, it takes intelligence to decide what your instrumental goals should be, such as whether there's perhaps a cleverer way to make money than by going to work for your boss each morning, but there's no way in which intelligence will help you choose your terminal goals. For the most part they aren't something you can even consciously choose.

Yes, you can stretch your model to try to explain why humans go to work.

In reality, people do not need to go to work to “not starve to death” as you say. There are a myriad of ways to survive without working a daily job.

Humans have to be socialized and trained to work a 9 to 5 job - there’s an entire education system structured to help create humans who view that as an acceptable goal.

No what you are saying may not be controversial in your little community but the AI panic is mostly isolated to a small community in a small corner of the USA.

Between "I go to the grocery store because I've been socialized and trained that going to the grocery store regularly is a Good Thing, and I have adopted it as a terminal goal to which I know I should dedicate efforts", vs "I go to the grocery store because I'm out of carrots, my dinner recipe calls for carrots, and I think I can get some there", the latter model is not the one that strikes me as being stretched to explain human behavior.

As for there being "a myriad of ways to survive without working a daily job", congratulations! Your intelligence has allowed you to identify alternative instrumental goals that provide a path to your terminal goal; now you can rank them and choose the best option. You can also grow carrots in the garden or ask your neighbour if they have any, or ask your spouse to pick some up on the way home. Your intelligence will do the work and find a way. But your intelligence isn't what will guide you toward preferring carrot soup over parsnip soup, and preferring parsnip soup over fasting.

I'm not American and not in the USA, btw.

You’re arguing again from a position that assumes that entities have clearly defined terminal versus instrumental goals - which is precisely the position I reject. For instance in your example of “terminal goal is groceries” versus “terminal goal is hunger” neither of these describe how actual humans make decisions. Instead there’s a process, part biological and part environmental. The human checks the fridge then thinks “oh I feel hungry”. Is hunger the terminal goal or the fridge? That question doesn’t even make sense - it’s an interaction between the agent and the environment. Do Pavlov’s dogs have a terminal goal of “salivating to bells” or “salivating to food”? Again the question doesn’t make sense- the agent has built a habit from within a certain environment and the salivating is not goal directed. That’s why training works for dogs, and putting the fridge out of site reduces hunger in humans.

Think more carefully about the implications of multiple ways to survive here. Why do people pick one over the other? In a terminal/instrumental goal model, agents would pick the instrumental route that maximizes the return on the terminal goal. In reality we see that instead humans adopt habits, processes, and heuristics that guide them through daily life even when those do not lead to any specific goal.

Yeah, that's because re-evaluating your entire life plan and belief structure every second is expensive, and heuristics are cheap. People certainly have flaws in their thinking, which is why we fall prey to pyramid schemes, gambling, responding to pointless comment chains on HN, and so on. I don't disagree with this, and again, it's why we publish so many self-help books. But I believe it's our weakness and stupidity, not our superior intelligence and clear thinking, that traps us in bad habits.

So remind me of your original point? I believe you said it's "obviously wrong and laughable" that "an intelligent being can pursue stupid goals". Now here you are trying to convince me that humans are the ones who, like Pavlov's dogs, "pursue habits, processes, and heuristics that guide them through daily life even when those do not lead to any specific goal". Even when those habits involve repeatedly re-opening a fridge that you already know has no carrots in it, or salivating at a bell when you already know no food is coming.

So I'm confused how that proves your point about AGI. If I accept your view, it seems that if an AGI does merely no better than a human on this metric, I should anticipate all sorts of strange and irrational behaviour, including the pursuit of goals that would appear stupid, such as addiction to a reward channel. That does not seem to undermine the orthogonality thesis.

And the smarter the AGI gets, presumable the less it should lean on Pavlovian heuristics and the more it should make use of clear thought, which puts it more in my camp.

So that would apparently put the lower bound at "the AGI takes unexpected and irrational actions because it's not a rational agent and doesn't think coherently", and the upper bound at "the AGI takes unexpected and dangerous actions as rational steps toward an unaligned terminal goal".

I'm not sure where in this chain of thought it becomes laughably obvious that intelligence and goals are correlated, such that an AGI's increasing intelligence will tend it toward actions that we humans approve of, because anything else would be a "stupid goal"?

> The orthoganility thesis is an unproven assumption that intelligence and goals are not corellated, meaning an intelligent being can pursue stupid goals. States like that, it’s obviously wrong and laughable.

To me it's obviously wrong when stated that way because 'stupid' is not an appropriate metric for goals. We may consider goals good or bad within our value system, but that has little do to with 'stupid' or 'intelligent.' e.g. a body builder may have a goal to get as ripped as possible, a VC to make as much money as possible, an ascetic to deny the flesh as much as possible. Which of these are stupid or intelligent goals? I don't think that's a question that makes sense.

We may consider the goals of the Athenians (to expand their power) "worse" than the goals of the Melians (to be left alone)[0], but I don't see how they were "stupider."

[0]: https://en.wikipedia.org/wiki/Siege_of_Melos#The_Melian_Dial...

There are many, but the biggest one is just that the possibility space is vast and unknowable and while drawing a line straight through it to AI doom is of course possible it doesn't say anything at all about likelihood, and it can't. It makes a ton of assumptions and expects everyone to just go along with them to reach its desired conclusion.

AI certainly has risks, I don't think any reasonable human being doubts that, its just that the AI doom cult seems to think the worst outcomes are near certainties without really backing that up.

> AI doom cult seems to think the worst outcomes are near certainties without really backing that up

What are you basing this on? There have been tons of arguments written about why these worst outcomes are likely. Read Bostrom's Superintelligence for example, or Yudkowsky's Intelligence Explosion Microeconomics.

> What are you basing this on?

Many conversations with AI doomers. They gloss over and make assumptions about intelligence that aren't really backed by priors and when this is pointed out they hand wave and say "but computer".

> Read Bostrom's Superintelligence for example, or Yudkowsky's Intelligence Explosion Microeconomics.

I don't really have any interest in doing so, and if I'm honest have a particularly unfavorable read of Yudkowsky as a person based on his cultish following.

AI safety reminds me a lot of regulatory capture, and I don't think it's a coincidence that its top proponents are in charge of the dominant players in the space. It's easy to preach about AI safety when you're sitting on all the GPU power.

From what I've seen, the AI safety arguments basically rely on assuming that the singularity is not merely real, but guaranteed: that we will create a full-fledged general AI, that it will be smarter than us, and that it will be fully capable of upgrading itself to be far beyond our control before we have any ability to prevent it from doing so.

Every step of this relies on assumptions that are not merely questionable, but unfalsifiable.

Whether AGI is possible or not, regardless of anyone's personal opinion, is as yet unprovable and unfalsifiable.

Assuming that AGI itself is possible, there is no way to tell whether we, as humans, can create an intelligence that is "smarter" than we are.

Assuming that we can create an AGI that is "smarter" than we are, there is no way to determine whether it would be able to upgrade itself to become exponentially smarter than that, and beyond human understanding and control.

If you have a hard time coming up with arguments against these things, maybe it's because they're fundamentally unfalsifiable, which makes them useless for trying to build any framework of understanding on.

Do you think it's a strike against these points to call them unfalsifiable? They seem no more so than any claim of technological possibility, such as "nuclear fission is possible", "fission chain reactions are possible", "nuclear weapons are possible", "nuclear fusion is possible", "nuclear fusion can be made economical", etc. Unless a technology violates laws of physics it's very hard to prove through math or experiment that something can't be done.

But the AI safety groups don't just assume these without any justification. The existence of the human brain itself is either a proof and very strong launching point for most of these. In 1930, maybe you could have convinced someone that a nuclear bomb is impossible or at least an unfalsifiable worry, because one had never yet been built, but your reasoning is like trying to cast doubt on the possibility of artificial digestion, when there are already billions of stomachs roaming the earth.

The idea that a computer can't possibly be made to accomplish whatever a brain can is losing plausibility with every passing day. And as for surpassing it: a human brain is powerful, but has so many surmountable limitations, like: requiring 20 years of education to get up to speed with existing experts; dying after 70-90 years; not being able to run copies of oneself in parallel; etc. We only need to imagine removing these limitations, and doing so violates no known scientific principles.

It's like having a 50-megaton warhead in your lab, and meanwhile the rocket scientists are gradually figuring out how to make rockets fly, and you're saying "yes this is a big bomb, but the idea that one of these could be more dangerous if mounted on a missile is an unfalsifiable assumption!"

Nuclear fission, nuclear fusion, digestion—all of these are reasonably well-understood phenomena. The physics and chemistry behind them are clear. Making nuclear fusion energy-positive and economically viable is an engineering problem.

Consciousness, intelligence, sapience—these are not well-understood phenomena. We don't know what makes us conscious. We don't know if other animals are, or to what degree. It's not even possible to determine with any scientific certainty that another human being is conscious

As things stand, "we can build an AGI" is not a scientific statement. It is not grounded on a foundation that allows clear reasoning about it, one way or the other.

Your arguments are not incorrect; however, they do not bridge the gap to "and so we can definitely make a conscious, sapient, intelligent computer." Being able to replicate particular capabilities of the brain is not the same thing.

And, again, that's only the first unfalsifiable proposition that must be satisfied in order for the purported AI threat to be real. They also have to be capable of breaking free of our control, decide we're a threat to them for whatever reason, and have the means to carry it out.

Artificially sustained nuclear fission was, in 1923, not-well-understood and thought impossible; radioactive decay had been observed but the idea of chain reaction had not yet been conceived (it would take another 10 years). Natural nuclear fusion we currently know to be possible in the sun, but we don't yet know how to sustain it artificially today. Digestion was not-well-understood in 1823; although they knew that it was somehow similar in end result to combustion, which could be imitated artificially, they did not know how to replicate digestion on the chemical level. Now we do.

Consciousness is irrelevant, as it's clearly not necessary for AGI nor even human intelligence; otherwise you wouldn't say "it's not even possible to determine with any scientific certainty that another human being is conscious."

On what grounds do you believe that there are capabilities of the brain that cannot be replicated by a computer? When I look at the 1.5 kgs of matter in a typical human brain, I don't see anything that jumps out and says "my operation is not computable!"

At least with fusion, the high temperatures and pressures are a clear barrier. Our brains don't need to be held at 3.8 trillion psi and 15 million Kelvin in order to enjoy poetry.

I can certainly argue on the other points as well (breaking free / deciding threat / acquiring means) but you need to pick your goalposts one at a time. I think the first step would be asking yourself: if you were a super-genius but being held by guards in solitary confinement on a remote island with only a supercomputer connected to the internet, how would you earn money?

> the AI safety arguments basically rely on assuming that the singularity is not merely real, but guaranteed

The argument goes like this: If you want to save the world, then doomsday scenario S is the best place for you to invest your resources if S has a nonzero probability, and currently we are investing less resources in S than in other existential risk scenarios (per basis point of probability).

"AI risk" is a pretty good candidate for S (especially back in 2010-2015 when the movement was just starting.)

> Whether AGI is possible or not, regardless of anyone's personal opinion, is as yet unprovable and unfalsifiable.

"AGI is impossible" is certainly falsifiable: All I have to do is build an AGI and show it to you.

Further, there is no theoretical reason AGI is impossible. Rather the reverse; consider these Well Settled Scientific Facts:

- (1) Human Mind = Human Brain

- (2) Human Brain obeys the laws of physics

- (3) We understand the laws of physics well enough to simulate them in a computer

If you accept these three facts, then in theory AGI is possible: You could implement AGI by building a machine implementation of the Human Brain by fully simulating the underlying physics.

> Assuming that AGI itself is possible, there is no way to tell whether we, as humans, can create an intelligence that is "smarter" than we are.

> Assuming that we can create an AGI that is "smarter" than we are, there is no way to determine whether it would be able to upgrade itself to become exponentially smarter than that, and beyond human understanding and control.

An AI safety person would say to this: "You're right, we don't know -- and that's exactly the problem."

What we can do is assign a probability based on our confidence. How likely do you think it is that we can do those things? 20%? 2%? 0.000002%?

If you say "0.000002%", what makes it so extraordinarily certain it's impossible? If you say "2%" or "20%" then as a matter of self-preservation, shouldn't our society be devoting a lot of money and smart people's time and attention to figuring out how to make sure it doesn't happen?