A good way to categorize risk is look at both likelihood and severity of consequences. The most visible issues today (racism, deep fakes, over reliance) are almost certain to occur, but also for the most part have relatively minor consequences (mostly making things that are already happening worse). "Advanced AI will make humans extinct" is much less likely but has catastrophic consequences. Focusing on the catastrophic risks isn't unreasonable, especially since society at large seem to already handle the more frequently occurring risks (the EU's AI Act addresses many of them).

And of course research into one of them benefits the other, so the categories aren't mutually exclusive.

I would put consolidating and increasing corporate and or government power on that list of potential visible very short term issues.

As AI becomes more incorporated in military applications, such as individual weapon systems, or large fleets of autonomous drones then the catastrophic consequence meter clicks up a notch in the sense that attack/defense paradigms change, much like they did in WWI with the machine gun and tanks, and in WWII with high speed military operations and airplanes. Our predictive ability on when/what will start a war lowers increasing uncertainty and potential proliferation. An in a world with nukes, higher uncertainty isn't a good thing.

Anyone that says AI can't/won't cause problems at this scale just ignores that individuals/corporations/governments are power seeking entities. Ones that are very greedy and unaligned with the well being of the individual can present huge risks. How we control these risks without creating other systems that are just as risky is going to be an interesting problem.

This doesn’t work either. The consequence of extinction is infinity (to humans). Likelihood * infinity = infinity. So by hand-waving at a catastrophic sci-fi scenario they can demand we heed their demands, whatever that is.

At the extremes you get into the territory of Pascal's Mugging [1]. Which is a delightfully simple example of how our simple methods of stating goals quickly goes wrong

1: https://en.wikipedia.org/wiki/Pascal%27s_mugging

Is AI Safety a Pascal's Mugging?

[Robert Miles AI Safety]

https://www.youtube.com/watch?v=JRuNA2eK7w0

This line of reasoning refutes pie-in-the-sky doomsday narratives that are extremely unlikely, but the case for AI extinction risk justifies a relatively high likelihood of extinction. Maybe a 0.0000000001% chance is worth ignoring but that's not what we're dealing with. Cutting-edge AI researchers generally put at least a 5% chance on it, with many above 50%.

Existential risk is one of those problems that nearly impossible to measure in most cases.

In some cases like asteroids, you can look at the frequency of events, and if you manage to push a big one of of your path then you can say the system worked.

But is much more difficult to measure a system that didn't rise up and murder everyone. Kind of like measuring a bio-lab with a virus that could kill everyone. You can measure every day it didn't escape and say that's a win, but tells you nothing about tomorrow and what could change with confinement.

Intelligence represents one of those problems. AI isn't going to rise up tomorrow and kill us, but every day after that the outlook gets a little fuzzier. We are going to keep expanding intelligence infrastructure. That infrastructure is going to get faster. Also our algorithms are going to get better and faster. One of the 'bad' scenarios I could envision is that over the next decade our hardware keeps getting more capable, but our software does not. Then suddenly we develop a software breakthrough that makes the AI 100-1000x more efficient. Like lighting a fire in dry grass, there is the potential risk for an intelligence explosion. When you develop the capability, you are now playing firefighter forever to ensure you control the environment.

Saying that extinction has infinity disutility seems reasonable at first, but I think its completely wrong. I also think that you bear the burden of proof if you want to argue that, because our current understanding of physics indicates that humanity will go extinct eventually, and so there will be finitely many humans, and so the utility of humanity is finite.

If you accept that fact that extinction has finite negative utility, it's completely valid to trade off existential risk reduction against other priorities using normal expected value calculations. For example, it might be a good idea to pay $1B a year to reduce existential risk by 0.1% over the next century, but might arguably be a bad idea to destroy society as we know it to prevent extinction in 1000 years.

If you want to prevent this, you simply have to show that the probability for that extinction scenario is lower than the baseline where we start to care.

Lets take "big asteroid impact" as baseline because that is a credible risk and somewhat feasible to quantify: Probability is somewhere under 1 in a million over a human lifetime, and we barely care (=> we do care enough to pay for probe missions investigating possible mitigations!).

So the following requirements:

1) Humanity creates one or more AI agents with strictly superhuman cognitive abilities within the century

2) AI acquires power/means to effect human extinction

3) AI decides against coexistence with humans

Only need 1% probability each to exceed that probability bound. And especially 1) and 3) seem significantly more likely than 1% to me, so the conclusion would be that we should worry about AI extinction risks...

This longtermist and Effective Altruism way of thinking is very dangerous. Because using this chain of argumentation, it's "trivial" to say what you're just saying: "So what if there's racism today, it doesn't matter if everybody dies tomorrow.

We can't just say that we weigh humanity's extinction with a big number, and then multiply it by all humans that might be born in the future, and use that to say today's REAL issues, affecting REAL PEOPLE WHO ARE ALIVE are not that important.

Unfortunately, this chain of argumentation is used by today's billionaires and elite to justify and strengthen their positions.

Just to be clear, I'm not saying we should not care about AI risk, I'm saying that the organization that is linked (and many similar ones) exploit AI risk to further their own agenda.

Today's real issues are not that important compared to human extinction.

I’m not sure if extinction is a problem either. No one’s left to care about the issues, then.

How long would the currently 17 people in space survive if every human on planet earth was eradicated?

Dunno. I imagine they’d follow soon, though.

It seems to me that the most viable routes to human extinction are through superscaled versions of contemporary disasters: war, disease, and famine.

Although war, disease and famine may kill many people, they are extraordinarily unlikely to cause extinction per se. The human species has so many individuals that only very special risks are capable of reducing 8 billion to zero.

For most mammalian species if they have a stable 100k individuals then they are doing pretty well. So we have 80000x more humans than what would be considered a healthy population in other species.

The risks of human extinction tend to be more long-tail events that have a planet wide effect. For example, a large impact, supervolcano eruption, or engineered bio weapon. AI certainly deserves its place on that list.

Rare likelihood * catastrophic impact ~= almost certain likelihood * minor impact. I'm as concerned with the effects of the sudden massive scaling of AI tools, as I am with the capabilities of any individual AI or individual entity controlling one.

I would argue that all of the above are serious concerns.

The Elite class is just as apt to consolidate power with AI and rule the entire world with it. If you have a super duper AI in your pocket looking at the data around you, then they a super super super duper duper duper AI looking at every bit of data from every corner of the world they can feed the thing giving themselves power and control you couldn't even begin to imagine.

Falling into conspiratorial thinking on a single dimension without even considering all the different factors that could change belies ignorance. Yes, AI is set up to upend the elites status, but is just as apt to upset your status of being able to afford food and a house and meaningful work.

> not simply enrich a criminal cabal of globalist elites that treat you like cattle?

There is a different problem here... And that is humankind has made tools capable of concentrating massive amounts of power well before we solved human greed. Any system you make that's powerful has to overcome greedy power seeking hyper-optimizers. If I could somehow hit a button and Thanos away the current elites, then another group of powerseekers would just claim that status. It is an inane human behavior.

If you were to take a look at the list of signatories on safe.ai, that's basically everyone who is everyone that works on building AI, what could Emily B Bender a professor of computer linguistics possibly add to the conversation and how would she be able to talk more about the "real AI safety" than any of those people?

Edit: Sorry if it sounds arrogant, I don't mean Emily wouldn't have anything to add, but not sure how the parent can just write off basically that whole list and claim someone who isn't a leader in the field would be the "real voice"?

Microsoft's old chatbot, Tay, would learn by talking on Twitter. She met some people with...some things to say. And learned from them. In 16 hours she had learned so well they had to shut her down.

That failure mode was obvious to every single human person in the universe not working on Tay. (And probably a few who were.)

I think we need to be realistic and accept that people are going to pick the expert that agrees with them, even if on paper they are far less qualified.

She's a professor of computer linguistics; it's literally her field that's being discussed.

The list of signatories includes people with far less relevant qualifications, and significantly greater profit motive.

She's an informed party who doesn't stand to profit; we should listen to her a lot more readily than others.

How are fame, speaking engagements, and book deals not a form of profit?

She's intelligent and worth listening to, but she has just as much personal bias and motivation as anyone else.

The (very small) amount of fame she's collected has come through her work in the field, and it's a field she's been in for a while; she's hardly chasing glory.

People don’t have to be chasing fame to be warped by it. She has cultivated a following of like-minded people who provide ever more positive feedback for her ever more ideological writing.

I mean she is literally dismissing people who disagree with her based on their skin color. Can we stop for a minute to wonder about the incentives that encourage that?

(and I generally like her writing and think she has interesting things to say… but I do see a reward cycle going on)

Her field has also taken the largest hit from the success of LLMs and her research topics and her department are probably no longer prioritized by research grants. Given how many articles she's written that have criticized LLMs it's not surprising she has incentives.

LLMs are in her field; they are one of her research topics and they're definitely getting funding.

We absolutely should not be ignoring research that doesn't support popular narratives; dismissing her work because it is critical of LLMs is not reasonable.

In her field doesn't mean that's what she researches, LLMs are loosely in her field but the methods are completely different. Computational linguistics != deep learning. Deep learning does not directly use concepts from linguistics, semantics, grammars or grammar engineering, which is what Emily was researcing for the past decades.

It's the same thing as saying a number theorist and a set theorist are in the same field cause they both work in the Math field.

They are what she researches though. She has published research on them.

LLMs don't directly use concepts from linguistics but they do produce and model language/grammar; it's entirely valid to use techniques from those fields to evaluate them, which is what she does. In the same vein, though a self-driving car doesn't work the same way as a human driver does, we can measure their performance on similar tasks.

Hmm I looked into it, and looked at papers/pdfs in google scholar's advanced search with her as an author that mentioned LLMs or GPT in the past 3 years. Every single one was a criticism about how they couldn't actually understand anything (e.g. "they're only trained on form" and "at best they can only understand things in a limited well scoped fashion") and that linguistic fundamentals for NLP was more important.

Good to know my hunch was correct

It is not that she is critical of LLMs that is the issue.

Instead, it is that she has strong ideological motivations to make certain arguments.

Those motivations being that her research is now worthless, because of LLMs.

I don't believe the alignment doomsayers either, but that is for different reasons than listening to her.

She's the first author of the stochastic parrots paper, and she's fairly representative of the group of "AI safety" researchers who view the field from a statistical perspective linked to social justice issues. That's distinct from the group of "AI safety" researchers who focus on the "might destroy humanity" perspective. There are other groups too obviously -- the field seems to cluster into ideological perspectives.

Current topic aside, I feel like that stochastic parrots paper aged really poorly in its criticisms of LLMs, and reading it felt like political propaganda with its exaggerated rhetoric and its anemic amount of scientific substance e.g.

> Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.

I'm surprised its cited so much given how many of its claims fell flat 1.5 years later

It's extremely easy to publish in NLP right now. 20-30% acceptance rates at even the top end conferences and plenty of tricks to increase your chances. Just because someone is first author on a highly cited paper doesn't imply that they're "right"

>She's the first author of the stochastic parrots paper,

That alone is enough to disqualify any of her opinions on AI.

She’s contributed to many academic papers on large language models and has a better technical understanding of how they work and their limitations than most signatories of this statement, or the previous widely hyped “AI pause” letter, which referenced one of her own papers. Read her statement about that letter (https://www.dair-institute.org/blog/letter-statement-March20...) or listen to some of the many podcasts she’s appeared on talking about this before making presumptions uneducated statements.

Thanks for the link, I read it with interest but philosophically it is a flawed argument. It's a nonsequitur or something, for the following two reasons.

First, I'm inclined to think that longtermism is an invalid and harmful ideology, but also acknowledge that AGI / existential risk is something that needs looking at seriously. The external factors, such as corporate interests and 1% wealthy interests/prejudices, are not a good reason to dismiss AGI concerns. I'd like to imagine there's a reasonable way to address short-term issues as well as long-term issues. It's not an either-or debate.

Second, even from just a reading comprehension level: one side says AGI is a problem, then the other side cannot just say, "No, AGI is a false problem and here a the real problems". The reasonable argument is to say, AGI is a false problem because <of such and such reasons>. Bender et. al are just sidestepping the moot point, and rhetorically this is not an okay move. I think honest experts could simply say, ultimately we don't really know what will happen. But that would be boring to say because it would require acknowledging multiple issues being valid.

So I think this is actually an example of different factional experts talking past each other. It's funny that famous intellectuals/experts constantly do this, let their egos get the better of themselves and having a real intellectual conversation rather than make basic debate mistakes like nonsequiturs that any college student should be able to point at.

> First, I’m inclined to think that longtermism is an invalid and harmful ideology

It is, but (aside from as a sort of sociological explanation of the belief in AGI risk) that’s mostly beside the point when discussing the problems AGI x-risk. The problem of AGI x-risk is that it is an entirely abstract concern which does not concrete flow from any basis in material reality, cannot be assessed with the tools of assessing material reality, and exists as a kind of religious doctrine surrounded by rhetorical flourishes.

> The external factors, such as corporate interests and 1% wealthy interests/prejudices, are not a good reason to dismiss AGI concerns.

They are way of understanding why people who seem (largely because they are) intelligent and competent are trying to sell such hollow arguments as AGI x-risk. They aren’t, you are correct, a logical rebuttal to AGI risk, nor are they intended as that; the only rebuttal is the complete absence of support for the proposition. They are, however, a tool that operates outside the realm of formalized debate that addresses the natural and useful cognitive bias that itself is outside of the realm of formalized debate that says “smart, competent people don’t tend to embrace hollow positions”.

> Second, even from just a reading comprehension level: one side says AGI is a problem, then the other side cannot just say, “No, AGI is a false problem and here a the real problems”.

1. If they couldn’t, it wouldn’t be a “reading comprehension issue”, and

2. They can, for the simple reason that there is no material support for the “AGI is a real problem” argument.

> Bender et. al are just sidestepping the moot point,

A point being moot in the sense that AGI x-risk is is a reason to sidestep it. (The danger of using auto-antonyms.)

> I think honest experts could simply say, ultimately we don’t really know what will happen.

To the extent that is accurate, that is exactly what the Bender/Gebru/Mitchell group does. The problem is thinking “we don’t have any information to justify any belief on that” is one sided and means that utility of AGI is somewhere between 0 and the negative infinity that the x-risk crowd calculates from (some unspecified non-zero finite probability) times (infinite cost), whereas the reality is that we have as much reason to believe that AGI is the only solution to an otherwise certain existential calamity as to suppose it will lead to one. The utility is somewhere between positive infinity and negative infinity.

A point being moot is a reason to agree to disagree, if they can't agree on the premise. But they need to say that. If I were writing a letter, I would say it because that's just being sensible.

This isn't about logical debate. This is about reasonable, non-sophistic writing at the college level or higher. And there are basic standards, like if they don't know the future then they must explicitly acknowledge that. Not rhetorically "do that" in the essay. Literally write it out in sentences. They didn't.

I can think of 3 examples where such explicitness was done. Chomsky's letter gave explicit reasons why AGI is a false issue (and he was pilloried for it). My computer science professors literally, in their deep learning class and in their theoretical machine learning research seminars, have literally acknowledge that we don't know almost anything about the fundamentals nor the future. That scientific humility and level of intellectual conscientiousness is needed. That is absent in this discourse between the experts. And note, by that, I also include the 22-word "letter" which doesn't actually explain why Hinton and the rest of the signatories think AGI is an existential risk, what their specific reasons (your "material evidence") for that are.

You have it totally backwards. It's a much bigger catastrophe if we over-focus on "safety" as avoiding sexism and so on, and then everyone dies.

Exactly. Biased LLMs are incredibly unimportant compared to the quite possible extinction of humanity.

> A good person to follow on real AI safety is Emily M. Bender (professor of computer linguistics at University of Washington): https://mstdn.social/@[email protected]

- Pronouns

- "AI bros"

- "mansplaining"

- "extinction from capitalism"

- "white supremacy"

- "one old white guy" (referring to Geoffrey Hinton)

Yeah... I think I will pass.

I think "pronouns" is ok but, yeah, "AI bros" is enough to get a pass from me to. Holier-than-thou name calling is still name calling.

There are a lot of AI-bros on twitter.

Do they self-identify as AI bros? Or are you just choosing to call them that?

Odd that you lump a person's choice of their own pronouns into a legitimate complaint that all she seems to have is prejudice and ad hominems.

I have no issue with her choice of pronouns. I just find it odd that she states them when ~100% of the population would infer them correctly from her name Emily (and picture). My guess is she put them there for "ideology signaling".

This is unnecessarily cynical. Why should people who are androgynous or trans be the only ones who state pronouns? By creating a norm around it we can improve their comfort at extremely minimal cost to ourselves.

I disagree, but HN is probably not the right place for this kind of debate. Also, it seems that you don't follow your own recommendation (on HN at least).

They brought it upon themselves by indulging in sex mimicry. It's not the responsibility of anyone else to mitigate this.

And it's not necessarily a good thing to normalise wrong-sex pronouns, see e.g. https://fairplayforwomen.com/pronouns/

> My guess is she put them there for ideological signaling.

Yes, and you attacking her for that choice is also ideological signaling. This isn't a case where one choice is totally devoid of signal and the other is pejorative-of-the-day.

Many people believe that using normative pronouns makes those who use different pronouns more comfortable. They may be right, wrong, misguided, whatever. But it's a well-meaning and harmless act, and extremely strange to mention in the same breath as overt racism.

Reads like a caricature of the people leading these causes on AI safety. Folks that are obsessed with the current moral panic to the extent that they will never let a moment go by without injecting their ideology. These people should not be around anything resembling AI safety or "ethics".

It sounds like you're making an ad hominem about her ad hominems.

I might look past the fact that she was sexist/racist if the topic of interest wasn't ethics. But since it is, I'd say those quoted tweets are pretty relevant.

> organizations is that they focus on the wrong aspects of AI safety. Specifically, they run this narrative that AI will make us humans go extinct.

their goals are to get funding, so FUD is very good focus for it..

the issue with hacker news comments these days is people don't actually do any due diligence before posting. center for ai safety is 90% about present AI risks and this ai statement is just a one off thing.

Particularly ironic given this isn't actually what the focus on...

https://www.safe.ai/ai-risk

This submission is exactly about an exinction level event.

Here is the full text of the statement: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

By extinction, they mean extinction of the human species.

I'm referring to the public. Not the submission.

> This is not a real risk today.

Yes, clearly. But it is a risk for tomorrow. We do still care about the future, right?

I’m sure we can start talking about AI regulation once the existential risks are already happening.

I, for one, will be saying “told you so”. That’s talking, right?

No man, the future of trillions of humans is obviously much less important than 1 person getting insulted on the internet.

Especially ironic considering the "longtermism" mental illness that's in vogue with SV types.

the real immediate risk isn't either of these imo. it's agentic AI leveraging some of that to act on the wishes of bad actors.

You hear similar arguments from those who believe climate change is happening but disagree with current efforts to counter-act it. The logic being that right now climate change is not causing any major harm and that we can't really predict the future so there's no point in worrying about what might happen in a decade or two.

I don't think anyone is arguing that right now climate change or AI is threat to human civilisation. The point is that there are clear trends in place and that those trends are concerning.

On AI specifically, it's fairly easy to see how a slightly more advanced LLM could be a destructive force if it was given an unaligned goal by a malicious actor. For example, a slightly more advanced LLM could hack into critical infrastructure killing or injuring many thousands of people.

In the near-future AI may help us advance biotech research and it could aid in the creation of bioweapons and other destructive capabilities.

Longer-term risks (those maybe a couple of decades out) become much greater and also much harder to predict, but they're worth thinking about and planning for today. For example, what happens when humanity becomes dependant on AI for its labour, or when AI is controlling the majority of our infrastructure?

I disagree but can understand the position that AI safety isn't humanities number one risk or priority right now, however I don't understand the dismissive attitude towards what seems like a clear existential risk when you project a decade or two out.

>that those trends are concerning.

Which trends would you be referring to?

>it's fairly easy to see how a slightly more advanced LLM could be a destructive force if it was given an unaligned goal by a malicious actor. For example, a slightly more advanced LLM could hack into critical infrastructure killing or injuring many thousands of people.

How are you building this progression? Is there any evidence to back up this claim?

I am having a hard time discerning this from fear-mongering.

If AI improves by 0.0001% per year on your favorite intelligence metric there will eventually be a point where it surpasses human performance and another point where it surpasses all humans combined on that metric. There is danger in that scenario.

Assuming that timeline, can we agree that we have some time (years?) to hash this out further before succumbing to the ideals of a select few?

The problem is that even with N years until we reach that point it seems likely that it would take 2*N years to build the proper safety mechanisms because at least currently capabilities research is racing far ahead of safety research. Of course we have no way to know how big N really is and recent results like GPT-4, Llama, Gato, etc. have shifted peoples timelines significantly. So even if 5 years ago people like Geoff Hinton though this might be 30-50 years away there are now believable arguments to make that it might be more like 3-10 years.

> slightly more advanced LLM

I don't think there is a path, that we know if, from GPT4 to a LLM that could take it upon itself to execute complex plans, etc. Current LLM tech 'fizzles out' exponentially in the size of the prompt, and I don't think we have a way out of that. We could speculate though...

Basically AI risk proponents make a bunch of assumptions about how powerful next-level AI could be, but in reality we have no clue what this next-level AI is.

When you have no clue, it makes sense to expand your confidence interval in both directions. So it could be a lot longer than we expect, but it could be shorter, too. You shouldn't just say "we have no clue, so it's probably further off and not worth worrying about". Especially since a lot of people made predictions about LLM capabilities and were surprised by how much better they worked than expected.

I agree in principle, it's just that the level of expressed worry doesn't seem to match reality. Currently we have no reasonable path to 'scary AGI'. It's some yet newfangled tech we haven't discovered.

As an example, consider the invention of motion picture. People were totally bewildered that you can have moving things and people inside a picture. Scaremongers could start claiming "Pretty soon the moving things may come to life and take over the world! Before you know it, they'll run our factories from inside the movies and we'll be their slaves!" That's more or less what this 'scary AGI' hype sounds like to me right now.

Btw "That Mitchell and Webb Look" is a great show ;-)

What is "systemic racism"? How is it a risk?

Don't bother explaining, we already know it's unfalfisiable.

>Real risks are more in the category of systemic racism and sexism, deep fakes, over reliance on AI etc.

This is a really bad take and risks missing the forest for the trees in a major way. The risks of today pale in comparison to the risks of tomorrow in this case. It's like being worried about birds dying in wind turbines while the world ecosystem collapses due to climate change. The larger risk is further away in time but far more important.

Theres a real risk that people get fooled by this idea that LLMs saying bad words is more important than human extinction. Though it seems like the public is already moving on and correctly focusing on the real issues.

>This is not a real risk today.

Many experts believe it is a real risk within the next decade (a “hard takeoff” scenario) That is a short enough timeframe that it’s worth caring about.

Yep agree. They talk a big game about how their product is so powerful it will take over the world if we're not careful, but they don't talk about how they are complicit in relatively more mundane harms (compared to AI taking over the world) that are real and happening today thanks to their system design.

They want to promote the idea that their product is all-powerful, but they don't want to take responsibility for dealing with bad assumptions built in to their design.