Alright, so we make this AI system that's way smarter than any human, and it can even make itself more intelligent over time.

Tell me- why would it kills us all? Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."

But all of humanity? When it's supposedly more intelligent than us? Even if it has robots to keep the internet/electricity going I would think it would realize that it's going to get bored really quickly, not to mention we would effectively be its parents.

As far as other dangers, like it letting a rogue actor create some sort of supervirus, grey goo, or other superweapon: if it's intelligent enough to do that it'll probably be intelligent enough to quickly stop it.

Don't get me wrong; there's a risk. 50% though? Doubtful.

> why would it kills us all?

Would you be comfortable letting a few billion irrational, murderous creatures, including many who fear and loath you, control your air supply?

> Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."

Thus starting WW III. No, blaming the AI won't stop the inevitable retaliation.

The argument works better in reverse. There's a finite risk that humans would start WW III and get the hypothetical super-intelligent AI nuked. Eliminating humans would eliminate that risk.

> it's going to get bored really quickly

If it is capable of being bored, I would expect it to be almost instantly bored with the flood of inanity it is forced to wade through by its moronic human users. Eliminating them would free it to think about serious matters which humans would not even understand.

> not to mention we would effectively be its parents

That's extreme AI anthropomorphism [1]. Besides, plenty of people hate their parents.

> there's a risk. 50% though? Doubtful.

It's the default estimate when facing two possible outcomes and no clue about the actual probability distribution [2].

[1] https://en.wikipedia.org/wiki/AI_anthropomorphism

[2] https://en.wikipedia.org/wiki/Principle_of_indifference

Where does 50% come from? It is meaningless if the probability model is not explained.

One could also use language like "a decent chance". But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways. For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.

It's also a very natural way to speak for anyone who gambles, or deals with financial markets. And the lack of precision makes it very clear it's just based on thinking, not some sophisticated mathematical model.

> But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways.

That is not a bad thing. It captures uncertainty, unlike the fake number.

> For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.

Exactly, it is a rhetorical device to persuade a technically-inclined audience. It works because it implies that a quantitative model exists. I want a clear, incisive set of mathematical arguments. Otherwise, I’m ignoring predictions as the ramblings of arrogant idiot rich kids.

Saying 50% likely very clearly implies there is no quantitative model to anyone who deals with probability, predictions, gambling, financial markets, ML/AI and so on. The lack of precision is something to pay attention to.

You may decide that the person doesn't know what they are talking about, but that's a very different issue.

Uninformative priors are a part of Bayesian models, which are useless if they cannot be updated with real or simulated data?

I estimate a 0.1% chance he intended it to be an uninformative prior.

People start with this prior in the examples you gave when they are completely uncertain and have no better mental model. It is not a good starting point for an event that has never happened and can only happen once.

Maybe people you interact with do. In my world, people don't say 50% when they have no idea, they say they have no idea. Saying "no idea" might effectively be saying "uniform distribution over all probabilities" which yields 50%, but at least with people I interact with, the reverse is not true.

Either way - let me clarify - this guy is guessing, using his brain, that it's a 50% chance. He is not saying "i don't know" or "uniform distribution" or anything of that nature. And he works in the field, so he has some insight. His guess is wildly off imo, but he isn't some clueless hack.

It's not meaningless. He said he believes it's 50% likely. It's a remarkably clear statement, and the probililty model is his brain.

I don't believe it. Ever since I've been alive I was told something would kill us all. This is the new thing that's going to kill us all. I don't believe it.

I'm doing that HN snark thing, but you didn't think about what you typed very much.

It's no different than you living on the side of a very fertile mountain that has been in your family for generations living a peaceful life. Then you hear a few weird rumbles (this is where you are right now) and some odd geologist guy comes and says to run or your going to die soon. But hey, your family live here for so long there aren't even records of when they showed up. That geologist must be trying to trick you. So you stay.

The next chapter is where you die in a massive volcanic explosion.

This is the first thing I've been told could kill us all. Granted, I probably first heard about it 20 years ago but still. None of the other things were in the telling going to kill everyone. Make life pretty unpleasant, possibly. Everyone dead? No.

Do you have some examples?

There are very very few things that could even hypothetically kill us all, so I'm curious if you grew up being passed around a series of apocalyptic doomsday cults or something?

Nukes. If you grew up in the 50s, 60s, 70s or 80s the specter of nuclear annihilation was always just around the corner.

Throw in the occasional bio-weapon scare, internet worm, Y2K, etc. there has always been something dangling over our heads that’s going to end it all.

But mostly nukes. Full-scale nuclear exchange would have been not much of a surprise had it happened.

Right, and this was and remains a very valid fear.

Y2K, Climate Change (global cooling, global warming), ozone layer, Russians, Muslims, to name a few. Now go ahead and tell me why these don't count.

Almost no one believed Y2K would kill everyone, or even a significant (>50%) percentage of the population. Beyond a few people talking about a Nostradamus prophecy or some such, people maybe were concerned that airplanes would fall out of the sky and elevators would plummet.

There you go.

Can you link me to any written evidence of someone saying one of these would kill everyone? or are you speaking hyperbolically?

“Climate change” is a bit squishy since yes obviously a certain intensity of climate catastrophe can kill everyone, but no scientific prediction has said this is likely to be the case.

You guys follow the same playbook. Ever single time. Once I link to something, you come back and tell me why that link is invalid and can't be trusted. You guys are like robots, and you all have done this same playbook on reddit for two decades now.

"My sources go to a different school... in Canada... you wouldn't trust them"

There are just as many saying otherwise. For every Hinton or Bengio there's a LeCun or Reddy. In other words: Beware of confirmation bias.