Which is very worrying! That's part of why I left big tech to work on detecting novel pandemics. [1] Luckily, the people capable of killing everyone seem not to have wanted to so far.

But if the bar keeps lowering, at some point it will be accessible to "humanity is a cancer" folks and I do think we'll see serious efforts to wipe out humanity.

[1] https://www.jefftk.com/p/computational-approaches-to-pathoge...

Who are the "humanity is a cancer" folk? I thought this was a trope used to discredit environmentalists?

Movies and strawmen alike.

Most people think continued existence of humanity is good, which includes environmentalists. But there are some people who either think (a) avoiding suffering is overwhelmingly important, to the extent that a universe without life is ideal or (b) the world would be better off without humans because of the vast destruction we cause. I'm not claiming this is a common view, but it doesn't have to be a common view to result in destruction if destruction gets sufficiently straightforward.

For some writing in this direction, see https://longtermrisk.org/risks-of-astronomical-future-suffer... which argues that suffering-focused people should not try to kill everyone primarily because they are unlikely to succeed and https://philarchive.org/archive/KNUTWD ("Third, the negative utilitarian can argue that losing what currently exists on Earth would not be much of a loss, because of the following very plausible observation: overall, sentient beings on Earth fare terribly badly. The situation is not terrible for every single individual, but it is terrible when all individuals are considered. We can divide most sentient beings on Earth into the three categories of humans, non-human animals in captivity, and wild non-human animals, and deal with them in turn. ...")

This doesn't really answer my question. These are arguments for why threat actors might hold the belief that "humanity is cancer", but it doesn't actually provide evidence that this is a credible threat in the real world.

The two people I cited at the end are examples of philosophers who think a lifeless world is preferable to the status quo and to futures we are likely to get. These aren't arguments for why someone might hold the belief, but evidence that some people do seriously hold the belief.

(Another example here would be https://en.m.wikipedia.org/wiki/Aum_Shinrikyo)

Fair enough. I don't doubt there are people in the world that hold this view. I just don't know how credible they are as a threat to humanity.

Elon Musk and Joe Rogan were just complaining about one of them on Rogan’s podcast a day or two ago. He was on the cover of Time magazine, but I can’t find it on Time’s site. If you dig up that podcast on YT you should be able to see it.

The calculation of considering something is a credible threat directly corresponds with the ease of implementation.

If I said "I am going to drop a nuke on jebarker" that's not a credible threat unless I happen to be a nation state.

Now, if I said "I'm going to come to jebarkers house and blast him with a 12 gauge" and I'm from the US, that is a credible threat in most cases due to the ease of which I can get a weapon here.

And this comes down to the point of having a magic machine in which you could ask for the end of the world. The more powerful that machine gets the bigger risk it is for everyone. You ever see those people that snap and start killing everyone around them? Would you want them to have a nuke in that mood? Or a 'lets turn humanity in to grey goo' option?

The problem with this analysis is that we have complete uncertainty about the probability that AGI/ASI will be developed in any particular time frame. Anyone that says otherwise is lying or deluded. So the risk equation is impossible to calculate for AGI/ASI wiping out humanity. And since you appear to be evaluating the damage as essentially infinite, i.e. existential risk, you're advocating that as long as there's a greater than zero probability of someone using that capability then the risk is infinite. Which is not useful for deciding a course of action.

No, no need to calculate infinities. Instead lets imagine something with an IQ within one exponent higher of ours. This is exceptionally difficult to even imagine as below 70 we consider a person disabled in some form or another. Once we get in the range of IQ over 170 or so it really starts to break. When we get into multimodal self learning systems there may be vast amounts of knowledge encoded into its networks that we simply don't know the right question to ask as humans. Attempt to think of what other AIs at or near that level could ask each other.

Our different systems of intelligence keep falling back to ideas of human intelligence and its limitations. Things like we don't multitask well, our best attention is given to one thing at a time. Things like our highest bandwidth senses are our ears and eyes, and even then they are considerably bandwidth constricted. Human limits like our abilities cannot be horizontally scaled in an easily measurable way, and adding more people massively increases the networking costs of accomplishing a goal.

If for some reason the smartest AI can only reach the levels of the smartest human beings (which at least to me makes no sense), then this is still massively powerful, as of this point pesky human problems like 8 hours of sleep are not needed. If human level AI power can be shrunk down to near cellphone size and power consumption, which again doesn't seem out of the realm of physics, lays down the framework for an intelligence explosion at the rate which we can print chips out and assemble them.

But the problem still stands that we can't estimate how likely any of this is to occur. So in the overall risk equation we have a term that is essentially completely uncertain and therefore varying that term can yield any arbitrary risk level you want.

And how many of them are radical religious nutjobs? I think it is plausible that "can spend decades learning a scientific craft" and "wants to kill everyone on the planet" are mutually exclusive personality traits.

there is no OR gates in biology, only mutable distributions. the probability might be small for the two traits to appear, the overlap will occur with some frequency over billions of people. Also, and more importantly, the first condition is pretty heavily biological, the second is more of a social consequence. Just take a genius, give him internet and a healthy dollop of trauma, push the mysanthropism to 11 and watch the fireworks.

Wasn't there a study that showed there are disproportionally many engineers among the more violent "radical religious nutjobs"? The hypothesis I heard for this was that STEM people are used to complex arguments and more abstract reasoning, so while they may be harder to initially convince, when they do get convinced, they're more likely to follow the new beliefs to their logical conclusions...

How does that number compare to the number of people (experts and laypeople) who have access to the mentioned AI models?

Risk = severity x likelihood

I think the OP's point was that AI increases the likelihood by dramatically increasing the access to that level of knowledge.

So it’s open and shared knowledge you think is bad? Good to know.

Please go review the HN guidelines: https://news.ycombinator.com/newsguidelines.html

And to address the snarky mischaracterization, like with most things in life "it depends." As a general rule, I'm in favor of democratizing most resources, to include information. But there are caveats. I don't think, for example, non-anonymized health information should be open knowledge. Also, from the simple risk equation above, as the severity of consequence or the likelihood of misuse go up, there should probably be additional mitigations in place if we care about managing risk.

There is a massive difference between "there are a few thousand people in the world who might have the specialized knowledge to kill billions" and "anyone with a handful of GPUs or access to a handful of cloud systems could have a conversation with LatestAIUnlimited-900T and make a few specialized mail orders".

The Venn diagram of "people who could arrange to kill most of humanity without being stopped" and "people who want to kill most of humanity" is thankfully the empty set. If the former set expands to millions of people, that may not remain true.

But someone could ask the AI how to stop it.

It tends to be a lot easier to destroy than create, because entropy sucks like that.

there's a much easier way to create a global pandemic: wait for the next Ebola outbreak, hop on a plane, get infected, fly to NYC and spread your bodily fluids everywhere. I was (pleasantly) surprised that ISIL didn't try this.

and making a pandemic virus is considerably more involved than making "a few specialized mail orders." and if it becomes easier in the future, far better to lock down the mail orders than the knowledge, no?

Thankfully, Ebola is not a good candidate for a global pandemic for a number of reasons. It is much easier to avoid a virus spread by bodily fluid contact than one which is airborne, and Ebola tends to kill its hosts rather quickly. I'm more excited about SARS-CoV-5 (Captain Trips).

Generating bioweapons is not a trivial task even if you know exactly "how to do it". Ask any biologist how easy it is to grow X your very first attempt.

Then remember buying starter cultures for most bioweapons isn't exactly easy and something anyone is allowed to do.

Even an omniscient AI cannot overcome a skill issue