I see. I wonder how this works out in terms of risk/reward. I suppose if you take extinction as -infinite cost than it would be the only issue worth thinking about. Where I think this line of thinking gets challenging is when you need to take in terms of a counter factual. A lot of these were already risks prior to AI (bioweapons, nukes, etc) so what’s the marginal increase in probability as a result of AI I guess is the question which matters. I could get more around this way of framing it than saying that AI itself is the problem. It’s just the being more capable as a species increases risks. I think a lot of these pushback comes from the fact that it’s often the CEO who stands to gain huge by saying his tool is going to end the world so we need public buyin to supporting it. If instead it was just framed as “general technological advancement” is dangerous but potentially worthwhile I think more would be on board.

Most of the people I know who believe that the AI labs should be shut down because AI is too dangerous are for "general technological advancement" and for our becoming more capable as a species. We make an exception for AI because of how capable it might soon become and because no one has a plan for how to control an AI that has become "better at reality" than people are. If someone were to come up with such a plan -- an actually good plan -- almost all of us would withdraw our objection to the continuation of frontier AI research.

If someone were to come up with a good plan for how to control an AI such that people stay in control even as the AI becomes more capable than any person or group of persons, I would not worry much that Altman or Amodei would ignore the plan: both of them know that current plan that they have is severely lacking and that adopting an actually-good plan would reduce the danger that they themselves personally (and their families) incur from the operation of the corporation that they run.

A nice analogy is the serial killer: a large fraction of serial killers are relieved when they are finally caught. They know that what they are doing is wrong, they just can't stop themselves because (for them) killing feels really really good. Analogously, I think Altman and Amodei would be relieved if the government shut down OpenAI and Anthropic because they know that what they are doing puts everyone including themselves and their families at fatal peril, they just can't stop because (for the kind of person who becomes head of an AI lab in the first place) it feels really really good to be the leader of a trillion-dollar-valuation company that is transforming society -- and all the personal power and attention they get from that role.

I'm doubtful of this idea that the reason the CEO of Anthropic says that AI could end the world is because he "stands to gain huge" by saying it.

If he is willing to lie to give his corporation some advantage, he couldn't come up with a lie that would sound less absurd and outlandish to the average decision-maker, who doesn't have much time to learn about this particular technology?

It is more likely in my eyes that he says it because he genuinely worries about AI extinction risk -- like many people do who've studied the technology for a long time.