I have no clue why more people don't agree with his arguments. I tend not to hold strong opinions in general, but for me the danger of AGI is so obvious I'd argue it's even more of a clear and present threat than nuclear weapons. Literally everything about AGI is bad and there's almost zero incentive not to press the metaphorical red button.

The problem with Yudkowsky is that he's too smart for his own good. We need to stop talking about AI safety like it's a field of philosophy and explain the threat very plainly to people who don't understand how neural nets work.

Those who disagree with him from what I see generally fall into three camps:

1. The 99.5% of the population who are not educated enough to speak on the subject. These are the kinds of people who are so misinformed they think AGI will be good because it's smart or because it's "programmed" by humans.

2. Misaligned actors. Eg, CTOs and CEOs of AI startups. These people have both a financial risk and reputation risk from talking about the dangers of their hobby. However, these people are unfortunately the media's favourite gotos when they want to understand the dangers of the technology.

3. Naive techno optimists. These are the people who don't even bother making counter arguments and instead rely on slandering people like Yudkowsky as doomers or as being anti-progress. They generally think AGI will be good because some incomparable past technology was good.

I've literally never had a good debate on this subject and I've been debating with people about this for over a decade now. It's frustrating to the point of exhaustion trying to explain the risks. Every now and then you come across someone who gets it and that's nice, but they're hard to find. Generally these people are educated and are familiar with ML algorithms despite not having their reputation and income tied to it.

So yeah, if you couldn't tell I think Yudkowsky is a genius and way ahead of most people on this, but by the time humanity realises it it will be too late.