It depends on what you mean by alignment. If you mean the sense it's used now, meaning bias in models, security and safety for users and so on, then that's a way more mundane version of alignment than what Yudkowsky and ilk promoted. They are talking about a superintelligent being possibly destroying humanity / the earth. And it wasn't that hard to predict. We have many examples of more technological / "advanced" people ruling or extinguishing weaker people. So a superintelligent AI being a threat is not a huge leap.

IMO they have been consistently too aggressive on timelines. When GPT-4 came out, Yudkowsky said it might be conscious. I think he has written interesting stuff but let's be real

> When GPT-4 came out, Yudkowsky said it might be conscious. I think he has written interesting stuff but let's be real

I don't expect GPT-4 to be conscious, it would be quite remarkable if we've stumbled upon whatever evolution did that made us conscious.

But how would you even tell, given we don't know the mechanism or physical process that un-thinking and un-planning natural selection happened to stumble upon to give it to us?

Sure, we can't tell, but that's a really bad argument to make the claim that it _is_ conscious. I just have a really hard time believing that a transformer can be conscious, but yes I can't prove it.

Agreed. I push back equally against all who are confident, regardless of which way.

If we do a survey of stories about AI from the last 50 years, it seems like it's a worry everyone knows about, and that people generally take seriously in proportion to how much they think an AI will actually exist.

Why? Science fiction talked about technology’s behavior and human expectations of it even way before computers became mechanical.

Stories about golems going rogue and genies requiring careful wishing aren’t modern era inventions.