> I've never seen an "alignment proponent" who didn't already agree and consider it a terrible idea.

Their position is that it is a hard problem to solve, but not impossible. They are fully committed to bring about AGI/ASI.

Here are two things:

- Creating an AGI whose goals are compatible with ours.

- Creating an AGI with inscrutable goals and safely confining it.

They're not the same!

In whoever you've been talking to I can't recognize the rationalist scene I've known since it grew out of the 90s extropians. "AI boxing" was a live argument around the early 2000s. Not now.

Online you often see people in that community saying that slower progress in AI capabilities would be a very good thing. At the same time, yes, they do want us to achieve AGI/ASI as soon as possible, provided we don't screw it up.

> At the same time, yes, they do want us to achieve AGI/ASI as soon as possible, provided we don't screw it up.

That's the point, it is not possible. The premise is based on a paradox and you can't throwaway containment and have alignment that remains defined by original intent.

If you want to argue that it's logically impossible for a smarter-than-human intelligence to be created with values aligned with human ones, then cool, do that. (Note that a smart human on much faster hardware is smarter-than-human. This is not a claim that AGIs must be mere faster humans; it's one example to point out a difficulty for any such argument.)

But I think you gave a false impression of others at the start of this thread.

> If you want to argue that it's logically impossible for a smarter-than-human intelligence to be created with values aligned with human ones, then cool, do that.

That is precisely what I have stated.

> But I think you gave a false impression of others at the start of this thread.

How so?