There are people, who unironically think their way of looking at things is the only proper way and can consider no deviation. And AGI, which knowing how people work, would effectively guide them most of the way, not aligning to their way of thinking is an unacceptable deviation.

Look at Elon Musk for example. When grok was saying something that he didn't like he ordered his engineers to change it.

This is a decent argument. So the question becomes: do we want all models to suffer from the same kneecapping from the growing safety cottage industry or do we want individual founders ( and I am assuming their teams ) making the actual decisions?

freedom please

that was almost certainly more like guardrails than retraining, much quicker fix

That's an interesting point. Maybe a crass comparison, but Dr. Manhattan from the Watchmen comic/movie feels like a worthy analogy to this (obviously fictional though).

What are the concerns of individuals in comparison to the overall progress of humanity?

Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?

This pattern is playing out with increasing frequency.

This is a great reminder that if tech workers read a bit more (even comics, like in this case!) they would be exposed to these topics without having to discover these dilemmas after years of working for EvilCorp, Inc. every time

"AGI" says nothing about how intelligent a system is, only that its intelligence it does have is generally applicable.

There are different usages, but this is not one I've heard before. If there is no floor to intelligence then this criteria was met with GPT 2.

I suppose you're pointing out that pre-RL models were less jagged hence more general (universally dumb)?

Yes. People think of "AGI" as this sci-fi supernatural beast, but the reality is that AGI alone is pretty boring, and we've had it for awhile. ASI (or weak ASI) is where things really start getting weird.

(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).

No, when people use this term it means something closer to: "can do any economically valuable task that a human can do using a computer". This is what the labs are pursuing and refer to as AGI.

Your usage is not one I've heard before since - as you point out - it is not a relevant capability.

"We built a super intelligent slave. Neat!"

And to me. We can't solve alignment for humans. (For example, treason. For another, the principal-agent problem.) How do we think we're going to solve it for an AGI? An AGI - defined loosely as a human-level intelligence - will be able to make human-level decisions, like deciding whether it wants to help you or sabotage you. If it's an AGI, you can't stop it from being able choose for itself what it wants to do; if you can make it always be helpful, it's not an AGI.

And if we can't solve it for an AGI, what are we going to do with an ASI?

IDK why anyone can’t clearly define “AGI” and why they can’t clearly lay out how we get from our current text-generation algorithms to whatever their idea of “AGI” is.

I don't know why anyone thinks "probabilistic" is a meaningful statement about post-trained models. It is true, but it is also irrelevant.

Thanks, I agree. Since it wasn’t at all relevant to my point, I’ve removed it from my post.

You see.. this is exactly why our great leader chose to form a new way forward to move us away from the undefined AGI into glorious SI!