Just curious, why might we not be able to align super intelligence? I’m extremely ignorant in this space so forgive me if it’s a dumb question but I am definitely curious to learn more

1. Models aren't "programmed" so much as "grown". We know how GPT is trained but we don't know what it is learning exactly to predict the next token. What do the weights do ? We don't know. This is obviously problematic because it makes interpretability not much better than for humans. How can you ascertain to control something you don't even understand ?

2. Hundreds of thousands of years on earth and we can't even align ourselves.

3. SuperIntelligence would be by definition unpredictable. If we could predict its answers to our problems, it wouldn't be necessary. You can't control what you can't predict.

A more cynical take would be they'll be spending the compute on more mundane engineering problems like making sure the AI doesn't say any naughty words, while calling it "Super Intelligence Alignment Research."

This effort is led by Ilya Sutskever. Listening to a bunch of interviews with him, and talking to a bunch of people who know him personally, I don't think he cares at all about AIs saying naughty words.

Openai certainly does, they are sending out emails to people using the AI for NSFW roleplay and warning them they'll be banned if they continue. They've also recently updated their API to make it harder to generate NSFW content.

It's very obvious that it is an issue. Everyone but a few denialists get it instantly "hey, would you like to build something smarter without knowing how to control it"?